pagespeed

[trambulho]

Hello, does it not work with pagespeed anymore?

[omrilotan]

In my experience, all tests from https://pagespeed.web.dev/ use the Chrome-Lighthouse substring to identify as bot.

Examples:

• Mozilla/5.0 (Linux; Android 7.0; Moto G (4)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4590.2 Mobile Safari/537.36 Chrome-Lighthouse
• Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4590.2 Safari/537.36 Chrome-Lighthouse

Check example on isbot.js.org

[trambulho]

Right, when executing the website url in pagespeed before it alerted that it was a bot, now not anymore.

https://pagespeed.web.dev/analysis/https-isbot-js-org/69ojbpf9jt?form_factor=mobile

[omrilotan]

I can see Pagespeed have decided to not recognise themselves as a bot using the user agent string. As you can see in Issue #213

What does "isbot" do?

This package aims to identify "Good bots". Those who voluntarily identify themselves by setting a unique, preferably descriptive, user agent, usually by setting a dedicated request header.

What doesn't "isbot" do?

It does not try to recognise malicious bots or programs disguising themselves as real users.

If a tool uses legitimate browser user agent string with no indication of being an automated service - we can not use this tool to identify it.

In Discussion #162, we describe how to use other attributes of the request to identify crawlers.

Using reverse-dns-lookup, we can use reverse DNS to identify some crawlers

import { verify } from "reverse-dns-lookup";
import requestIp from "request-ip";

const crawler_domains = [
  '.google.com', '.googlebot.com',
  'search.msn.com', // Bing
  '.applebot.apple.com',
  '.twttr.com', // Twitter
  '.crawl.baidu.com' // Baidu craler
];

const isCrawlerServer = await verify(
    requestIp.getClientIp(request),
    ...crawler_domains
);

[trambulho]

Thank you very much for the excellent support