Verifying bots via dns

[chrillep]

add a feature for dns verification as per
Verifying Googlebot

Not sure if this is possible on all bots.
but here is an implementation for google
https://www.npmjs.com/package/googlebot-verify
https://github.com/jcowley/googlebot-verify/blob/b30ad73ae6e7ca8936b7a45e885148fdf1072c49/index.js

[omrilotan]

Funny you should mention this. I've also implemented a slightly more general reverse DNS lookup implementation:

https://www.npmjs.com/package/reverse-dns-lookup

const reverseDNSLookup = require('reverse-dns-lookup');
const isGoogleServers = await reverseDNSLookup(ip, 'google.com', 'googlebot.com');

I feel like this should be complementary functionality.

---

You can read in the readme https://github.com/omrilotan/isbot#clarifications:

This package aims to identify "Good bots". Those who voluntarily identify themselves by setting a unique, preferably descriptive, user agent, usually by setting a dedicated request header.

This package has not set out to verify bots, only to recognise (happy flow, I guess). I don't think I'll start maintaining IP address lists and DNS provider names. Verification is a security matter, it should be under better scrutiny than I'm able to provide.

I'll try to make this point clearer in the readme.

[chrillep]

No it's a fair point. and maintaining a list of ips and dns names would be horrendous by yourself. But maybe there already is a package for that =)

[chrillep]

ill just use both packages then :). Thanks for sharing!

[chrillep]

ok so this guy did it but it's basically just different domain names

https://www.npmjs.com/package/is-google
https://github.com/roccomuso/is-google/blob/73e3a8705f163bbf3198956c1fd5db4a283f78cb/index.js#L10

https://www.npmjs.com/package/is-bing
https://github.com/roccomuso/is-bing/blob/6f9e9241212d8da38aa41947d3733914cabc2773/index.js#L11

https://www.npmjs.com/package/is-apple
https://github.com/roccomuso/is-apple/blob/7e080844a702f67dfeaaabc6feadaf651ee905a6/index.js#L10

https://www.npmjs.com/package/is-twitter
https://github.com/roccomuso/is-twitter/blob/dd591834463cf119c7f22bc5d7266d37da63ea34/index.js#L10

https://www.npmjs.com/package/is-baidu
https://github.com/roccomuso/is-baidu/blob/277d1360a01b71550ee385f92359097d0e3b66f6/index.js#L10

[omrilotan]

Just released a more sustainable version (2.0.0) of my other package, you can use that instead of all these specific ones if you want to maintain the domain list:

https://www.npmjs.com/package/reverse-dns-lookup

const { verify } = require('reverse-dns-lookup');
const { getClientIp } = require('request-ip');

const crawler_domains = [
  '.google.com', '.googlebot.com',
  'search.msn.com', // Bing
  '.applebot.apple.com',
  '.twttr.com', // Twitter
  '.crawl.baidu.com' // Baidu craler
];

const isCrawlerServer = await verify(clientIp, ...crawler_domains);