Google爬虫名称汇总

2020-12-12  乐帮网

seo google

Google所有的爬虫汇总如下:

抓取工具 用户代理令牌(产品令牌) 完整的用户代理字符串
APIs-Google

APIs-Google

APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html)
AdSense

Mediapartners-Google

Mediapartners-Google

AdsBot Mobile Web Android

(检查 Android 网页广告质量)

AdsBot-Google-Mobile

Mozilla/5.0 (Linux; Android 5.0; SM-G920A) AppleWebKit (KHTML, like Gecko) Chrome Mobile Safari (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)

AdsBot Mobile Web

(检查 iPhone 网页广告质量)

AdsBot-Google-Mobile

Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)

AdsBot

(检查桌面设备网页广告质量)

AdsBot-Google

AdsBot-Google (+http://www.google.com/adsbot.html)
Googlebot Image
  • Googlebot-Image
  • Googlebot
Googlebot-Image/1.0
Googlebot News
  • Googlebot-News
  • Googlebot
Googlebot-News
Googlebot Video
  • Googlebot-Video
  • Googlebot
Googlebot-Video/1.0

Googlebot(桌面版)

Googlebot

  • Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  • Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36

    或(很少使用):
  • Googlebot/2.1 (+http://www.google.com/bot.html)

Googlebot(智能手机版)

Googlebot

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Mobile AdSense

Mediapartners-Google

(各类移动设备)(compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)

Mobile Apps Android

(检查 Android 应用页面广告质量。遵守 AdsBot-Google 漫游器规则。)

AdsBot-Google-Mobile-Apps

AdsBot-Google-Mobile-Apps

Feedfetcher

FeedFetcher-Google

error不遵循 robots.txt 规则

FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)

Google Read Aloud

Google-Read-Aloud

error不遵循 robots.txt 规则

现用代理:

  • 桌面版代理:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36 (compatible; Google-Read-Aloud; +https://developers.google.com/search/docs/advanced/crawling/overview-google-crawlers)
  • 移动版代理:Mozilla/5.0 (Linux; Android 7.0; SM-G930V Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36 (compatible; Google-Read-Aloud; +https://developers.google.com/search/docs/advanced/crawling/overview-google-crawlers)

曾用代理(已弃用):google-speakr

网页上的 Duplex

DuplexWeb-Google

error可能会忽略 * 用户代理通配符

Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; DuplexWeb-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Mobile Safari/537.36

Google Favicon

(检索各种服务的网站图标)

Google Favicon

error对于用户发起的请求,会忽略 robots.txt 规则

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon
Web Light

googleweblight

error不遵循 robots.txt 规则

Mozilla/5.0 (Linux; Android 4.2.1; en-us; Nexus 5 Build/JOP40D) AppleWebKit/535.19 (KHTML, like Gecko; googleweblight) Chrome/38.0.1025.166 Mobile Safari/535.19

相关文章:

Google云自动扣款如何要回来?

无情的被Google云收割啦

robots.txt 示例(百度、google、bing)

Google爬虫名称汇总

Google 云服务器免费申请使用12个月Google Cloud

公众号二维码

关注我的微信公众号
在公众号里留言交流
投稿邮箱:1052839972@qq.com

庭院深深深几许?杨柳堆烟,帘幕无重数。
玉勒雕鞍游冶处,楼高不见章台路。
雨横风狂三月暮。门掩黄昏,无计留春住。
泪眼问花花不语,乱红飞过秋千去。

欧阳修

付款二维码

如果感觉对您有帮助
欢迎向作者提供捐赠
这将是创作的最大动力