Rework bot taxonomy: provider field, social/analytics kinds, spoof detection
- Distinguish Meta's five documented crawlers (Facebook, FacebookBot, Meta-ExternalAgent/ExternalFetcher/WebIndexer) with distinct kinds - Rename kind preview -> social for link-sharing unfurlers; BingPreview moves to search as it is a search-engine feature - Add analytics kind for monitoring and ad/SEO measurement crawlers (UptimeRobot, Pingdom, MJ12bot, Mediapartners-Google, AdsBot-Google) - Reclassify Google family by main product use: Storebot-Google and Feedfetcher-Google/InspectionTool are search, Read-Aloud and GoogleOther are ai; AhrefsBot is search - Drop retired entries: DuplexWeb-Google, SkypeUriPreview, PhantomJS - Detect the Qualys SSL Labs scanner by its frozen exact UA string - Mark ancient (pre-2023) auto-updating browser claims with " (spoofed)" in pretty, leaving engine/os/kind empty - UA dataclass: all fields default to "", new provider field, constructions use kwargs - PROVIDERS is now dict[provider, frozenset[names]] with derived PROVIDER_OF and NAME_KIND reverse lookups - README and prettytable.py updated to match
This commit is contained in:
@@ -55,10 +55,6 @@ CASES = [
|
||||
"Googlebot (disguised)",
|
||||
"Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)",
|
||||
),
|
||||
(
|
||||
"Claude-SearchBot (disguised)",
|
||||
"Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.3245.171 Mobile Safari/537.36; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot",
|
||||
),
|
||||
(
|
||||
"Facebook preview (disguised)",
|
||||
"Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.6885.65 Mobile Safari/537.36; compatible; facebookexternalhit/1.1; +http://www.facebook.com/externalhit_uatext.php",
|
||||
|
||||
Reference in New Issue
Block a user