2 Commits
Author SHA1 Message Date
LeoVasanko 84b9e27ced Rework bot taxonomy: provider field, social/analytics kinds, spoof detection
- Distinguish Meta's five documented crawlers (Facebook, FacebookBot,
  Meta-ExternalAgent/ExternalFetcher/WebIndexer) with distinct kinds
- Rename kind preview -> social for link-sharing unfurlers; BingPreview
  moves to search as it is a search-engine feature
- Add analytics kind for monitoring and ad/SEO measurement crawlers
  (UptimeRobot, Pingdom, MJ12bot, Mediapartners-Google, AdsBot-Google)
- Reclassify Google family by main product use: Storebot-Google and
  Feedfetcher-Google/InspectionTool are search, Read-Aloud and
  GoogleOther are ai; AhrefsBot is search
- Drop retired entries: DuplexWeb-Google, SkypeUriPreview, PhantomJS
- Detect the Qualys SSL Labs scanner by its frozen exact UA string
- Mark ancient (pre-2023) auto-updating browser claims with
  " (spoofed)" in pretty, leaving engine/os/kind empty
- UA dataclass: all fields default to "", new provider field,
  constructions use kwargs
- PROVIDERS is now dict[provider, frozenset[names]] with derived
  PROVIDER_OF and NAME_KIND reverse lookups
- README and prettytable.py updated to match
2026-09-08 23:20:06 +00:00
LeoVasanko b9664aabe4 Initial commit 2026-09-05 00:36:02 +00:00