Files

T

LeoVasanko 84b9e27ced Rework bot taxonomy: provider field, social/analytics kinds, spoof detection

- Distinguish Meta's five documented crawlers (Facebook, FacebookBot,
  Meta-ExternalAgent/ExternalFetcher/WebIndexer) with distinct kinds
- Rename kind preview -> social for link-sharing unfurlers; BingPreview
  moves to search as it is a search-engine feature
- Add analytics kind for monitoring and ad/SEO measurement crawlers
  (UptimeRobot, Pingdom, MJ12bot, Mediapartners-Google, AdsBot-Google)
- Reclassify Google family by main product use: Storebot-Google and
  Feedfetcher-Google/InspectionTool are search, Read-Aloud and
  GoogleOther are ai; AhrefsBot is search
- Drop retired entries: DuplexWeb-Google, SkypeUriPreview, PhantomJS
- Detect the Qualys SSL Labs scanner by its frozen exact UA string
- Mark ancient (pre-2023) auto-updating browser claims with
  " (spoofed)" in pretty, leaving engine/os/kind empty
- UA dataclass: all fields default to "", new provider field,
  constructions use kwargs
- PROVIDERS is now dict[provider, frozenset[names]] with derived
  PROVIDER_OF and NAME_KIND reverse lookups
- README and prettytable.py updated to match

2026-09-08 23:20:06 +00:00

data

Initial commit

2026-09-05 00:36:02 +00:00

bench.py

Updated README.

2026-09-05 05:12:22 +00:00

download_data.py

Initial commit

2026-09-05 00:36:02 +00:00

prettytable.py

Rework bot taxonomy: provider field, social/analytics kinds, spoof detection

2026-09-08 23:20:06 +00:00

README.md

Initial commit

2026-09-05 00:36:02 +00:00

README.md

Benchmark scripts

Recreate every statistic and table quoted in the top-level README. uarite must be importable (e.g. pip install -e . or run with PYTHONPATH pointing at the repo root); the reference parsers are benchmark-only dependencies, pulled ad hoc via uv run --with.

download_data.py — fetch the external datasets into data/: the 100 modern browser UAs (top-user-agents npm package) and the crawler corpus (monperrus/crawler-user-agents). data/ua.txt is a committed real-traffic sample, not downloaded.
```
uv run scripts/download_data.py
```

prettytable.py — prints the accuracy-comparison markdown table:

uv run --with ua-parser --with user-agents python scripts/prettytable.py

bench.py — prints browser-accuracy counts, crawler-detection rates, URL-extraction coverage, and the timing tables (unique UAs, realistic mix, bot storm) with cache statistics:
```
uv run --with ua-parser --with user-agents --with user-agent-parser \
    python scripts/bench.py
```