diff --git a/README.md b/README.md index 3749421..fb0491b 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,6 @@ # User-Agent Parsing Done Right -There are plenty of UA parsers for Python: ua-parser is the official Python port of the large upstream project, while user-agents builds on top of it with higher-level device detection. Among newer implementations, user-agent-parser is the most promising and is included here for comparison. - -This module takes a smaller, faster, modern approach. It's a dependency-free pure-Python parser weighing only 25 kB, with strong handling of current browsers and crawlers. Despite its light weight, uarite identifies both browsers and crawlers more accurately than any competing implementation tested here. +This module takes a smaller, faster, modern approach to User-Agent parsing. It's a dependency-free pure-Python parser weighing only 25 kB, with strong handling of current browsers and crawlers. Despite its light weight, uarite identifies both browsers and crawlers more accurately than any competing implementation tested here. It returns structured classification, but also the thing most applications eventually need: **a short pretty description**. @@ -52,56 +50,61 @@ The engine and os fields are intentionally broad. The kind field distinguishes b Detection is necessarily limited by what the User-Agent reveals. Crawlers can masquerade as ordinary browsers or other crawlers, so sites that need stronger identification should use additional methods rather than relying on UA detection alone. -## Accuracy +## Comparison -The table below compares representative User-Agent formats with ua-parser. user-agents produces nearly identical results and is left out. +There are plenty of UA parsers for Python. This comparison includes the ua-parser family — the official port of the large upstream project, benchmarked with all three of its regex backends (Python, RE2, Rust) — plus user-agents, which builds on it with higher-level device detection, and fastuaparser, a single-file heuristic parser in the same weight class as uarite. Among those excluded: user-agent-parser crashes on real-world strings; httpagentparser misidentifies most bots and browser OSes; device-detector has a massive 26 MB install and extremely slow parsing; and crawlerdetect only distinguishes bots from non-bots. The user-agents package has been unmaintained since 2020 but is kept as a reference point. -| Case | uarite¹ | ua-parser² | -| ---------------------------- | ------------------------- | --------------------------------------------- | -| Chrome, Windows | Chrome/152 Windows | Chrome/152 Windows | -| Safari, macOS | Safari/18 macOS | Safari/18 Mac OS X Mac | -| Opera, Linux | Opera/106 Linux | Opera/106 Linux | -| Chrome, Android (no model) | Chrome/152 Android | Chrome Mobile/152 Android K❌ | -| Chrome, Android (Pixel) | Chrome/118 Pixel 6 | Chrome Mobile/118 Android Pixel 6 | -| Edge, Android (model code) | Edge/110 Galaxy S7 | Edge Mobile/110 Android Samsung SM-G930P | -| Firefox, Android | Firefox/154 Android 15 | Firefox Mobile/154 Android Generic Smartphone | -| Safari, iPhone | iPhone iOS 17 | Mobile Safari/17 iOS iPhone | -| Huawei HarmonyOS phone | HuaweiBrowser/6 HarmonyOS | Huawei Browser/6 Android❌ Huawei Browser | -| GPTBot | GPTBot (AI) | GPTBot/1 Spider | -| Googlebot (disguised) | Googlebot (search) | Googlebot/2 Android❌ Spider | -| Facebook preview (disguised) | Facebook | FacebookBot/1 Android Pixel 7 ❌ | -| Meta crawler (disguised) | Meta-ExternalAgent (AI) | Chrome/145 Windows ❌ | -| WhatsApp preview | WhatsApp | WhatsApp/10 Spider | -| Bytespider | Bytespider | Bytespider/ Android❌ Generic Smartphone | -| BingPreview | BingPreview | BingPreview/1 Windows❌ Spider | -| AhrefsBot | AhrefsBot | AhrefsBot/7 Spider | -| python-requests | python-requests/2.32.5 | Python Requests/2 | - -- ❌ marks an incorrect data such as OS from disquise or Android 10; K (compat) on modern devices -- ¹ `r.pretty` shown as is -- ² `{r.user_agent.family}/{r.user_agent.major} {r.os.family} {r.device.family}` - -On our modern-browser test set, **uarite resolves family, version, and OS at 100%**. ua-parser and user-agents score 80%, while user-agent-parser reaches 92%. - -Crawler detection was tested against real-world UAs from [monperrus/crawler-user-agents](https://github.com/monperrus/crawler-user-agents). user-agent-parser scored 32%, user-agents 60%, and ua-parser 64%. **uarite scores 97%**, identifying the specific crawler by name in 80% of cases. +Installed size tracks the design. The minimal heuristic parsers are tiny: fastuaparser installs in 15 kB, uarite in 30 kB. Everything built on ua-parser's regex database starts around half a megabyte, and its accelerated backends add several megabytes of native code. The benchmark and test scripts are available in the repository's scripts folder. -## Performance +### Accuracy -Import and first parse takes about **10 ms** for uarite, compared with 50–90 ms for ua-parser depending on backend. +The table below compares representative User-Agent formats with fastuaparser and ua-parser. The user-agents wrapper produces nearly identical results to ua-parser and is left out. -![user-agent-parser 120 thousand, uarite 56 thousand, ua-parser native code variants Rust 21 thousand, RE2 13 thousand, and finally plain Python ua-parser and user-agents 3 thousand](https://git.zi.fi/LeoVasanko/uarite/raw/branch/main/docs/bench-speed.svg) -_User-Agents parsed per second per CPU core, first parse of unseen strings, with equal shares of browser and crawler UAs. One-off setup costs excluded._ +| Case | uarite¹ | fastuaparser² | ua-parser³ | +| ---------------------------- | ------------------------- | ------------------------- | --------------------------------------------- | +| Chrome, Windows | Chrome/152 Windows | Chrome - Windows | Chrome/152 Windows | +| Safari, macOS | Safari/18 macOS | Safari - Mac OS X | Safari/18 Mac OS X Mac | +| Opera, Linux | Opera/106 Linux | Opera - Linux | Opera/106 Linux | +| Chrome, Android (no model) | Chrome/152 Android | Chrome - Android Mobile | Chrome Mobile/152 Android K❌ | +| Chrome, Android (Pixel) | Chrome/118 Pixel 6 | Chrome - Android Mobile | Chrome Mobile/118 Android Pixel 6 | +| Edge, Android (model code) | Edge/110 Galaxy S7 | Edge - Android Mobile | Edge Mobile/110 Android Samsung SM-G930P | +| Firefox, Android | Firefox/154 Android 15 | Firefox - Android Mobile | Firefox Mobile/154 Android Generic Smartphone | +| Safari, iPhone | iPhone iOS 17 | Safari - iOS Mobile | Mobile Safari/17 iOS iPhone | +| Huawei HarmonyOS phone | HuaweiBrowser/6 HarmonyOS | Chrome - Android Mobile❌ | Huawei Browser/6 Android❌ Huawei Browser | +| GPTBot | GPTBot (AI) | Bot | GPTBot/1 Spider | +| Googlebot (disguised) | Googlebot (search) | Bot | Googlebot/2 Android❌ Spider | +| Facebook preview (disguised) | Facebook | Chrome - Android Mobile❌ | FacebookBot/1 Android Pixel 7 ❌ | +| Meta crawler (disguised) | Meta-ExternalAgent (AI) | Bot | Chrome/145 Windows ❌ | +| WhatsApp preview | WhatsApp | Browser - Other❌ | WhatsApp/10 Spider | +| Bytespider | Bytespider | Bot | Bytespider/ Android❌ Generic Smartphone | +| BingPreview | BingPreview | Browser - Windows❌ | BingPreview/1 Windows❌ Spider | +| AhrefsBot | AhrefsBot | Bot | AhrefsBot/7 Spider | +| python-requests | python-requests/2.32.5 | Other | Python Requests/2 | -All parsers cache results. With cache hits, **uarite reaches about 36 million lookups per second**, compared with about 5 million for ua-parser and 600 000 for user-agents. +- ❌ marks incorrect data such as an OS from disguise, Android 10; K (compat) on modern devices, or a missed identity +- ¹ `r.pretty` shown as is +- ² `parse_ua(ua)` shown as is +- ³ `{r.user_agent.family}/{r.user_agent.major} {r.os.family} {r.device.family}` + +On our modern-browser test set, scoring only fully correct results (family, version, and OS all right): **uarite 100%**, ua-parser and user-agents 79%. The fastuaparser heuristic resolves the family and OS in 91% of cases, but reports no version numbers at all, so it cannot be scored on the full criterion. + +Crawler detection was tested against [real world UAs](https://github.com/monperrus/crawler-user-agents): **uarite 97%**, fastuaparser 84%, ua-parser 64%, user-agents 60%. + +### Performance + +Import and first parse takes about **10 ms** for uarite and effectively nothing for fastuaparser, compared with 50–90 ms for ua-parser depending on backend. + +![uarite 56 thousand, ua-parser native code variants Rust 26 thousand, RE2 14 thousand, and finally plain Python ua-parser and user-agents 3 thousand](https://git.zi.fi/LeoVasanko/uarite/raw/branch/main/docs/bench-speed.svg) +_User-Agents parsed per second per CPU core, first parse of unseen strings, with equal shares of browser and crawler UAs. One-off setup costs excluded. Cached results and fastuaparser (1 million) are too far off this scale to draw meaningfully and are left out of the graph._ + +On raw speed fastuaparser wins: a few string searches per UA, no cache needed. The trade-off shows in the accuracy table above — no versions, no crawler names. All other parsers cache results. With cache hits, **uarite reaches about 36 million lookups per second**, compared with about 5 million for ua-parser and 600 000 for user-agents. ## Why yet another UA parser -Rather than relying on a large historical regex database, the parser focuses on modern UA formats and parses them directly, choosing the most specific interpretation available. This keeps the implementation small while handling today's browsers and crawler traffic well. - -Until now, I had been using those other modules and building my own pretty-UA formatting on top of them, fixing by post processing issues the upstream didn't care of. +Rather than relying on a large historical regex database, the parser focuses on modern UA formats and parses them directly, choosing the most specific interpretation available. This keeps the implementation small while handling today's browsers and crawler traffic well. Until now, I had been using those other modules and building my own pretty-UA formatting on top of them, fixing by post processing issues the upstream didn't care of. Eventually it became easier to start over with a parser designed around modern traffic. The result is uarite. -Hopefully it helps you too. +Hopefully it helps you too. Star my [GitHub](https://github.com/leovasanko/uarite) if it did. diff --git a/docs/bench-speed.svg b/docs/bench-speed.svg index 42daa8b..562b253 100644 --- a/docs/bench-speed.svg +++ b/docs/bench-speed.svg @@ -1,12 +1,12 @@ - + - 2026-09-08T21:11:49.096249 + 2026-09-09T18:32:39.096963 image/svg+xml @@ -21,67 +21,128 @@ - - +" clip-path="url(#p4041b231bc)" style="fill: #6e6e6e"/> - +" clip-path="url(#p4041b231bc)" style="fill: #6e6e6e"/> - +" clip-path="url(#p4041b231bc)" style="fill: #6e6e6e"/> - +" clip-path="url(#p4041b231bc)" style="fill: #6e6e6e"/> - - - - +" clip-path="url(#p4041b231bc)" style="fill: #2b6cb0"/> - - + + + + + + + - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + - - + + - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + - - + + + - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + - - + + diff --git a/scripts/README.md b/scripts/README.md index 5a08516..94c1c73 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -17,7 +17,7 @@ benchmark-only dependencies, pulled ad hoc via `uv run --with`. - `prettytable.py` — prints the accuracy-comparison markdown table: ``` - uv run --with ua-parser --with user-agents python scripts/prettytable.py + uv run --with ua-parser --with fastuaparser python scripts/prettytable.py ``` - `bench.py` — prints browser-accuracy counts, crawler-detection rates, @@ -25,6 +25,6 @@ benchmark-only dependencies, pulled ad hoc via `uv run --with`. mix, bot storm) with cache statistics: ``` - uv run --with ua-parser --with user-agents --with user-agent-parser \ + uv run --with ua-parser --with user-agents --with fastuaparser \ python scripts/bench.py ``` diff --git a/scripts/bench.py b/scripts/bench.py index 979598e..df8f45a 100644 --- a/scripts/bench.py +++ b/scripts/bench.py @@ -1,16 +1,16 @@ # /// script # requires-python = ">=3.14" # dependencies = [ +# "fastuaparser>=0.1.4", # "ua-parser[re2,regex]>=1.0.2", # "uarite", -# "user-agent-parser>=0.2.1", # "user-agents>=2.2.0", # ] # # [tool.uv.sources] # uarite = { path = "..", editable = true } # /// -"""Benchmark uarite vs ua-parser vs user-agents vs user-agent-parser. +"""Benchmark uarite vs ua-parser vs user-agents vs fastuaparser. Reproduces the README's numbers: browser accuracy on 100 modern UAs, crawler detection on 2163 real-world crawler UAs, and timing (unique UAs, @@ -29,8 +29,8 @@ import timeit from pathlib import Path import ua_parser +from fastuaparser import parse_ua as fua_parse from ua_parser import parse as ua_parse -from user_agent_parser import parse as uap_parse from user_agents import parse as uas_parse from uarite import uaparse @@ -74,6 +74,7 @@ def uap_re2(ua): def uap_rust(ua): return ua_variant("rust")(ua) + DATA = Path(__file__).parent / "data" BROWSERS = json.loads((DATA / "top-user-agents.json").read_text()) CRAWLERS = json.loads((DATA / "crawler-user-agents.json").read_text()) @@ -125,7 +126,6 @@ def score_browsers(): for name, fn in ( ("ua-parser", ua_parse), ("user-agents", uas_parse), - ("user-agent-parser", uap_parse), ): fam_ok = ver_ok = os_ok = 0 for ua in BROWSERS: @@ -135,10 +135,6 @@ def score_browsers(): fam = r.user_agent.family or "" maj = r.user_agent.major or "" osf = r.os.family or "" - elif name == "user-agent-parser": - fam = r[0] or "" - maj = (r[1] or "").split(".")[0] - osf = r[2] or "" else: fam = r.browser.family or "" maj = str(r.browser.version[0]) if r.browser.version else "" @@ -148,6 +144,29 @@ def score_browsers(): ver_ok += emaj == maj os_ok += eos == norm_os(osf) res[name] = (fam_ok, ver_ok, os_ok) + # fastuaparser returns one pretty string ("Chrome - Windows") and no + # version numbers, so score it on the substrings it does produce. + fam_ok = ver_ok = os_ok = 0 + for ua in BROWSERS: + efam, emaj, eos = expected(ua) + pretty = fua_parse(ua) + fam_ok += efam.split()[0].lower() in pretty.lower() + ver_ok += bool(emaj) and emaj in pretty + os_ok += bool( + re.search( + { + "ios": r"iOS|iPhone|iPad", + "macos": r"Mac", + "windows": r"Windows", + "linux": r"Linux", + "android": r"Android", + }[eos], + pretty, + ) + if eos + else True + ) + res["fastuaparser"] = (fam_ok, ver_ok, os_ok) fam_ok = ver_ok = os_ok = 0 for ua in BROWSERS: efam, emaj, eos = expected(ua) @@ -183,7 +202,7 @@ def score_browsers(): def score_crawlers(): out = {} crashes = 0 - for name in ("ua-parser", "user-agents", "user-agent-parser", "uarite"): + for name in ("ua-parser", "user-agents", "fastuaparser", "uarite"): det = 0 for ua in CRAWLER_UAS: try: @@ -195,14 +214,16 @@ def score_crawlers(): ) elif name == "user-agents": bot = uas_parse(ua).is_bot - elif name == "user-agent-parser": - bot = uap_parse(ua)[4] == "Bot" + elif name == "fastuaparser": + # "Bot" is a real verdict; "Other" is a non-browser + # client (wget etc.), which is automated traffic too. + bot = fua_parse(ua).split(" - ")[0] in ("Bot", "Other") else: # Anything not recognized as a real browser is automated: # known bots, generic spiders, clients, spoofed claims. bot = uaparse(ua).kind != "browser" except Exception: - crashes += name == "user-agent-parser" + crashes += name == "fastuaparser" continue det += bot out[name] = det @@ -248,7 +269,7 @@ def bench_realistic(): ("ua-parser (re2)", uap_re2), ("ua-parser (rust)", uap_rust), ("user-agents", uas_parse), - ("user-agent-parser", uap_parse), + ("fastuaparser", fua_parse), ("uarite", uaparse), ): fn = safe(fn) @@ -261,7 +282,7 @@ def bench_realistic(): ("ua-parser (re2)", uap_re2), ("ua-parser (rust)", uap_rust), ("user-agents", uas_parse), - ("user-agent-parser", uap_parse), + ("fastuaparser", fua_parse), ("uarite", uaparse), ): fn = safe(fn) @@ -285,11 +306,8 @@ def cache_info(name): if name == "uarite": i = uaparse.cache_info() return f"{i.hits} hits / {i.misses} misses (cap 1024)" - if name == "user-agent-parser": - from user_agent_parser.parser import _cached_parse_user_agent - - i = _cached_parse_user_agent.cache_info() - return f"{i.hits} hits / {i.misses} misses (cap 512 LRU)" + if name == "fastuaparser": + return "none (branch parser, ~1 µs flat)" if name == "user-agents": from ua_parser.user_agent_parser import _PARSE_CACHE @@ -318,7 +336,7 @@ def bench(): ("ua-parser (re2)", uap_re2), ("ua-parser (rust)", uap_rust), ("user-agents", uas_parse), - ("user-agent-parser", uap_parse), + ("fastuaparser", fua_parse), ("uarite", uaparse), ): fn = safe(fn) @@ -331,8 +349,10 @@ def bench(): if __name__ == "__main__": res, nwork = bench() - print(f"## speed (µs per cold parse, {nwork} unique UAs," - " half browsers / half crawlers)") + print( + f"## speed (µs per cold parse, {nwork} unique UAs," + " half browsers / half crawlers)" + ) for k, v in res.items(): print(f"{k:20} {v:8.1f}") print(f"\n## browser accuracy (n={len(BROWSERS)}): family / version / OS correct") @@ -342,6 +362,6 @@ if __name__ == "__main__": print(f"\n## crawler detection (n={len(CRAWLER_UAS)})") for k, v in det.items(): print(f"{k:18} {v:5} ({v / len(CRAWLER_UAS):.1%})") - print(f"user-agent-parser crashed on {crashes} UAs") + print(f"fastuaparser crashed on {crashes} UAs") print(f"\nuarite URL extraction: {url_got}/{url_have} of instances carrying a URL") bench_realistic() diff --git a/scripts/prettytable.py b/scripts/prettytable.py index 0939ab0..0729d16 100644 --- a/scripts/prettytable.py +++ b/scripts/prettytable.py @@ -1,15 +1,15 @@ """Print the README's accuracy-comparison table as markdown. -uarite's column is simply `r.pretty`. The reference modules have no -display format; their columns use one plain format string each on their -structured output (footnotes ²³ in the README). Requires uarite (installed) -plus the benchmark-only reference parsers: +uarite's and fastuaparser's columns are simply their pretty-string output. +ua-parser has no display format; its column uses one plain format string +on the structured output (footnotes ¹² in the README). Requires uarite +(installed) plus the benchmark-only reference parsers: - uv run --with ua-parser --with user-agents python scripts/prettytable.py + uv run --with ua-parser --with fastuaparser python scripts/prettytable.py """ +from fastuaparser import parse_ua as fua_parse from ua_parser import parse as ua_parse -from user_agents import parse as uas_parse from uarite import uaparse @@ -86,15 +86,8 @@ def imitate_uaparser(ua): return f"{fam}/{maj} {osf} {dev}" -def imitate_useragents(ua): - r = uas_parse(ua) - fam = r.browser.family or "" - maj = str(r.browser.version[0]) if r.browser.version else "" - return f"{fam}/{maj} {r.os.family or ''} {r.device.family or ''}" - - -print("| Case | uarite¹ | ua-parser² | user-agents³ |") +print("| Case | uarite¹ | fastuaparser² | ua-parser³ |") print("|---|---|---|---|") for name, ua in CASES: ours = uaparse(ua).pretty or "—" - print(f"| {name} | {ours} | {imitate_uaparser(ua)} | {imitate_useragents(ua)} |") + print(f"| {name} | {ours} | {fua_parse(ua)} | {imitate_uaparser(ua)} |") diff --git a/scripts/speedplot.py b/scripts/speedplot.py index cf70864..09400a1 100644 --- a/scripts/speedplot.py +++ b/scripts/speedplot.py @@ -23,13 +23,14 @@ matplotlib.rcParams["svg.fonttype"] = "path" # text as paths: renders anywhere import matplotlib.pyplot as plt # noqa: E402 # µs per cold parse, from scripts/bench.py. +# fastuaparser (~1 µs, 1M parses/s) and cached results are too far off +# this scale to draw meaningfully and are only mentioned in the README. US = { - "user-agent-parser": 8.2, - "uarite": 17.7, - "ua-parser (Rust)": 46.6, - "ua-parser (RE2)": 77.1, - "ua-parser": 321.5, - "user-agents": 334.4, + "uarite": 18.0, + "ua-parser (Rust)": 38.1, + "ua-parser (RE2)": 71.7, + "ua-parser": 304.7, + "user-agents": 324.6, } #: Readable on both white and dark backgrounds. @@ -41,29 +42,26 @@ values = [v for _, v in data][::-1] colors = ["#6e6e6e"] * len(data) colors[names.index("uarite")] = "#2b6cb0" -fig, ax = plt.subplots(figsize=(12, 1.7), dpi=100) +fig, ax = plt.subplots(figsize=(12, 1.5), dpi=100) bars = ax.barh(names, values, color=colors, height=0.82) -ax.set_xlim(0, max(values)) +xmax = max(values) +ax.set_xlim(0, xmax) ax.axis("off") for bar, name, v in zip(bars, names, values): # Round to two significant digits: 121951 -> "120 000". rounded = round(v, 1 - int(f"{v:.0e}".split("e")[1])) - label = f"{name} {rounded:,.0f}".replace(",", " ") + label = f"{rounded:,.0f} {name}".replace(",", " ") # va="center" centers the font bbox incl. descender space, which leaves # the glyphs slightly high; nudge down to optically center on the bar. y = bar.get_y() + bar.get_height() / 2 - 0.09 - if bar.get_width() > max(values) * 0.28: - # Long bar: white text inside, right-aligned at the bar end. - ax.text(bar.get_width() - max(values) * 0.012, y, label, - va="center", ha="right", color="white", fontsize=11) - else: - # Short bar: theme-neutral gray text just past the bar end. - ax.text(bar.get_width() + max(values) * 0.012, y, label, - va="center", color=OUTSIDE, fontsize=11) + # All labels after the bar: theme-neutral gray, number before name. + # The longest bar's label overflows the axes; bbox_inches="tight" + # below expands the canvas to include it, so no dead space remains. + ax.text(bar.get_width() + xmax * 0.012, y, label, + va="center", color=OUTSIDE, fontsize=11) -fig.tight_layout(pad=0.2) out = Path("docs/bench-speed.svg") out.parent.mkdir(exist_ok=True) -fig.savefig(out, transparent=True) +fig.savefig(out, transparent=True, bbox_inches="tight", pad_inches=0.02) print("wrote", out)