LeoVasanko/uarite
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit

Browse Source
This commit is contained in:
LeoVasanko
2026-09-05 00:36:02 +00:00
commit b9664aabe4
14 changed files with 19829 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+11
View File
@@ -0,0 +1,11 @@
# Python-generated files
__pycache__/
*.py[oc]
build/
dist/
wheels/
*.egg-info
.*
!.gitignore
*.lock
README.md
+119
View File
@@ -0,0 +1,119 @@
# uarite
User-Agent parsing in Python has a long lineage. ua-parser is the official Python implementation of the ua-parser project, built around uap-core: the regex database extracted from BrowserScope's original parser and shared by implementations in many languages. user-agents wraps ua-parser with higher-level device and capability detection; its last release was in 2020. user-agent-parser is a separate implementation first released in 2022 and substantially updated in 2026, taking its own approach rather than building on uap-core. None of the three has further dependencies, but the regex databases weigh something: ua-parser installs at about 499 KB (531 KB with user-agents on top), user-agent-parser at 166 KB.
```sh
uv add uarite
```
This module is another take on the same problem: a small, dependency-free, compact pure-Python parser — 29 KB installed. It returns structured classifications, but also the thing most applications eventually need: a short human-readable description.
It is particularly aimed at server-side analytics, where correctly recognizing crawlers and modern reduced User-Agents matters. It detects disguised crawlers, distinguishes AI/search/preview traffic, handles HarmonyOS and bots without calling them Android, resolves common device model codes and falls back to reasonable output even when all else fails.
## Usage
```python
from uarite import uaparse
r = uaparse("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36")
r.pretty # "Chrome/152 Windows"
r.kind # "browser"
r.bot # ""
r.url # ""
r = uaparse("Mozilla/5.0 (Linux; Android 13; Pixel 7) ... Chrome/134.0.6885.65 "
"Mobile Safari/537.36; compatible; facebookexternalhit/1.1; "
"+http://www.facebook.com/externalhit_uatext.php")
r.pretty # "Facebook"
r.kind # "preview"
r.bot # "Facebook"
r.url # "http://www.facebook.com/externalhit_uatext.php"
```
## Output
`uaparse(ua)` returns a frozen `UA` dataclass:
| Field | Content |
| -------- | ------------------------------------------------------------------------------------------------ |
| `pretty` | Compact display string (below); `""` for empty/missing UAs, the raw UA when unrecognized |
| `bot` | Crawler/previewer display name, or `""` |
| `kind` | `"browser"`, `"ai"`, `"search"`, `"preview"`, `"spider"`, or `""` (scripts/HTTP libraries) |
| `url` | The crawler's info URL (`+https://…` pointer), or `""`; not part of `pretty` — link it in the UI |
`kind` is `"browser"` for Mozilla-format UAs with no bot token, `"ai"` for training-data and AI-assistant fetchers (GPTBot, ClaudeBot, Google-Extended, ...), `"search"` for search-engine indexing (Googlebot, Bingbot, ...), `"preview"` for social link-preview fetchers (Facebook, WhatsApp, Slack, ...), `"spider"` for generic or unknown crawlers, and `""` for scripts and HTTP libraries.
`pretty` is intended to be shown directly:
- Desktop: `Chrome/152 Windows`, `Safari/18 macOS`
- iPhone/iPad: `iPhone iOS 17` — the device and iOS version, not Safari (the only browser iOS has)
- Android: `Chrome/118 Pixel 6`, or `Chrome/152 Android` when the device is unknown
- Crawlers: `GPTBot (AI)`, `Googlebot (search)`, `Facebook` — the kind suffix appears only where a provider runs crawlers of more than one kind; single-kind providers stay plain
- Scripts: `python-requests/2.32.5`, `pip/24.3.1 Linux`
Chrome's reduced Android UA reports the frozen values `Android 10; K`; neither is real device information, so uarite deliberately reports simply `Android`. HarmonyOS compatibility strings are similarly recognized before their misleading Android tokens.
## Performance
All compared parsers cache repeated User-Agents, making cache hits effectively free. The useful difference is therefore the first parse of a new string.
In our benchmarks, uncached uarite parses take roughly **3–7 µs**. user-agent-parser is in the same general range at **~7 µs**, while the pure-Python ua-parser/user-agents path takes roughly **130–250 µs**.
The cache strategies differ in ways that matter under adversarial traffic. uarite caches only browser/client results (1024-entry LRU): crawlers tend to be unique and would otherwise evict the repeating UAs where caching is useful. user-agent-parser's 512-entry LRU lets a bot storm evict browsers, and user-agents' 200-entry dict clears entirely when full.
## Accuracy
The main difference is not how many fields can be returned, but what the parser believes the User-Agent actually says.
For example, reduced Chrome does not really tell us that the device is named `K` or that it runs Android 10; an Android compatibility token does not make HarmonyOS Android; and a Facebook or Google crawler containing a plausible Chrome UA is still a crawler, not a Chrome visitor.
The table below compares representative results. uarite shows `r.pretty`; the ua-parser display strings are assembled from its structured output for comparison. user-agents is omitted: it shares the ua-parser backend and returns virtually identical data in a slightly different structure.
| Case | uarite¹ | ua-parser² |
| ---------------------------- | ------------------------- | --------------------------------------------- |
| Chrome, Windows | Chrome/152 Windows | Chrome/152 Windows |
| Safari, macOS | Safari/18 macOS | Safari/18 Mac OS X Mac |
| Opera, Linux | Opera/106 Linux | Opera/106 Linux |
| Chrome, Android (no model) | Chrome/152 Android | Chrome Mobile/152 Android K❌ |
| Chrome, Android (Pixel) | Chrome/118 Pixel 6 | Chrome Mobile/118 Android Pixel 6 |
| Edge, Android (model code) | Edge/110 Galaxy S7 | Edge Mobile/110 Android Samsung SM-G930P |
| Firefox, Android | Firefox/154 Android 15 | Firefox Mobile/154 Android Generic Smartphone |
| Safari, iPhone | iPhone iOS 17 | Mobile Safari/17 iOS iPhone |
| Huawei HarmonyOS phone | HuaweiBrowser/6 HarmonyOS | Huawei Browser/6 Android❌ Huawei Browser |
| GPTBot | GPTBot (AI) | GPTBot/1 Spider |
| Googlebot (disguised) | Googlebot (search) | Googlebot/2 Android❌ Spider |
| Facebook preview (disguised) | Facebook | FacebookBot/1 Android Pixel 7 ❌ |
| Meta crawler (disguised) | Meta | Chrome/145 Windows ❌ |
| WhatsApp preview | WhatsApp | WhatsApp/10 Spider |
| Bytespider | Bytespider | Bytespider/ Android❌ Generic Smartphone |
| BingPreview | BingPreview (preview) | BingPreview/1 Windows❌ Spider |
| AhrefsBot | AhrefsBot | AhrefsBot/7 Spider |
| python-requests | python-requests/2.32.5 | Python Requests/2 |
❌ marks an incorrect browser, OS, or device interpretation.
¹ `r.pretty` shown as is
² `{user_agent.family}/{user_agent.major} {os.family} {device.family}`
Measured on 100 current browser UAs, family / version / OS accuracy is 80% / 91% / 99% for both ua-parser and user-agents, and 90% / 90% / 100% for uarite — its nominal "misses" are the iPhone rows, where it reports `iPhone iOS 17` rather than Mobile Safari, a deliberate choice since Safari is the only browser iOS has. user-agent-parser is absent from the table: it detected under a third of the crawlers in the test below and crashed on five inputs, so a side-by-side formatting comparison adds little.
Crawler detection was also tested against 2163 real-world crawler UAs from [monperrus/crawler-user-agents](https://github.com/monperrus/crawler-user-agents):
| Parser | Detected |
| ----------------- | ------------------------ |
| ua-parser | 63.8% |
| user-agents | 60.1% |
| user-agent-parser | 31.9% (crashed on 5 UAs) |
| uarite | 79.9% |
The remaining uarite misses are mostly ancient tokenless crawler names and ordinary HTTP libraries, which are intentionally classified as scripts rather than bots.
## Design
uarite uses a small hand-maintained regex/rule database rather than the much larger uap-core dataset. Rules are kept simple enough to audit directly and ordered so that specific identities such as crawlers or HarmonyOS win over browser compatibility tokens. The generic tells are few: a product token containing bot/spider/crawl/scan/verify/check, or an info URL in the UA — real browsers never carry one.
Unknown UAs remain visible: `pretty` falls back to the original string rather than discarding them, and malformed input never raises.
Some information simply is not present in a User-Agent. Modern Brave is normally indistinguishable from Chrome without browser-side detection, and iPhone UAs do not contain the device model. uarite prefers an incomplete answer to an invented one.
pyproject.toml
+35
View File
@@ -0,0 +1,35 @@
[build-system]
requires = [
"hatchling>=1.25.0",
"hatch-vcs>=0.4.0",
]
build-backend = "hatchling.build"
[project]
name = "uarite"
dynamic = ["version"]
description = "Compact, human-readable User-Agent formatting and bot detection — no regex database, stdlib only"
authors = [
{ name = "Leo Vasanko" },
]
keywords = ["user-agent", "ua-parser"]
readme = "README.md"
requires-python = ">=3.10"
classifiers = [
"Intended Audience :: Developers",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3",
"Topic :: Internet :: WWW/HTTP",
"Topic :: Software Development :: Libraries :: Python Modules",
]
dependencies = []
[project.urls]
Repository = "https://git.zi.fi/LeoVasanko/uarite"
Issues = "https://github.com/LeoVasanko/uarite"
[tool.hatch.version]
source = "vcs"
[tool.hatch.build.targets.wheel]
packages = ["uarite"]
scripts/README.md
+30
View File
@@ -0,0 +1,30 @@
# Benchmark scripts
Recreate every statistic and table quoted in the top-level README.
`uarite` must be importable (e.g. `pip install -e .` or run with
`PYTHONPATH` pointing at the repo root); the reference parsers are
benchmark-only dependencies, pulled ad hoc via `uv run --with`.
- `download_data.py` — fetch the external datasets into `data/`:
the 100 modern browser UAs (top-user-agents npm package) and the
crawler corpus (monperrus/crawler-user-agents). `data/ua.txt` is a
committed real-traffic sample, not downloaded.
```
uv run scripts/download_data.py
```
- `prettytable.py` — prints the accuracy-comparison markdown table:
```
uv run --with ua-parser --with user-agents python scripts/prettytable.py
```
- `bench.py` — prints browser-accuracy counts, crawler-detection rates,
URL-extraction coverage, and the timing tables (unique UAs, realistic
mix, bot storm) with cache statistics:
```
uv run --with ua-parser --with user-agents --with user-agent-parser \
python scripts/bench.py
```
scripts/bench.py
+277
View File
@@ -0,0 +1,277 @@
"""Benchmark uarite vs ua-parser vs user-agents vs user-agent-parser.
Reproduces the README's numbers: browser accuracy on 100 modern UAs,
crawler detection on 2163 real-world crawler UAs, and timing (unique UAs,
a realistic repeat/unique mix, a pure bot storm) with cache introspection.
Data lives in scripts/data (see download_data.py). Requires uarite
(installed) plus the benchmark-only reference parsers:
uv run --with ua-parser --with user-agents --with user-agent-parser \
python scripts/bench.py
"""
import json
import random
import re
import timeit
from pathlib import Path
from ua_parser import parse as ua_parse
from user_agent_parser import parse as uap_parse
from user_agents import parse as uas_parse
from uarite import uaparse
from uarite.core import _parse_client
DATA = Path(__file__).parent / "data"
BROWSERS = json.loads((DATA / "top-user-agents.json").read_text())
CRAWLERS = json.loads((DATA / "crawler-user-agents.json").read_text())
OWN = (DATA / "ua.txt").read_text().splitlines()
CRAWLER_UAS = [ua for c in CRAWLERS for ua in (c.get("instances") or [c["pattern"]])]
def expected(ua):
"""De-facto ground truth for real-browser UAs: (family, major, os)."""
fam = major = ""
for tok, name in (
("EdgA", "Edge"),
("Edg", "Edge"),
("OPR", "Opera"),
("SamsungBrowser", "Samsung Internet"),
("Firefox", "Firefox"),
("Chrome", "Chrome"),
):
m = re.search(re.escape(tok) + r"/(\d+)", ua)
if m:
fam, major = name, m.group(1)
break
if not fam and "Safari/" in ua:
m = re.search(r"Version/(\d+)", ua)
fam, major = "Safari", m.group(1) if m else ""
if not fam and ("iPhone" in ua or "iPad" in ua):
fam = "Safari" # iOS webview UA: no browser token, Safari engine
if "iPhone" in ua or "iPad" in ua:
os = "ios"
elif re.search(r"Android [\d.]", ua):
os = "android"
elif "Windows NT" in ua:
os = "windows"
elif "Mac OS X" in ua:
os = "macos"
elif "Linux" in ua or "X11" in ua:
os = "linux"
else:
os = ""
return fam, major, os
def norm_os(s):
s = (s or "").lower().replace(" ", "").replace("_", "")
return {"macosx": "macos", "ubuntu": "linux"}.get(s, s)
def score_browsers():
res = {}
for name, fn in (
("ua-parser", ua_parse),
("user-agents", uas_parse),
("user-agent-parser", uap_parse),
):
fam_ok = ver_ok = os_ok = 0
for ua in BROWSERS:
efam, emaj, eos = expected(ua)
r = fn(ua)
if name == "ua-parser":
fam = r.user_agent.family or ""
maj = r.user_agent.major or ""
osf = r.os.family or ""
elif name == "user-agent-parser":
fam = r[0] or ""
maj = (r[1] or "").split(".")[0]
osf = r[2] or ""
else:
fam = r.browser.family or ""
maj = str(r.browser.version[0]) if r.browser.version else ""
osf = r.os.family or ""
fam = fam.split()[0]
fam_ok += efam.split()[0].lower() == fam.lower()
ver_ok += emaj == maj
os_ok += eos == norm_os(osf)
res[name] = (fam_ok, ver_ok, os_ok)
fam_ok = ver_ok = os_ok = 0
for ua in BROWSERS:
efam, emaj, eos = expected(ua)
r = uaparse(ua)
if emaj:
fam_ok += f"{efam}/{emaj}" in r.pretty
ver_ok += f"/{emaj}" in r.pretty
else:
# iOS webview: no version; "iPhone iOS 18" is the right answer
fam_ok += efam in r.pretty or "iPhone" in r.pretty or "iPad" in r.pretty
ver_ok += 1
oslabel = {
"ios": ("iPhone", "iPad"),
"macos": ("macOS",),
"windows": ("Windows",),
"linux": ("Linux",),
"android": ("Android",),
}.get(eos, ())
# Android with a known model drops the OS by design
model = re.search(r"Android [\d.]+; ([^;()]+?)(?:;|\))", ua)
has_model = eos == "android" and model and model.group(1) not in ("K", "Mobile")
os_ok += bool(any(os in r.pretty for os in oslabel) or has_model)
res["uarite"] = (fam_ok, ver_ok, os_ok)
return res
def score_crawlers():
out = {}
crashes = 0
for name in ("ua-parser", "user-agents", "user-agent-parser", "uarite"):
det = 0
for ua in CRAWLER_UAS:
try:
if name == "ua-parser":
r = ua_parse(ua)
fam = r.user_agent.family if r.user_agent else ""
bot = (r.device.family == "Spider" if r.device else False) or bool(
re.search(r"bot|spider|crawl", fam, re.I)
)
elif name == "user-agents":
bot = uas_parse(ua).is_bot
elif name == "user-agent-parser":
bot = uap_parse(ua)[4] == "Bot"
else:
bot = bool(uaparse(ua).bot)
except Exception:
crashes += name == "user-agent-parser"
continue
det += bot
out[name] = det
url_have = sum(1 for ua in CRAWLER_UAS if re.search(r"https?://", ua))
url_got = sum(1 for ua in CRAWLER_UAS if uaparse(ua).url)
return out, url_have, url_got, crashes
def realistic_mix():
"""Realistic traffic: browsers repeat (weighted — popular combos many
times), interleaved with a steady stream of mostly-unique bot UAs that
hammer every parser's cache."""
rng = random.Random(42)
browsers = []
for i in range(2000):
# zipf-ish: early (popular) UAs repeat heavily
browsers.append(BROWSERS[int(100 * (rng.random() ** 3))])
bots = list(CRAWLER_UAS)
rng.shuffle(bots)
bots = bots[:1000]
# synthetic cache-busters: unique disguised bot UAs
for i in range(1000):
bots.append(
f"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
f" (KHTML, like Gecko) Chrome/{100 + i % 50}.0.{i}.0 Safari/537.36;"
f" compatible; ScrapeBot{i}/1.{i}; +http://scrapebot{i}.example.com/"
)
mix = [None] * (len(browsers) + len(bots))
mix[::2] = browsers
mix[1::2] = bots
return mix
def bench_realistic():
mix = realistic_mix()
storm = [f"UniqueBot{i}/2.{i} (+http://ub{i}.example.com/)" for i in range(5000)]
print(
f"\n## realistic mix ({len(mix)} UAs: 2000 repeating browsers interleaved"
f" with 2000 mostly-unique bots)"
)
for name, fn in (
("ua-parser", ua_parse),
("user-agents", uas_parse),
("user-agent-parser", uap_parse),
("uarite", uaparse),
):
fn = safe(fn)
t = timeit.timeit(lambda: [fn(u) for u in mix], number=1)
info = cache_info(name)
print(f"{name:20} {t / len(mix) * 1e6:7.1f} µs/UA cache: {info}")
print(f"\n## pure bot storm ({len(storm)} unique UAs, zero cache value)")
for name, fn in (
("ua-parser", ua_parse),
("user-agents", uas_parse),
("user-agent-parser", uap_parse),
("uarite", uaparse),
):
fn = safe(fn)
t = timeit.timeit(lambda: [fn(u) for u in storm], number=1)
print(
f"{name:20} {t / len(storm) * 1e6:7.1f} µs/UA cache: {cache_info(name)}"
)
def safe(fn):
def f(u):
try:
return fn(u)
except Exception:
return None
return f
def cache_info(name):
if name == "uarite":
i = _parse_client.cache_info()
return f"{i.hits} hits / {i.misses} misses (cap 1024, browsers only)"
if name == "user-agent-parser":
from user_agent_parser.parser import _cached_parse_user_agent
i = _cached_parse_user_agent.cache_info()
return f"{i.hits} hits / {i.misses} misses (cap 512 LRU)"
if name == "user-agents":
from ua_parser.user_agent_parser import _PARSE_CACHE
return f"{len(_PARSE_CACHE)} entries (cap 200, CLEARS when full)"
if name == "ua-parser":
return "cap 2000 S3-FIFO (scan-resistant)"
return ""
def bench():
alluas = BROWSERS + CRAWLER_UAS + OWN
n = 3
res = {}
for name, fn in (
("ua-parser", ua_parse),
("user-agents", uas_parse),
("user-agent-parser", uap_parse),
("uarite", uaparse),
):
fn = safe(fn)
t = timeit.timeit(lambda: [fn(u) for u in alluas], number=n)
res[name] = t / n / len(alluas) * 1e6
# warm cache: repeat a small realistic working set many times
working = (BROWSERS + OWN[:50]) * 10
t = timeit.timeit(lambda: [uaparse(u) for u in working], number=n)
res["uarite (warm cache)"] = t / n / len(working) * 1e6
return res
if __name__ == "__main__":
print(f"## browser accuracy (n={len(BROWSERS)}): family / version / OS correct")
for k, (f, v, o) in score_browsers().items():
print(f"{k:14} {f:3}/100 {v:3}/100 {o:3}/100")
det, url_have, url_got, crashes = score_crawlers()
print(f"\n## crawler detection (n={len(CRAWLER_UAS)})")
for k, v in det.items():
print(f"{k:18} {v:5} ({v / len(CRAWLER_UAS):.1%})")
print(f"user-agent-parser crashed on {crashes} UAs")
print(f"\nuarite URL extraction: {url_got}/{url_have} of instances carrying a URL")
print(
"\n## speed (µs per parse, mixed set of %d UAs)"
% (len(BROWSERS) + len(CRAWLER_UAS) + len(OWN))
)
for k, v in bench().items():
print(f"{k:20} {v:8.1f}")
bench_realistic()
scripts/data/crawler-user-agents.json
+18391
View File
File diff suppressed because it is too large Load Diff
scripts/data/top-user-agents.json
+102
View File
@@ -0,0 +1,102 @@
[
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36",
"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Mobile Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36 Edg/151.0.0.0",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/126 Safari/537.36",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36",
"Mozilla/5.0 (iPhone; CPU iPhone OS 18_7 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.6 Mobile/15E148 Safari/604.1",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:154.0) Gecko/20100101 Firefox/154.0",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36 OPR/134.0.0.0",
"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Mobile Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36 OPR/134.0.0.0",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.5.2 Safari/605.1.15",
"Mozilla/5.0 (iPhone; CPU iPhone OS 18_7 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.5.2 Mobile/15E148 Safari/604.1",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36",
"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/30.0 Chrome/143.0.0.0 Mobile Safari/537.36",
"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Mobile Safari/537.36",
"Mozilla/5.0 (iPhone; CPU iPhone OS 18_7 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.6.1 Mobile/15E148 Safari/604.1",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36 Edg/152.0.0.0",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:154.0) Gecko/20100101 Firefox/154.0",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36",
"Mozilla/5.0 (iPhone; CPU iPhone OS 26_6_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/151.0.7922.112 Mobile/15E148 Safari/604.1",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36",
"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36",
"Mozilla/5.0 (iPhone; CPU iPhone OS 18_7 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.5 Mobile/15E148 Safari/604.1",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.6 Safari/605.1.15",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.6.2 Safari/605.1.15",
"Mozilla/5.0 (iPhone; CPU iPhone OS 18_7 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.5 Safari/605.1.15",
"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Mobile Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.7922.34 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) obsidian/1.13.4 Chrome/150.0.7871.114 Electron/43.1.1 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36",
"Mozilla/5.0 (iPhone; CPU iPhone OS 18_7 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.7.5 Mobile/15E148 Safari/604.1",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36",
"Mozilla/5.0 (X11; Linux x86_64; rv:154.0) Gecko/20100101 Firefox/154.0",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Mobile Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:153.0) Gecko/20100101 Firefox/153.0",
"Mozilla/5.0 (iPhone; CPU iPhone OS 18_7 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/27.0 Mobile/15E148 Safari/604.1",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36",
"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Mobile Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36 Edg/151.0.0.0",
"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36",
"Mozilla/5.0 (Linux; Android 16; SM-S928B Build/BP4A.251205.006; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/151.0.7922.199 Mobile Safari/537.36 DibburitAPK/1.6",
"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36",
"Mozilla/5.0 (iPhone; CPU iPhone OS 18_7 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.3 Mobile/15E148 Safari/604.1",
"Mozilla/5.0 (X11; CrOS x86_64 14541.0.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.6 Safari/605.1.15",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36",
"Mozilla/5.0 (iPhone; CPU iPhone OS 18_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.5 Mobile/15E148 Safari/604.1",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.2 Safari/605.1.15",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.5938.132 Safari/537.36",
"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Mobile Safari/537.36",
"Mozilla/5.0 (iPhone; CPU iPhone OS 26_5_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/151.0.7922.112 Mobile/15E148 Safari/604.1",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36",
"Mozilla/5.0 (iPhone; CPU iPhone OS 18_7 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.4 Mobile/15E148 Safari/604.1",
"Mozilla/5.0 (iPhone; CPU iPhone OS 18_7 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.2 Mobile/15E148 Safari/604.1",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:153.0) Gecko/20100101 Firefox/153.0",
"Mozilla/5.0 (iPhone; CPU iPhone OS 26_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/23G71 Twitter for iPhone/12.19.1",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36 Edg/150.0.0.0",
"Mozilla/5.0 (Linux; Android 12; Pixel 6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Mobile Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/121.0",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/27.0 Safari/605.1.15",
"Mozilla/5.0 (iPhone; CPU iPhone OS 26_6_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/152.0.7977.64 Mobile/15E148 Safari/604.1",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36",
"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Mobile Safari/537.36 EdgA/151.0.0.0",
"Mozilla/5.0 (X11; Linux x86_64; rv:153.0) Gecko/20100101 Firefox/153.0",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.6422.60 Safari/537.36 Edge/12.246",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 13_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) ScalboostBrowser/4.6.0 Chrome/144.0.7559.177 Electron/40.6.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.7444.175 Safari/537.36",
"Mozilla/5.0 (iPhone; CPU iPhone OS 26_3_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/144.0.7559.95 Mobile/15E148 Safari/604.1",
"Mozilla/5.0 (Android 16; Mobile; rv:154.0) Gecko/154.0 Firefox/154.0"
]
scripts/data/ua.txt
+316
View File
@@ -0,0 +1,316 @@
ALittle Client
Go-http-client/1.1
Go-http-client/2.0
Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Mobile Safari/537.36
Mozilla/5.0 (Linux; Android 12; Pixel 6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.6677.221 Mobile Safari/537.36; compatible; GrokBot/1.0; +https://x.ai/grokbot
Mozilla/5.0 (Linux; Android 12; Pixel 6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.7874.237 Mobile Safari/537.36; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot
Mozilla/5.0 (Linux; Android 12; Pixel 6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.5636.120 Mobile Safari/537.36; compatible; Discordbot/2.0; +https://discordapp.com
Mozilla/5.0 (Linux; Android 12; Pixel 6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.5904.94 Mobile Safari/537.36; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user
Mozilla/5.0 (Linux; Android 12; Pixel 6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.6999.179 Mobile Safari/537.36; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user
Mozilla/5.0 (Linux; Android 12; Pixel 6) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Extended/1.0; +http://www.google.com/bot.html) Chrome/136.0.3622.183 Mobile Safari/537.36
Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.1714.57 Mobile Safari/537.36; compatible; Discordbot/2.0; +https://discordapp.com
Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.2331.116 Mobile Safari/537.36; compatible; Twitterbot/1.0
Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.3871.45 Mobile Safari/537.36; compatible; Bytespider; +https://zhanzhang.toutiao.com/
Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.6885.65 Mobile Safari/537.36; compatible; facebookexternalhit/1.1; +http://www.facebook.com/externalhit_uatext.php
Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.179.134 Mobile Safari/537.36; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user
Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.304.223 Mobile Safari/537.36; compatible; GPTBot/1.4; +https://openai.com/gptbot
Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.2633.116 Mobile Safari/537.36; compatible; Google-Extended/1.0; +http://www.google.com/bot.html
Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.3245.171 Mobile Safari/537.36; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot
Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.675.120 Mobile Safari/537.36; compatible; ClaudeBot/1.0; +claudebot@anthropic.com
Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user) Chrome/136.0.5265.172 Mobile Safari/537.36
Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) Chrome/109.0.142.149 Mobile Safari/537.36
Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko; compatible; WhatsApp/10.0.2.1) Chrome/147.0.8581.235 Mobile Safari/537.36
Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.8358.226 Mobile Safari/537.36; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user
Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.2863.23 Mobile Safari/537.36; compatible; GrokBot/1.0; +https://x.ai/grokbot
Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.581.205 Mobile Safari/537.36; compatible; Claude-User/1.0; +Claude-User@anthropic.com
Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.7679.116 Mobile Safari/537.36; compatible; Claude-User/1.0; +Claude-User@anthropic.com
Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.4412.199 Mobile Safari/537.36; compatible; TelegramBot/1.0
Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.1359.77 Mobile Safari/537.36; compatible; ChatGPT-User/1.0; +https://openai.com/bot
Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.3533.159 Mobile Safari/537.36; compatible; Twitterbot/1.0
Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.1665.247 Mobile Safari/537.36; compatible; Google-Extended/1.0; +http://www.google.com/bot.html
Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/148.0.5629.104 Mobile Safari/537.36
Mozilla/5.0 (Linux; Android 15; SM-G930P; Build/AP4A.190211.226) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.1587.89 Mobile Safari/537.36 EdgA/110.0.1587.89
Mozilla/5.0 (Linux; Android 15; SM-S918B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.6752.19 Mobile Safari/537.36; compatible; Twitterbot/1.0
Mozilla/5.0 (Linux; Android 15; SM-S918B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.456.90 Mobile Safari/537.36; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot
Mozilla/5.0 (Linux; Android 15; SM-S918B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.562.77 Mobile Safari/537.36; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot
Mozilla/5.0 (Linux; Android 15; SM-S918B) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Applebot/0.1; +http://www.apple.com/go/applebot) Chrome/130.0.4812.112 Mobile Safari/537.36
Mozilla/5.0 (Linux; Android 15; SM-S918B) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user) Chrome/147.0.2792.237 Mobile Safari/537.36
Mozilla/5.0 (Linux; Android 15; SM-S918B) AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) Chrome/130.0.6971.196 Mobile Safari/537.36
Mozilla/5.0 (Linux; Android 15; SM-S918B) AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot) Chrome/130.0.6187.117 Mobile Safari/537.36
Mozilla/5.0 (Linux; Android 15; SM-S918B) AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot) Chrome/147.0.8778.3 Mobile Safari/537.36
Mozilla/5.0 (Linux; Android 15; SM-S918B) AppleWebKit/537.36 (KHTML, like Gecko; compatible; GrokBot/1.0; +https://x.ai/grokbot) Chrome/128.0.7214.102 Mobile Safari/537.36
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.7922.173 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.12) Gecko/20100101 Firefox/121.12; compatible; GPTBot/1.4; +https://openai.com/gptbot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:127.15) Gecko/20100101 Firefox/127.15; compatible; Bytespider; +https://zhanzhang.toutiao.com/
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.7) Gecko/20100101 Firefox/133.7; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:149.4) Gecko/20100101 Firefox/149.4; compatible; ClaudeBot/1.0; +claudebot@anthropic.com
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.6) Gecko/20100101 Firefox/78.6; compatible; LinkedInBot/1.0; +http://www.linkedin.com
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.1781.181 Safari/537.36; compatible; meta-externalagent/1.1; +https://developers.facebook.com/docs/sharing/webmasters/crawler
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.7456.176 Safari/537.36 Edg/114.0.7456.176; compatible; Google-Extended/1.0; +http://www.google.com/bot.html
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.4134.99 Safari/537.36; compatible; Applebot/0.1; +http://www.apple.com/go/applebot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.6701.62 Safari/537.36; compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.7297.17 Safari/537.36; compatible; Slackbot-LinkExpanding/1.0; +https://api.slack.com/robots
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36 CloudflareWebScanner/1.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.4253.212 Safari/537.36; compatible; Discordbot/2.0; +https://discordapp.com
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.393.116 Safari/537.36; compatible; GPTBot/1.2; +https://openai.com/gptbot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/1.0; +https://developer.amazon.com/support/amazonbot) Chrome/131.0.5818.195 Safari/537.36 Edg/131.0.5818.195
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Applebot/0.1; +http://www.apple.com/go/applebot) Chrome/128.0.2374.13 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Bytespider; +https://zhanzhang.toutiao.com/) Chrome/122.0.8582.72 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Extended/1.0; +http://www.google.com/bot.html) Chrome/122.0.6224.91 Safari/537.36 Edg/122.0.6224.91
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot) Chrome/85.0.796.31 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) Chrome/147.0.9065.204 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko; compatible; TelegramBot/1.0) Chrome/148.0.2316.113 Safari/537.36 Edg/148.0.2316.113
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Safari/605.1.15; compatible; meta-externalagent/1.1; +https://developers.facebook.com/docs/sharing/webmasters/crawler
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot
Mozilla/5.0 (Macintosh; Intel Mac OS X 13_6_1; rv:119.0) Gecko/20100101 Firefox/119.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 13_6_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.5 Safari/605.1.15; compatible; Bytespider; +https://zhanzhang.toutiao.com/
Mozilla/5.0 (Macintosh; Intel Mac OS X 13_6_7) AppleWebKit/605.1.15 (KHTML, like Gecko; compatible; Amzn-SearchBot/1.0; +https://developer.amazon.com/support/amazonbot) Version/19.1 Safari/605.1.15
Mozilla/5.0 (Macintosh; Intel Mac OS X 13_6_7) AppleWebKit/605.1.15 (KHTML, like Gecko; compatible; LinkedInBot/1.0; +http://www.linkedin.com) Version/18.1 Safari/605.1.15
Mozilla/5.0 (Macintosh; Intel Mac OS X 14.5; rv:127.8) Gecko/20100101 Firefox/127.8; compatible; TelegramBot/1.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 14.5; rv:132.11) Gecko/20100101 Firefox/132.11; compatible; Twitterbot/1.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 14.5; rv:133.5) Gecko/20100101 Firefox/133.5; compatible; Claude-User/1.0; +Claude-User@anthropic.com
Mozilla/5.0 (Macintosh; Intel Mac OS X 14.5; rv:149.5) Gecko/20100101 Firefox/149.5; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot
Mozilla/5.0 (Macintosh; Intel Mac OS X 14.5; rv:78.16) Gecko/20100101 Firefox/78.16; compatible; ChatGPT-User/1.0; +https://openai.com/bot
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.9070.130 Safari/537.36 Edg/128.0.9070.130; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.610.178 Safari/537.36; compatible; WhatsApp/10.0.2.1
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot) Chrome/151.0.779.246 Safari/537.36 Edg/151.0.779.246
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/537.36 (KHTML, like Gecko; compatible; TelegramBot/1.0) Chrome/133.0.153.92 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/537.36 (KHTML, like Gecko; compatible; WhatsApp/10.0.2.1) Chrome/131.0.7140.164 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/537.36 (KHTML, like Gecko; compatible; meta-externalagent/1.1; +https://developers.facebook.com/docs/sharing/webmasters/crawler) Chrome/118.0.8850.224 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.3 Safari/605.1.15; compatible; meta-externalagent/1.1; +https://developers.facebook.com/docs/sharing/webmasters/crawler
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.0 Safari/605.1.15; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.3295.40 Safari/537.36; compatible; GPTBot/1.4; +https://openai.com/gptbot
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.1535.25 Safari/537.36; compatible; Discordbot/2.0; +https://discordapp.com
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.6540.70 Safari/537.36; compatible; Google-Extended/1.0; +http://www.google.com/bot.html
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6_1) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/126.0.8218.208 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6_1) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user) Chrome/128.0.3723.179 Safari/537.36 Edg/128.0.3723.179
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6_1) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Extended/1.0; +http://www.google.com/bot.html) Chrome/133.0.4354.13 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6_1) AppleWebKit/537.36 (KHTML, like Gecko; compatible; LinkedInBot/1.0; +http://www.linkedin.com) Chrome/131.0.7882.202 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6_1) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) Chrome/131.0.2380.151 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6_1) AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) Chrome/148.0.6827.81 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6_1) AppleWebKit/605.1.15 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) Version/18.3 Safari/605.1.15
Mozilla/5.0 (Macintosh; Intel Mac OS X 15_0) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.7 Safari/605.1.15; compatible; Applebot/0.1; +http://www.apple.com/go/applebot
Mozilla/5.0 (Phone; OpenHarmony 6.1; Android 10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36 ArkWeb/6.1.0.120 Mobile HuaweiBrowser/6.1.6.310
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.6691.81 Safari/537.36; compatible; TelegramBot/1.0
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.4841.30 Safari/537.36; compatible; Claude-User/1.0; +Claude-User@anthropic.com
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.7168.194 Safari/537.36; compatible; GrokBot/1.0; +https://x.ai/grokbot
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.3
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.2240.231 Safari/537.36; compatible; Applebot/0.1; +http://www.apple.com/go/applebot
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.7895.118 Safari/537.36; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.1055.174 Safari/537.36 Edg/133.0.1055.174; compatible; Twitterbot/1.0
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.2238.87 Safari/537.36 Edg/133.0.2238.87; compatible; Slackbot-LinkExpanding/1.0; +https://api.slack.com/robots
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.3329.205 Safari/537.36; compatible; GrokBot/1.0; +https://x.ai/grokbot
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.7922.34 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.1090.96 Safari/537.36; compatible; Google-Extended/1.0; +http://www.google.com/bot.html
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.7074.69 Safari/537.36; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user) Chrome/114.0.3989.223 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) Chrome/151.0.8801.61 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot) Chrome/124.0.6057.104 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot) Chrome/130.0.4938.98 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Slackbot-LinkExpanding/1.0; +https://api.slack.com/robots) Chrome/134.0.4631.6 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; TelegramBot/1.0) Chrome/114.0.1358.24 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; WhatsApp/10.0.2.1) Chrome/133.0.3192.3 Safari/537.36
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.5) Gecko/20100101 Firefox/105.5; compatible; ChatGPT-User/1.0; +https://openai.com/bot
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0; compatible; GPTBot/1.2; +https://openai.com/gptbot
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.14) Gecko/20100101 Firefox/121.14; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:127.13) Gecko/20100101 Firefox/127.13; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:127.5) Gecko/20100101 Firefox/127.5; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:132.0) Gecko/20100101 Firefox/132.0
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:132.2) Gecko/20100101 Firefox/132.2; compatible; facebookexternalhit/1.1; +http://www.facebook.com/externalhit_uatext.php
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:133.1) Gecko/20100101 Firefox/133.1; compatible; GrokBot/1.0; +https://x.ai/grokbot
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:133.14) Gecko/20100101 Firefox/133.14; compatible; ClaudeBot/1.0; +claudebot@anthropic.com
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:143.13) Gecko/20100101 Firefox/143.13; compatible; GrokBot/1.0; +https://x.ai/grokbot
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:149.11) Gecko/20100101 Firefox/149.11; compatible; GrokBot/1.0; +https://x.ai/grokbot
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:154.0) Gecko/20100101 Firefox/154.0
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.4) Gecko/20100101 Firefox/78.4; compatible; meta-externalagent/1.1; +https://developers.facebook.com/docs/sharing/webmasters/crawler
Mozilla/5.0 (Windows NT 10.0; rv:105.8) Gecko/20100101 Firefox/105.8; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user
Mozilla/5.0 (Windows NT 10.0; rv:121.4) Gecko/20100101 Firefox/121.4; compatible; Twitterbot/1.0
Mozilla/5.0 (Windows NT 10.0; rv:132.17) Gecko/20100101 Firefox/132.17; compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot
Mozilla/5.0 (Windows NT 10.0; rv:143.9) Gecko/20100101 Firefox/143.9; compatible; Applebot/0.1; +http://www.apple.com/go/applebot
Mozilla/5.0 (Windows NT 10.0; rv:149.2) Gecko/20100101 Firefox/149.2; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user
Mozilla/5.0 (Windows NT 10.0; rv:150.11) Gecko/20100101 Firefox/150.11; compatible; Applebot/0.1; +http://www.apple.com/go/applebot
Mozilla/5.0 (Windows NT 11.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36
Mozilla/5.0 (Windows NT 11.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.2208.117 Safari/537.36; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot
Mozilla/5.0 (Windows NT 11.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.4903.6 Safari/537.36; compatible; ChatGPT-User/1.0; +https://openai.com/bot
Mozilla/5.0 (Windows NT 11.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.2677.151 Safari/537.36; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot
Mozilla/5.0 (Windows NT 11.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/1.0; +https://developer.amazon.com/support/amazonbot) Chrome/109.0.1953.144 Safari/537.36
Mozilla/5.0 (Windows NT 11.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Bytespider; +https://zhanzhang.toutiao.com/) Chrome/134.0.7068.89 Safari/537.36
Mozilla/5.0 (Windows NT 11.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot) Chrome/133.0.8680.93 Safari/537.36
Mozilla/5.0 (Windows NT 11.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user) Chrome/124.0.1422.145 Safari/537.36
Mozilla/5.0 (Windows NT 11.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) Chrome/122.0.8976.2 Safari/537.36
Mozilla/5.0 (Windows NT 11.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) Chrome/151.0.7263.231 Safari/537.36
Mozilla/5.0 (Windows NT 11.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot) Chrome/136.0.6556.236 Safari/537.36
Mozilla/5.0 (Windows NT 11.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; GrokBot/1.0; +https://x.ai/grokbot) Chrome/91.0.2098.213 Safari/537.36
Mozilla/5.0 (Windows NT 11.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; WhatsApp/10.0.2.1) Chrome/91.0.9073.141 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.8349.207 Safari/537.36; compatible; Amzn-SearchBot/1.0; +https://developer.amazon.com/support/amazonbot
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.870.128 Safari/537.36; compatible; Discordbot/2.0; +https://discordapp.com
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.2718.190 Safari/537.36 Edg/114.0.2718.190; compatible; facebookexternalhit/1.1; +http://www.facebook.com/externalhit_uatext.php
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.673.111 Safari/537.36; compatible; Google-Extended/1.0; +http://www.google.com/bot.html
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.8103.201 Safari/537.36 Edg/122.0.8103.201; compatible; GPTBot/1.4; +https://openai.com/gptbot
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.6793.243 Safari/537.36; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6864.83 Safari/537.36; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.671.204 Safari/537.36; compatible; LinkedInBot/1.0; +http://www.linkedin.com
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.2787.208 Safari/537.36; compatible; GPTBot/1.4; +https://openai.com/gptbot
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.763.59 Safari/537.36; compatible; Discordbot/2.0; +https://discordapp.com
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/133.0.7999.49 Safari/537.36 Edg/133.0.7999.49
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Bytespider; +https://zhanzhang.toutiao.com/) Chrome/124.0.3933.247 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot) Chrome/109.0.3436.159 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot) Chrome/134.0.8299.205 Safari/537.36 Edg/134.0.8299.205
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com) Chrome/120.0.164.234 Safari/537.36 Edg/120.0.164.234
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user) Chrome/131.0.4832.111 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Extended/1.0; +http://www.google.com/bot.html) Chrome/122.0.2110.32 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; GrokBot/1.0; +https://x.ai/grokbot) Chrome/148.0.7521.199 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot) Chrome/114.0.80.121 Safari/537.36
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Slackbot-LinkExpanding/1.0; +https://api.slack.com/robots) Chrome/118.0.7057.134 Safari/537.36 Edg/118.0.7057.134
Mozilla/5.0 (X11; Linux x86_64; rv:121.16) Gecko/20100101 Firefox/121.16; compatible; Slackbot-LinkExpanding/1.0; +https://api.slack.com/robots
Mozilla/5.0 (X11; Linux x86_64; rv:121.9) Gecko/20100101 Firefox/121.9; compatible; GrokBot/1.0; +https://x.ai/grokbot
Mozilla/5.0 (X11; Linux x86_64; rv:127.8) Gecko/20100101 Firefox/127.8; compatible; Google-Extended/1.0; +http://www.google.com/bot.html
Mozilla/5.0 (X11; Linux x86_64; rv:133.16) Gecko/20100101 Firefox/133.16; compatible; Google-Extended/1.0; +http://www.google.com/bot.html
Mozilla/5.0 (X11; Linux x86_64; rv:149.5) Gecko/20100101 Firefox/149.5; compatible; ChatGPT-User/1.0; +https://openai.com/bot
Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.8500.64 Safari/537.36; compatible; Bytespider; +https://zhanzhang.toutiao.com/
Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.8365.158 Safari/537.36 Edg/134.0.8365.158; compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot
Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/1.0; +https://developer.amazon.com/support/amazonbot) Chrome/128.0.4931.11 Safari/537.36
Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/1.0; +https://developer.amazon.com/support/amazonbot) Chrome/147.0.8835.247 Safari/537.36
Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Bytespider; +https://zhanzhang.toutiao.com/) Chrome/136.0.1808.113 Safari/537.36
Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot) Chrome/136.0.3467.118 Safari/537.36
Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; GrokBot/1.0; +https://x.ai/grokbot) Chrome/131.0.7648.87 Safari/537.36
Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) Chrome/133.0.1362.30 Safari/537.36
Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; WhatsApp/10.0.2.1) Chrome/120.0.4740.223 Safari/537.36 Edg/120.0.4740.223
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:105.13) Gecko/20100101 Firefox/105.13; compatible; GPTBot/1.4; +https://openai.com/gptbot
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:121.5) Gecko/20100101 Firefox/121.5; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.11) Gecko/20100101 Firefox/127.11; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.17) Gecko/20100101 Firefox/127.17; compatible; meta-externalagent/1.1; +https://developers.facebook.com/docs/sharing/webmasters/crawler
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:132.7) Gecko/20100101 Firefox/132.7; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:133.5) Gecko/20100101 Firefox/133.5; compatible; Slackbot-LinkExpanding/1.0; +https://api.slack.com/robots
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:149.0) Gecko/20100101 Firefox/149.0; compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:149.6) Gecko/20100101 Firefox/149.6; compatible; LinkedInBot/1.0; +http://www.linkedin.com
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:149.7) Gecko/20100101 Firefox/149.7; compatible; GPTBot/1.2; +https://openai.com/gptbot
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:150.1) Gecko/20100101 Firefox/150.1; compatible; LinkedInBot/1.0; +http://www.linkedin.com
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:78.8) Gecko/20100101 Firefox/78.8; compatible; Claude-User/1.0; +Claude-User@anthropic.com
Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)
Mozilla/5.0 (compatible; Amzn-SearchBot/1.0; +https://developer.amazon.com/support/amazonbot)
Mozilla/5.0 (compatible; Applebot/0.1; +http://www.apple.com/go/applebot)
Mozilla/5.0 (compatible; Bytespider; +https://zhanzhang.toutiao.com/)
Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot)
Mozilla/5.0 (compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot)
Mozilla/5.0 (compatible; Claude-User/1.0; +Claude-User@anthropic.com)
Mozilla/5.0 (compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user)
Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
Mozilla/5.0 (compatible; Discordbot/2.0; +https://discordapp.com)
Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)
Mozilla/5.0 (compatible; GPTBot/1.4; +https://openai.com/gptbot)
Mozilla/5.0 (compatible; Google-Extended/1.0; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; GrokBot/1.0; +https://x.ai/grokbot)
Mozilla/5.0 (compatible; LinkedInBot/1.0; +http://www.linkedin.com)
Mozilla/5.0 (compatible; MJ12bot/v2.0.5; http://mj12bot.com/)
Mozilla/5.0 (compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot)
Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Mozilla/5.0 (compatible; Reflectionbot/1.0; +https://reflection.ai/bot)
Mozilla/5.0 (compatible; Slackbot-LinkExpanding/1.0; +https://api.slack.com/robots)
Mozilla/5.0 (compatible; TelegramBot/1.0)
Mozilla/5.0 (compatible; Twitterbot/1.0)
Mozilla/5.0 (compatible; WP-Safe-Scanner/1.0)
Mozilla/5.0 (compatible; WhatsApp/10.0.2.1)
Mozilla/5.0 (compatible; facebookexternalhit/1.1; +http://www.facebook.com/externalhit_uatext.php)
Mozilla/5.0 (compatible; meta-externalagent/1.1; +https://developers.facebook.com/docs/sharing/webmasters/crawler)
Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1
Mozilla/5.0 (iPhone; CPU iPhone OS 16_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.3 Mobile/15E148 Safari/604.1; compatible; Claude-User/1.0; +Claude-User@anthropic.com
Mozilla/5.0 (iPhone; CPU iPhone OS 16_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/19.7 Mobile/15E148 Safari/604.1; compatible; Google-Extended/1.0; +http://www.google.com/bot.html
Mozilla/5.0 (iPhone; CPU iPhone OS 16_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot) Version/19.1 Mobile/15E148 Safari/604.1
Mozilla/5.0 (iPhone; CPU iPhone OS 16_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko; compatible; Google-Extended/1.0; +http://www.google.com/bot.html) Version/16.6 Mobile/15E148 Safari/604.1
Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.1229.179 Mobile Safari/537.36; compatible; Bytespider; +https://zhanzhang.toutiao.com/
Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.903.10 Mobile Safari/537.36; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user
Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.4431.64 Mobile Safari/537.36; compatible; GPTBot/1.2; +https://openai.com/gptbot
Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.4849.17 Mobile Safari/537.36; compatible; TelegramBot/1.0
Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.672.19 Mobile Safari/537.36; compatible; Google-Extended/1.0; +http://www.google.com/bot.html
Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.790.154 Mobile Safari/537.36; compatible; LinkedInBot/1.0; +http://www.linkedin.com
Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com) Chrome/151.0.5552.152 Mobile Safari/537.36
Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Discordbot/2.0; +https://discordapp.com) Chrome/131.0.8856.25 Mobile Safari/537.36
Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Mobile/15E148 Safari/604.1
Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot) Version/16.5 Mobile/15E148 Safari/604.1
Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko; compatible; TelegramBot/1.0) Version/18.6 Mobile/15E148 Safari/604.1
Mozilla/5.0 (iPhone; CPU iPhone OS 18_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Mobile/15E148 Safari/604.1; compatible; Claude-User/1.0; +Claude-User@anthropic.com
Mozilla/5.0 (iPhone; CPU iPhone OS 18_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/19.0 Mobile/15E148 Safari/604.1; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot
Mozilla/5.0 (iPhone; CPU iPhone OS 18_4 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.8733.214 Mobile Safari/537.36; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user
Mozilla/5.0 (iPhone; CPU iPhone OS 18_4 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.7561.171 Mobile Safari/537.36; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot
Mozilla/5.0 (iPhone; CPU iPhone OS 18_4 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Bytespider; +https://zhanzhang.toutiao.com/) Chrome/128.0.6985.201 Mobile Safari/537.36
Mozilla/5.0 (iPhone; CPU iPhone OS 18_4 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot) Chrome/131.0.3919.233 Mobile Safari/537.36
Mozilla/5.0 (iPhone; CPU iPhone OS 18_4 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) Chrome/118.0.5358.73 Mobile Safari/537.36
Mozilla/5.0 (iPhone; CPU iPhone OS 18_4 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; compatible; meta-externalagent/1.1; +https://developers.facebook.com/docs/sharing/webmasters/crawler) Chrome/122.0.3316.88 Mobile Safari/537.36
Mozilla/5.0 (iPhone; CPU iPhone OS 18_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Mobile/15E148 Safari/604.1; compatible; TelegramBot/1.0
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Amzn-SearchBot/1.0; +https://developer.amazon.com/support/amazonbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Applebot/0.1; +http://www.apple.com/go/applebot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Claude-User/1.0; +Claude-User@anthropic.com
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; +claudebot@anthropic.com
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Discordbot/2.0; +https://discordapp.com
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Google-Extended/1.0; +http://www.google.com/bot.html
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GrokBot/1.0; +https://x.ai/grokbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; LinkedInBot/1.0; +http://www.linkedin.com
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Slackbot-LinkExpanding/1.0; +https://api.slack.com/robots
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; TelegramBot/1.0
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Twitterbot/1.0
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; WhatsApp/10.0.2.1
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/1.0; +https://developer.amazon.com/support/amazonbot)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Applebot/0.1; +http://www.apple.com/go/applebot)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Bytespider; +https://zhanzhang.toutiao.com/)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Discordbot/2.0; +https://discordapp.com)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Extended/1.0; +http://www.google.com/bot.html)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GrokBot/1.0; +https://x.ai/grokbot)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; LinkedInBot/1.0; +http://www.linkedin.com)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Slackbot-LinkExpanding/1.0; +https://api.slack.com/robots)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; TelegramBot/1.0)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Twitterbot/1.0)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; WhatsApp/10.0.2.1)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/136.0.0.0 Safari/537.36
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/136.0.0.0 Safari/537.36 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; facebookexternalhit/1.1; +http://www.facebook.com/externalhit_uatext.php)
crusader-worker/1.0
null
pip/24.3.1 {"ci":null,"cpu":"x86_64","implementation":{"name":"CPython","version":"3.11.9"},"installer":{"name":"pip","version":"24.3.1"},"openssl_version":"OpenSSL 3.0.13 30 Jan 2024","python":"3.11.9","setuptools_version":"75.6.0","system":{"name":"Linux","release":"6.8.0"}}
python-httpx/0.28.1
python-requests/2.25.1
python-requests/2.32.5
sibsweb-verifier/0.1 (+mailto:ptwebcrawler@techremotehub.com)
scripts/download_data.py
+52
View File
@@ -0,0 +1,52 @@
"""Download the external datasets used by the benchmark scripts.
- data/top-user-agents.json — 100 current real-browser UAs, from the
top-user-agents npm package (src/index.json inside the tarball).
- data/crawler-user-agents.json — crawler patterns with sample UAs, from
github.com/monperrus/crawler-user-agents.
data/ua.txt is not downloaded: it is a real traffic sample (unique
User-Agent strings seen by one site), committed alongside this script.
"""
import json
import tarfile
import urllib.request
from io import BytesIO
from pathlib import Path
DATA = Path(__file__).parent / "data"
CRAWLERS_URL = (
"https://raw.githubusercontent.com/monperrus/crawler-user-agents/"
"master/crawler-user-agents.json"
)
NPM_REGISTRY = "https://registry.npmjs.org/top-user-agents/latest"
def fetch(url: str) -> bytes:
with urllib.request.urlopen(url, timeout=120) as r:
return r.read()
def main() -> None:
DATA.mkdir(exist_ok=True)
crawlers = DATA / "crawler-user-agents.json"
print(f"downloading {CRAWLERS_URL}")
crawlers.write_bytes(fetch(CRAWLERS_URL))
print("resolving latest top-user-agents from the npm registry")
meta = json.loads(fetch(NPM_REGISTRY))
tarball = fetch(meta["dist"]["tarball"])
with tarfile.open(fileobj=BytesIO(tarball), mode="r:gz") as tar:
top = tar.extractfile("package/src/index.json").read()
(DATA / "top-user-agents.json").write_bytes(top)
for f in ("crawler-user-agents.json", "top-user-agents.json"):
n = len(json.loads((DATA / f).read_text()))
print(f"{f}: {n} entries")
if __name__ == "__main__":
main()
scripts/prettytable.py
+104
View File
@@ -0,0 +1,104 @@
"""Print the README's accuracy-comparison table as markdown.
uarite's column is simply `r.pretty`. The reference modules have no
display format; their columns use one plain format string each on their
structured output (footnotes ²³ in the README). Requires uarite (installed)
plus the benchmark-only reference parsers:
uv run --with ua-parser --with user-agents python scripts/prettytable.py
"""
from ua_parser import parse as ua_parse
from user_agents import parse as uas_parse
from uarite import uaparse
CASES = [
(
"Chrome, Windows",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36",
),
(
"Safari, macOS",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.5 Safari/605.1.15",
),
(
"Opera, Linux",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 OPR/106.0.0.0",
),
(
"Chrome, Android (no model)",
"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Mobile Safari/537.36",
),
(
"Chrome, Android (Pixel)",
"Mozilla/5.0 (Linux; Android 12; Pixel 6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Mobile Safari/537.36",
),
(
"Edge, Android (model code)",
"Mozilla/5.0 (Linux; Android 15; SM-G930P; Build/AP4A.190211.226) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.1587.89 Mobile Safari/537.36 EdgA/110.0.1587.89",
),
(
"Firefox, Android",
"Mozilla/5.0 (Android 15; Mobile; rv:154.0) Gecko/154.0 Firefox/154.0",
),
(
"Safari, iPhone",
"Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Mobile/15E148 Safari/604.1",
),
(
"Huawei HarmonyOS phone",
"Mozilla/5.0 (Phone; OpenHarmony 6.1; Android 10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36 ArkWeb/6.1.0.120 Mobile HuaweiBrowser/6.1.6.310",
),
("GPTBot", "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)"),
(
"Googlebot (disguised)",
"Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)",
),
(
"Claude-SearchBot (disguised)",
"Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.3245.171 Mobile Safari/537.36; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot",
),
(
"Facebook preview (disguised)",
"Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.6885.65 Mobile Safari/537.36; compatible; facebookexternalhit/1.1; +http://www.facebook.com/externalhit_uatext.php",
),
(
"Meta crawler (disguised)",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))",
),
("WhatsApp preview", "Mozilla/5.0 (compatible; WhatsApp/10.0.2.1)"),
(
"Bytespider",
"Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://zhanzhang.toutiao.com/)",
),
(
"BingPreview",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b",
),
("AhrefsBot", "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"),
("python-requests", "python-requests/2.32.5"),
]
def imitate_uaparser(ua):
r = ua_parse(ua)
fam = (r.user_agent.family if r.user_agent else "") or ""
maj = (r.user_agent.major if r.user_agent else "") or ""
osf = (r.os.family if r.os else "") or ""
dev = (r.device.family if r.device else "") or ""
return f"{fam}/{maj} {osf} {dev}"
def imitate_useragents(ua):
r = uas_parse(ua)
fam = r.browser.family or ""
maj = str(r.browser.version[0]) if r.browser.version else ""
return f"{fam}/{maj} {r.os.family or ''} {r.device.family or ''}"
print("| Case | uarite¹ | ua-parser² | user-agents³ |")
print("|---|---|---|---|")
for name, ua in CASES:
ours = uaparse(ua).pretty or "—"
print(f"| {name} | {ours} | {imitate_uaparser(ua)} | {imitate_useragents(ua)} |")
uarite/__init__.py
+5
View File
@@ -0,0 +1,5 @@
"""Parse and pretty format user-agent strings."""
from .core import UA, uaparse
__all__ = ["UA", "uaparse"]
uarite/bots.py
+85
View File
@@ -0,0 +1,85 @@
"""Crawler and link-preview tables."""
#: Lowercase UA substring to (display name, kind). Ordered: first match
#: wins, so overlapping names go from most to least specific.
BOTS = {
"feedfetcher-google": ("Feedfetcher-Google", "search"),
"google-inspectiontool": ("Google-InspectionTool", "search"),
"google-read-aloud": ("Google-Read-Aloud", "search"),
"mediapartners-google": ("Mediapartners-Google", "spider"),
"adsbot-google": ("AdsBot-Google", "spider"),
"apis-google": ("APIs-Google", "spider"),
"storebot-google": ("Storebot-Google", "spider"),
"duplexweb-google": ("DuplexWeb-Google", "spider"),
"google-extended": ("Google-Extended", "ai"),
"googlebot": ("Googlebot", "search"),
"googleother": ("GoogleOther", "spider"),
"bingbot": ("Bingbot", "search"),
"applebot": ("Applebot", "search"),
"gptbot": ("GPTBot", "ai"),
"oai-searchbot": ("OAI-SearchBot", "search"),
"chatgpt-user": ("ChatGPT-User", "ai"),
"claude-searchbot": ("Claude-SearchBot", "search"),
"claudebot": ("ClaudeBot", "ai"),
"claude-user": ("Claude-User", "ai"),
"perplexity-user": ("Perplexity-User", "ai"),
"perplexitybot": ("PerplexityBot", "search"),
"grokbot": ("GrokBot", "ai"),
"bytespider": ("Bytespider", "ai"),
"reflectionbot": ("Reflectionbot", "ai"),
"amzn-searchbot": ("Amzn-SearchBot", "search"),
"amazonbot": ("Amazonbot", "search"),
"ahrefsbot": ("AhrefsBot", "spider"),
"mj12bot": ("MJ12bot", "spider"),
"facebookexternalhit": ("Facebook", "preview"),
"meta-externalagent": ("Meta", "preview"),
"skypeuripreview": ("Skype", "preview"),
"bingpreview": ("BingPreview", "preview"),
"pinterest": ("Pinterest", "preview"),
"embedly": ("Embedly", "preview"),
"iframely": ("Iframely", "preview"),
"discordbot": ("Discord", "preview"),
"slackbot": ("Slack", "preview"),
"telegrambot": ("Telegram", "preview"),
"twitterbot": ("Twitter", "preview"),
"linkedinbot": ("LinkedIn", "preview"),
"whatsapp": ("WhatsApp", "preview"),
"headlesschrome": ("HeadlessChrome", "spider"),
"phantomjs": ("PhantomJS", "spider"),
"uptimerobot": ("UptimeRobot", "spider"),
"pingdom": ("Pingdom", "spider"),
}
#: Pretty suffixes for the kinds more precise than a generic spider.
KIND_LABEL = {"ai": "AI", "search": "search", "preview": "preview"}
#: Providers with more than one crawler product; the kind label is kept
#: only where it distinguishes siblings within the group.
PROVIDERS = [
(
"Googlebot",
"Google-Extended",
"GoogleOther",
"Feedfetcher-Google",
"Google-InspectionTool",
"Google-Read-Aloud",
"Mediapartners-Google",
"AdsBot-Google",
"APIs-Google",
"Storebot-Google",
"DuplexWeb-Google",
),
("ClaudeBot", "Claude-User", "Claude-SearchBot"),
("GPTBot", "OAI-SearchBot", "ChatGPT-User"),
("PerplexityBot", "Perplexity-User"),
("Amazonbot", "Amzn-SearchBot"),
("Bingbot", "BingPreview"),
]
#: Bot names whose kind label is displayed, computed from the groups.
LABELED = {
name
for group in PROVIDERS
if len({kind for n, kind in BOTS.values() if n in group}) > 1
for name in group
}
uarite/clients.py
+100
View File
@@ -0,0 +1,100 @@
"""Browser and device-model tables."""
#: (UA token, display name) checked in order; first hit wins, so
#: overlapping tokens go from most to least specific.
BROWSERS = (
("HuaweiBrowser", "HuaweiBrowser"),
("EdgA", "Edge"),
("Edg", "Edge"),
("OPR", "Opera"),
("Vivaldi", "Vivaldi"),
("YaBrowser", "Yandex"),
("Brave", "Brave"),
("Whale", "Whale"),
("SamsungBrowser", "Samsung Internet"),
("MiuiBrowser", "Mi Browser"),
("UCBrowser", "UC Browser"),
("QQBrowser", "QQ Browser"),
("DuckDuckGo", "DuckDuckGo"),
("LibreWolf", "LibreWolf"),
("Firefox", "Firefox"),
("Chrome", "Chrome"),
("CriOS", "Chrome"),
("FxiOS", "Firefox"),
)
#: Samsung model codes (without the trailing region/carrier letter) to
#: marketing names; breadth over precision.
SAMSUNG = {
# Galaxy S (SM-G era)
"SM-G930": "Galaxy S7",
"SM-G935": "Galaxy S7 Edge",
"SM-G950": "Galaxy S8",
"SM-G955": "Galaxy S8+",
"SM-G960": "Galaxy S9",
"SM-G965": "Galaxy S9+",
"SM-G970": "Galaxy S10e",
"SM-G973": "Galaxy S10",
"SM-G975": "Galaxy S10+",
"SM-G977": "Galaxy S10 5G",
"SM-G980": "Galaxy S20",
"SM-G981": "Galaxy S20",
"SM-G985": "Galaxy S20+",
"SM-G986": "Galaxy S20+",
"SM-G988": "Galaxy S20 Ultra",
"SM-G990": "Galaxy S21 FE",
"SM-G991": "Galaxy S21",
"SM-G996": "Galaxy S21+",
"SM-G998": "Galaxy S21 Ultra",
# Galaxy S (SM-S9xx era)
"SM-S901": "Galaxy S22",
"SM-S906": "Galaxy S22+",
"SM-S908": "Galaxy S22 Ultra",
"SM-S911": "Galaxy S23",
"SM-S916": "Galaxy S23+",
"SM-S918": "Galaxy S23 Ultra",
"SM-S921": "Galaxy S24",
"SM-S926": "Galaxy S24+",
"SM-S928": "Galaxy S24 Ultra",
"SM-S931": "Galaxy S25",
"SM-S936": "Galaxy S25+",
"SM-S938": "Galaxy S25 Ultra",
# Galaxy Note
"SM-N930": "Galaxy Note 7",
"SM-N950": "Galaxy Note 8",
"SM-N960": "Galaxy Note 9",
"SM-N970": "Galaxy Note 10",
"SM-N975": "Galaxy Note 10+",
"SM-N980": "Galaxy Note 20",
"SM-N981": "Galaxy Note 20",
"SM-N985": "Galaxy Note 20 Ultra",
"SM-N986": "Galaxy Note 20 Ultra",
# Galaxy Z foldables
"SM-F700": "Galaxy Z Flip",
"SM-F707": "Galaxy Z Flip 5G",
"SM-F711": "Galaxy Z Flip 3",
"SM-F721": "Galaxy Z Flip 4",
"SM-F731": "Galaxy Z Flip 5",
"SM-F741": "Galaxy Z Flip 6",
"SM-F900": "Galaxy Fold",
"SM-F907": "Galaxy Fold 5G",
"SM-F916": "Galaxy Z Fold 2",
"SM-F926": "Galaxy Z Fold 3",
"SM-F936": "Galaxy Z Fold 4",
"SM-F946": "Galaxy Z Fold 5",
"SM-F956": "Galaxy Z Fold 6",
}
#: Samsung series for codes missing from the table above.
SAMSUNG_SERIES = {
"SM-S": "Galaxy S",
"SM-G": "Galaxy S",
"SM-N": "Galaxy Note",
"SM-A": "Galaxy A",
"SM-J": "Galaxy J",
"SM-M": "Galaxy M",
"SM-E": "Galaxy E",
"SM-F": "Galaxy Z",
"SM-T": "Galaxy Tab",
"SM-X": "Galaxy Tab",
}
uarite/core.py
+202
View File
@@ -0,0 +1,202 @@
"""User-Agent parsing."""
import re
from dataclasses import dataclass
from functools import lru_cache
from .bots import BOTS, KIND_LABEL, LABELED
from .clients import BROWSERS, SAMSUNG, SAMSUNG_SERIES
@dataclass(frozen=True)
class UA:
pretty: str
bot: str
kind: str
url: str
#: Fallback for unknown crawlers: a product token whose name says so.
BOT_TOKEN = re.compile(
r"[^\s();/]*(?:bot|spider|crawl|scan|verif|check)[^\s();/]*", re.IGNORECASE
)
#: A "+https://…" pointer is a crawler tell; real browsers carry no URL.
URL = re.compile(r"\+\s*(https?://[^\s;)]+)")
ANY_URL = re.compile(r"(https?://[^\s;)]+)")
#: Crawler name next to the info URL: "compatible; Page2RSS/0.7; +http://…".
COMPATIBLE_NAME = re.compile(r"compatible;\s*([^;/()]+?)(?:/[\d.vx]+)?\s*;")
#: Name from an info URL's host when no product token is available.
HOST = re.compile(r"https?://(?:www\.)?([^/\s;)]+)")
#: Android model token: "Android 15; SM-S918B)", "Android 12; Pixel 6; Build/…".
ANDROID_MODEL = re.compile(r"Android [\d.]+; ([^;()]+?)(?:;|\)| Build/)")
#: All known-bot substrings in one compiled pass — much faster than 50
#: individual ``in`` checks on long UAs.
BOTS_RE = re.compile("|".join(map(re.escape, BOTS)))
def url(ua: str) -> str:
"""The crawler's info URL from the UA, or "" (mailto: is not a URL)."""
m = URL.search(ua) or ANY_URL.search(ua)
return m.group(1) if m else ""
def bot(ua: str) -> tuple[str, str]:
"""(display name, kind) of the crawler/previewer the UA claims."""
low = ua.lower()
m = BOTS_RE.search(low)
if m:
return BOTS[m.group(0)]
# Cheap keyword gates keep the regexes off the hot path.
if any(k in low for k in ("bot", "spider", "crawl", "scan", "verif", "check")):
m = BOT_TOKEN.search(ua)
if m:
name = m.group(0).strip(";")
return name[:1].upper() + name[1:], "spider"
# An info URL in the UA is a crawler convention. Name it from the
# "compatible; Name/x" token, else the first product token, else the
# URL's host.
if "://" in ua:
name = ""
m = COMPATIBLE_NAME.search(ua)
if m:
name = m.group(1).strip()
elif not ua.startswith("Mozilla"):
name = ua.split()[0].split("/")[0]
if not name or name.lower().startswith(("mozilla", "msie")):
m = HOST.search(ua)
name = m.group(1) if m else ""
if name:
return name[:1].upper() + name[1:], "spider"
return "", ""
def version(ua: str, token: str) -> str:
"""Major version of a ``token/x.y`` product in the UA, or ""."""
m = re.search(re.escape(token) + r"/(\d+)", ua)
return m.group(1) if m else ""
def browser(ua: str) -> str:
"""``Browser/major`` for the browsers we care to distinguish."""
for token, name in BROWSERS:
ver = version(ua, token)
if ver:
return f"{name}/{ver}"
if "Safari/" in ua:
ver = version(ua, "Version")
return f"Safari/{ver}" if ver else "Safari"
return ""
def os(ua: str) -> str:
"""Desktop OS name, or "" when not recognizable."""
if "Windows NT" in ua:
return "Windows"
if "Mac OS X" in ua:
return "macOS"
if "Linux" in ua or "X11" in ua:
return "Linux"
if "Windows" in ua:
return "Windows"
if "Darwin" in ua:
return "macOS"
return ""
def model_name(model: str) -> str:
"""Human-readable phone name for an Android model code.
Returns the input unchanged when nothing is known about it (Pixel and
most other brands already send readable names).
"""
if model.startswith("SM-"):
# Strip the trailing region/carrier letter: SM-S918B -> SM-S918.
code = re.sub(r"[A-Z]{1,2}$", "", model)
if code in SAMSUNG:
return SAMSUNG[code]
series = SAMSUNG_SERIES.get(code[:4])
if series:
return series
return model
def uaparse(ua: str) -> UA:
"""Parse a User-Agent string into a compact ``UA`` record.
``pretty`` is "" for empty/missing UAs and the original string when
nothing is recognized.
Only the browser path is cached: real visitors repeat (cache hits),
while crawlers and scripts are mostly one-hit wonders whose entries
would just flush the cache.
"""
if not ua or not ua.strip() or ua in ("-", "null"):
return UA("", "", "", "")
name, kind = bot(ua)
if name:
# The browser/OS in crawler UAs is a disguise; the bot identity is
# the relevant information.
label = KIND_LABEL.get(kind, "") if name in LABELED else ""
pretty = f"{name} ({label})" if label else name
return UA(pretty, name, kind, url(ua))
return _parse_client(ua)
@lru_cache(maxsize=1024)
def _parse_client(ua: str) -> UA:
"""Browser/client parsing behind the cache; ``uaparse`` filters bots out."""
# Non-browser HTTP clients ("python-requests/2.32.5", "curl/8.0",
# "pip/24.3.1 {json…}"): the first product token, plus the OS when
# their payload mentions one in free text.
if not ua.startswith("Mozilla"):
token = ua.split()[0]
pretty = token if "/" in token else ua
os_name = os(ua)
if os_name and os_name not in pretty:
pretty = f"{pretty} {os_name}"
return UA(pretty, "", "", "")
# HarmonyOS carries an "Android" compatibility token, so it must be
# detected before Android.
if "OpenHarmony" in ua or "HarmonyOS" in ua or "ArkWeb" in ua:
b = browser(ua)
return UA(f"{b} HarmonyOS" if b else "HarmonyOS", "", "browser", "")
if "iPhone" in ua or "iPad" in ua:
device = "iPhone" if "iPhone" in ua else "iPad"
m = re.search(r"OS (\d+)", ua)
pretty = f"{device} iOS {m.group(1)}" if m else device
return UA(pretty, "", "browser", "")
m = re.search(r"Android ([\d.]+)", ua)
if m:
b = browser(ua)
mm = ANDROID_MODEL.search(ua)
token = mm.group(1).strip() if mm else ""
if token == "K":
# Chrome's reduced UA freezes both: "Android 10; K". Neither
# is real — report just the OS.
parts = [p for p in (b, "Android") if p]
else:
# Firefox sends the form factor ("Mobile"/"Tablet") in the
# model slot. A known model replaces the OS.
model = ""
if token and token not in ("wv", "Mobile", "Tablet"):
model = model_name(token)
parts = [p for p in (b, model or f"Android {m.group(1)}") if p]
return UA(" ".join(parts), "", "browser", "")
b = browser(ua)
pretty = f"{b} {os(ua)}".strip()
return UA(pretty or ua, "", "browser", "")
def is_bot(ua: str) -> bool:
"""True when the UA claims a crawler or link-preview identity."""
return bool(uaparse(ua).bot)