probelabs/aeo
Measure whether Claude, Codex, and Grok mention your product. Two-arm local-CLI AEO: knowledge vs search, verbatim tool-call strings, a playbook for what to ship. - probelabs/aeo
1 stars · 0 forks
README
aeo Measure whether coding agents mention your product. aeo asks Claude Code, Codex, and Grok the same realistic questions twice: once with search forced off, once with search allowed. It records the mention, whether they actually searched, the literal strings they typed into the search box, and which competitor names were already in those strings. That is the whole product. It is not Gemini grounding, not Google AI Overviews, and not a login to claude.ai. Methodology · Playbook · Skills Why two arms A mention from weights and a mention after a web search are different facts.
Arm What you learn
Knowledge What the model already believes. A new brand almost never wins this path in year one. Measure it anyway.
Search Whether they searched, and what they typed. Most "search" is confirmation of an incumbent they already named, not discovery of you.
If they never type your name, and your page is not in the backend they used, writing more blog posts will not change the grid. The playbook is the operating loop for that: measure, ship one URL per cluster, check the page is live and indexed, re-run only the affected seeds. Install Python 3.11+. Stdlib only. You need claude, codex, and/or grok on your machine. Do not run those CLIs from a VPS or datacenter. pip install -e .
or
PYTHONPATH=src python3 -m aeo --help Quick start python3 -m aeo init --brand Acme --domain acme.example --out aeo.config.jsonor copy the XERJ example roster:
python3 -m aeo init --from-example xerj --out aeo.config.jsonpython3 -m aeo run --config aeo.config.json --engine all --arm both python3 -m aeo run --config aeo.config.json --engine all --arm both --concurrency 4 python3 -m aeo board aeo-data/runs/<run_id>.json python3 -m aeo report --html --out report.html aeo-data/runs/<run_id>.json --dry-run prints the exact claude / codex / grok command and exits. --only-id re-runs one roster seed. --samples N repeats that invocation (default n=1; local CLIs are slow). --concurrency N (default 1) runs up to N remaining cells in one process; workers write temp shards and the parent merges them into --out so resume cannot drop a cell. Do not share one --out across multiple aeo run processes. Never put the brand, a stack word, or an incumbent into a core prompt. If the model injects those into its own search call, that is a finding. What a run gives you Each cell is isolated in a fresh empty /tmp/aeo-isolate-* directory so Grok cannot read your playbook and "discover" the brand.
Artifact What it is
aeo-data/runs/<run_id>.json Raw evidence. Source of truth. Schema.
aeo board Decision board: win / gap / search-blind / trap, plus markdown + agent JSON.
aeo report --html Compact self-contained report. Merges several engine files.
Per arm the runner stores: brand_mentioned, searched, search_queries (verbatim), vendors_in_search_queries, and token/spend when the CLI JSON has it. After the numbers A zero-mention grid is not a prompt to write fifty articles.
curl every URL you claim is live. Homepage-sized 200s do not count. Split cells: confirmation vs discovery vs search-blind. One URL per cluster, only if you can publish a run you actually did. If the pages are already live and mentions stay 0, it is a retrieval problem. Playbook §11: Search Console, Bing Webmaster, IndexNow. Not more slugs.
Portable agent skills live in skills/: aeo (run), aeo-board (read), aeo-playbook (decide). After a full grid, scripts/judge_run.py labels stance on brand hits and extracts product names from every completed arm. Config competitors is the seed / known set; names not on that list after normalize are surprises (flagged separately). Brand hit rate stays the deterministic brand_mentioned regex. See scripts/README.md. Example examples/xerj is a real workspace (84 seeds, watch vs focus), not a hard-coded only-brand. Walkthrough of the fixture: How to read a run. Tests PYTHONPATH=src python3 -m unittest discover -s tests -v pip install -e '.[test]' # optional, pulls jsonschema Raw CLI flags Use these if the wrapper is blocked. Never pass --bare to Claude (it skips keychain).