AML-memory/agent-memory-leaderboard
Contribute to AML-memory/agent-memory-leaderboard development by creating an account on GitHub.
1 stars · 0 forks · 0 watchers
README
Public Evaluation Code This directory publishes the answer-generation and evaluation contracts used by the Agent Memory Leaderboard for its public datasets. The scripts require Python 3.10 or newer and the dependency listed in requirements.txt. Service URLs, model names, and credentials must be supplied through CLI arguments or environment variables; no credential is bundled here. Provenance and integrity Internal shorthand labels are normalized to descriptive public names; answer construction and scoring behavior are unchanged. The disclosed runtime behavior covers Add/Search/Answer concurrency, timeouts, retries, key rotation, checkpoints, and aggregation. Internal deployment code, service addresses, credentials, databases, datasets, and run artifacts are intentionally outside this public evaluation release. Disclosed production parameters
Full task timeout: 72 hours; retrieval top_k: 100. Add: 64 global workers by default, 48-record per-dataset hard cap, 20-message chunks, 1,200-second HTTP timeout, and global-largest-next-record-v1 scheduling using estimated chunk counts. Search: 32 workers, 1,200-second HTTP timeout, six request attempts, up to three failed-record requeues, and adaptive concurrency reduction after five retryable 5xx responses. Answer: 32 workers, gpt-4o-mini, temperature 0, 180-second HTTP timeout, and six attempts. The configured checkpoint value is 128 tokens, but the outbound provider request intentionally does not send max_tokens. Judge: a configured evaluator, 360-second HTTP timeout, and a global cap of 15 requests. Worker phase limits are 8 for binary evaluation, 15 for rubric evaluation, and 16 for BEAM event alignment; the global cap means effective Judge request concurrency never exceeds 15. Judge failures: HTTP 429, timeout, and transport failures rotate the key. Pool-wide 429 exhaustion uses a 30-second cooldown and at most three cooldown rounds. Qwen JSON requests set enable_thinking=false. Scores are computed on [0, 1] internally and published on [0, 100].