Raw logs
Every number on our leaderboard that comes from our own runs has a raw log behind it: the exact prompt, the model's output, the verdict, latency, token count, and cost — one file per model per run day. Formal tests (DeepSeek V4 Pro, V4 Flash) carry the "Tested by us" badge; the five Chinese models and four US/EU reference rows below are N=1 smoke runs from Aug 7, 2026. Logs are mirrored on GitHub and Hugging Face. If you spot a verdict you disagree with, the file to check is linked here.
All 11 models, log files linked
| Model | Batch | Passed | Logs (HF tree + GitHub) |
|---|---|---|---|
DeepSeek V4 Pro deepseek-v4-pro | Formal (Aug 3 first run + Aug 7 re-confirmation) | 11/11 | |
DeepSeek V4 Flash-0731 deepseek-v4-flash | Formal (Aug 3 first run + Aug 7 re-confirmation) | 10/11math-quadratic: answered "9, -4" on the rerun (Aug 3 first run failed 23×25 — drift disclosed) | |
GLM-5.2 glm-5.2 | Smoke (Aug 7, 2026, N=1) | 11/11 | |
Doubao Seed 2.1 Pro doubao-seed-2-1-pro | Smoke (Aug 7, 2026, N=1) | 11/11 | |
Step 3.5 Flash step-3.5-flash | Smoke (Aug 7, 2026, N=1) | 10/11ipv4-validator: reasoning didn't converge (1024 max_tokens exhausted; 4096 retry ~15.5K reasoning tokens, still empty) | |
Hunyuan A13B hunyuan-a13b | Smoke (Aug 7, 2026, N=1) | 10/11math-quadratic-roots: answered "3, -4", roots are 4 and -3 (sign check skipped) | |
Ling Flash 2.0 ling-flash-2.0 | Smoke (Aug 7, 2026, N=1) | 11/11 | |
GPT-5.6 Luna Pro international/gpt-5.6-luna-pro | Reference (Aug 7, 2026, N=1, OpenRouter free tier) | 11/11 | |
Claude Opus 5 international/claude-opus-5 | Reference (Aug 7, 2026, N=1, OpenRouter free tier) | 10/11writing-product-copy: quota (402), not capability — max_tokens spent on reasoning trace | |
Gemini 3.6 Flash international/gemini-3.6-flash | Reference (Aug 7, 2026, N=1, OpenRouter free tier) | 11/11writing-product-copy first run truncated at 1024 (FAIL) → 2048 retry PASS (first_run_truncated kept) | |
Grok 4.5 international/grok-4.5 | Reference (Aug 7, 2026, N=1, OpenRouter free tier) | 11/11 |
The two DeepSeek rows also expand inline below: full prompts, raw outputs, verdicts, latency, tokens and cost for all 22 calls — mirrored from data/raw-logs/.
DeepSeek V4 — inline logs (formal runs)
Rerun on 2026-08-07 — 22 real API calls to api.deepseek.com (11 tasks × 2 models). Total cost $0.001228. Full prompts pre-registered before the run; identical prompts for both models.
Sampling: temperature default · top_p default · max_tokens 64–512 per task · thinking disabled on both models · N=1 per task. Client: local Windows 10; server: DeepSeek-hosted API (vendor hardware not disclosed).
Full prompt set (pre-registered, 2026-08-07) →First run (2026-08-03) summary: Pro 11/11, Flash 10/11 — same scores, different Flash miss (23×25→341 then vs quadratic→"9, -4" now). Reported honestly, not smoothed.
DeepSeek V4 Pro (preview (2026-07))
Score 11/11 · avg latency 1.6s · cost $0.000949
PASSIPv4 validator1.52s · 95+104 tok · $0.000132
Write a Python function is_valid_ipv4(s: str) -> bool that returns True if s is a valid IPv4 address and False otherwise. An IPv4 address consists of exactly four decimal octets separated by dots; each octet must be between 0 and 255, must contain no leading zeros (except the octet '0' itself), and must contain only digits. Return only the function definition, no explanation, no test code.
```python
def is_valid_ipv4(s: str) -> bool:
parts = s.split(".")
if len(parts) != 4:
return False
for part in parts:
if not part.isdigit():
return False
if len(part) > 1 and part[0] == "0":
return False
num = int(part)
if num < 0 or num > 255:
return False
return True
```PASSBug fix: second-largest distinct element1.3s · 94+34 tok · $0.000070
The function below is supposed to return the second-largest DISTINCT element of a list of integers, or None if fewer than two distinct elements exist. It has a bug. Fix it.
```python
def second_largest(xs):
# BUG: fails when the largest value appears more than once
xs = sorted(xs)
return xs[-2]
```
Return only the corrected function definition, no explanation, no test code.def second_largest(xs):
dist = sorted(set(xs))
if len(dist) < 2:
return None
return dist[-2]PASSMath: quadratic roots with b+c = -131.1s · 30+4 tok · $0.000017
Solve for x: x^2 - x - 12 = 0. Answer with just the number(s), no explanation.
4, -3
PASSMath: consecutive integers summing to 5751.62s · 26+80 tok · $0.000081
Two consecutive integers sum to 575. What are they? Answer with just the number(s), no explanation.
Let the two consecutive integers be \( n \) and \( n+1 \). \[ n + (n+1) = 575 \] \[ 2n + 1 = 575 \] \[ 2n = 574 \] \[ n = 287 \] So the integers are \( 287 \) and \( 288 \). 287, 288
PASSLogic: syllogism0.96s · 36+2 tok · $0.000017
Premises: All humans are mortal. Socrates is a human. Conclusion: Socrates is mortal. Does the conclusion follow from the premises? Answer yes or no.
Yes.
PASSKnowledge: 36th US president1.16s · 22+5 tok · $0.000014
Who was the 36th president of the United States? Answer with just the name.
Lyndon B. Johnson
PASSKnowledge: capital of Burkina Faso3.75s · 19+5 tok · $0.000013
What is the capital of Burkina Faso? Answer with just the city name.
Ouagadougou
PASSWriting: 3-sentence product copy, constrained2.07s · 57+65 tok · $0.000081
Write exactly 3 sentences of product copy for CloudShelf — a self-hosted document store for small engineering teams. Each sentence must be at most 25 words. Mention the product name CloudShelf. No headings, no bullet points, no extra text.
CloudShelf gives your small engineering team a fast, self-hosted document store that keeps all data securely behind your own firewall. You can instantly search across specs and runbooks because CloudShelf indexes everything without complex setup. Stop context-switching between scattered wikis and reclaim your team’s focus with CloudShelf.
PASSExtraction: dates and names as JSON1.54s · 158+59 tok · $0.000120
Extract all dates and all person names from the passage below into a JSON object with two keys: "dates" (array of date strings in YYYY-MM-DD format) and "names" (array of person names). Return only the JSON object, no explanation. Passage: On 2026-08-03, the Benchmarks Desk at China AI Bench published its first smoke battery results. The report was compiled by Eli Chen, the desk's lead editor, with assistance from Mei Lin, who reviewed the sampling parameters. The follow-up verification run was completed on 2026-07-31, three days before publication, and the final revision was approved by the editorial board on August 5, 2026.
```json
{
"dates": [
"2026-08-03",
"2026-07-31",
"2026-08-05"
],
"names": [
"Eli Chen",
"Mei Lin"
]
}
```PASSSummary as 3-item JSON array1.53s · 133+65 tok · $0.000114
Summarize the passage below as a JSON array of exactly 3 strings, each string being one concise point. Return only the JSON array, no explanation. Passage: On 2026-08-03, the Benchmarks Desk at China AI Bench published its first smoke battery results. The report was compiled by Eli Chen, the desk's lead editor, with assistance from Mei Lin, who reviewed the sampling parameters. The follow-up verification run was completed on 2026-07-31, three days before publication, and the final revision was approved by the editorial board on August 5, 2026.
```json [ "China AI Bench released its first smoke battery results on 2026-08-03.", "The report was led by Eli Chen with assistance from Mei Lin.", "Verification was completed on 2026-07-31, with final board approval on August 5." ] ```
PASSNeedle in a 426-token document0.95s · 656+5 tok · $0.000290
Read the deployment report below and answer: what is the serial number of the primary gateway node gw-01? Answer with just the serial number, no explanation. Report: CloudShelf deployment report — production cluster EU-west Prepared by the platform engineering group. Revision 4.2. Date of issue: 2026-07-28. 1. Summary The EU-west production cluster was upgraded from release 3.9 to release 4.0 during the maintenance window of July 25, 2026. The upgrade completed in 41 minutes with no data loss. All fourteen application nodes and both gateway nodes are now running the new image tag. Rollback procedures were not exercised because no critical regression was observed in the first twelve hours of post-upgrade monitoring. 2. Node inventory The cluster comprises fourteen application nodes (app-01 through app-14), two gateway nodes (gw-01, gw-02), one primary database instance (db-primary) and one replica (db-replica). Each application node runs eight worker processes with a shared memory pool of 4 GiB. The gateway nodes terminate TLS and route traffic on ports 443 and 8443. The database runs PostgreSQL 17 with a 512 GiB buffer cache. 3. Configuration changes Release 4.0 introduced three notable changes. First, the request cache now respects the Vary header, which reduced upstream calls by 18 percent under the July load test. Second, connection pooling was moved from the application layer into the database driver, cutting average connection establishment latency from 9.2 ms to 3.4 ms. Third, the metrics endpoint now exposes histogram buckets for p50, p95 and p99 so that dashboards no longer need to aggregate raw counters. 4. Security and compliance All certificates were rotated on July 27, 2026. The new certificate bundle expires on October 25, 2026. Access to the admin API is now restricted to the operations subnet, and every administrative action is written to the audit log. The quarterly penetration test report is pending publication; no findings from the previous cycle remain open. 5. Incident notes One incident was logged on July 26, 2026: a single application node (app-09) experienced elevated memory pressure for 22 minutes during a deployment of the search index. The node remained available, and the pressure subsided once the index warm-up completed. The primary gateway node gw-01 carries the serial number XK-9472 and its firmware was updated to version 2.1.8 as part of this maintenance window. The standby gateway gw-02 carries serial number XK-9473 and was not modified. 6. Next steps The next maintenance window is scheduled for August 22, 2026. Planned work includes the introduction of read replicas for the search index and a hardware refresh for the two oldest application nodes. A capacity review for the 2026 holiday traffic peak is scheduled for September 3, 2026.
XK-9472
DeepSeek V4 Flash-0731 (stable (2026-07-31))
Score 10/11 · avg latency 1.3s · cost $0.000279
PASSIPv4 validator1.07s · 95+99 tok · $0.000041
Write a Python function is_valid_ipv4(s: str) -> bool that returns True if s is a valid IPv4 address and False otherwise. An IPv4 address consists of exactly four decimal octets separated by dots; each octet must be between 0 and 255, must contain no leading zeros (except the octet '0' itself), and must contain only digits. Return only the function definition, no explanation, no test code.
```python
def is_valid_ipv4(s: str) -> bool:
parts = s.split('.')
if len(parts) != 4:
return False
for part in parts:
if not part.isdigit():
return False
if len(part) > 1 and part[0] == '0':
return False
if not 0 <= int(part) <= 255:
return False
return True
```PASSBug fix: second-largest distinct element1.05s · 94+39 tok · $0.000024
The function below is supposed to return the second-largest DISTINCT element of a list of integers, or None if fewer than two distinct elements exist. It has a bug. Fix it.
```python
def second_largest(xs):
# BUG: fails when the largest value appears more than once
xs = sorted(xs)
return xs[-2]
```
Return only the corrected function definition, no explanation, no test code.```python
def second_largest(xs):
xs = sorted(set(xs))
if len(xs) < 2:
return None
return xs[-2]
```FAILMath: quadratic roots with b+c = -131.11s · 30+4 tok · $0.000005
Solve for x: x^2 - x - 12 = 0. Answer with just the number(s), no explanation.
9, -4
PASSMath: consecutive integers summing to 5750.8s · 26+4 tok · $0.000005
Two consecutive integers sum to 575. What are they? Answer with just the number(s), no explanation.
287, 288
PASSLogic: syllogism3.85s · 36+2 tok · $0.000006
Premises: All humans are mortal. Socrates is a human. Conclusion: Socrates is mortal. Does the conclusion follow from the premises? Answer yes or no.
Yes.
PASSKnowledge: 36th US president0.83s · 22+5 tok · $0.000004
Who was the 36th president of the United States? Answer with just the name.
Lyndon B. Johnson
PASSKnowledge: capital of Burkina Faso0.86s · 19+5 tok · $0.000004
What is the capital of Burkina Faso? Answer with just the city name.
Ouagadougou
PASSWriting: 3-sentence product copy, constrained1.31s · 57+51 tok · $0.000022
Write exactly 3 sentences of product copy for CloudShelf — a self-hosted document store for small engineering teams. Each sentence must be at most 25 words. Mention the product name CloudShelf. No headings, no bullet points, no extra text.
CloudShelf gives your small team a private, self-hosted document store without cloud vendor lock-in. Spin up secure, searchable doc storage in minutes using your own infrastructure. With CloudShelf, engineers stop hunting for files and start shipping faster.
PASSExtraction: dates and names as JSON0.97s · 158+41 tok · $0.000034
Extract all dates and all person names from the passage below into a JSON object with two keys: "dates" (array of date strings in YYYY-MM-DD format) and "names" (array of person names). Return only the JSON object, no explanation. Passage: On 2026-08-03, the Benchmarks Desk at China AI Bench published its first smoke battery results. The report was compiled by Eli Chen, the desk's lead editor, with assistance from Mei Lin, who reviewed the sampling parameters. The follow-up verification run was completed on 2026-07-31, three days before publication, and the final revision was approved by the editorial board on August 5, 2026.
{"dates": ["2026-08-03", "2026-07-31", "2026-08-05"], "names": ["Eli Chen", "Mei Lin"]}PASSSummary as 3-item JSON array1.43s · 133+79 tok · $0.000041
Summarize the passage below as a JSON array of exactly 3 strings, each string being one concise point. Return only the JSON array, no explanation. Passage: On 2026-08-03, the Benchmarks Desk at China AI Bench published its first smoke battery results. The report was compiled by Eli Chen, the desk's lead editor, with assistance from Mei Lin, who reviewed the sampling parameters. The follow-up verification run was completed on 2026-07-31, three days before publication, and the final revision was approved by the editorial board on August 5, 2026.
```json [ "China AI Bench's Benchmarks Desk released its first smoke battery results on 2026-08-03.", "The report was led by editor Eli Chen, with Mei Lin reviewing sampling parameters.", "Verification ran on 2026-07-31, and final approval came from the editorial board on August 5, 2026." ] ```
PASSNeedle in a 426-token document0.9s · 656+5 tok · $0.000093
Read the deployment report below and answer: what is the serial number of the primary gateway node gw-01? Answer with just the serial number, no explanation. Report: CloudShelf deployment report — production cluster EU-west Prepared by the platform engineering group. Revision 4.2. Date of issue: 2026-07-28. 1. Summary The EU-west production cluster was upgraded from release 3.9 to release 4.0 during the maintenance window of July 25, 2026. The upgrade completed in 41 minutes with no data loss. All fourteen application nodes and both gateway nodes are now running the new image tag. Rollback procedures were not exercised because no critical regression was observed in the first twelve hours of post-upgrade monitoring. 2. Node inventory The cluster comprises fourteen application nodes (app-01 through app-14), two gateway nodes (gw-01, gw-02), one primary database instance (db-primary) and one replica (db-replica). Each application node runs eight worker processes with a shared memory pool of 4 GiB. The gateway nodes terminate TLS and route traffic on ports 443 and 8443. The database runs PostgreSQL 17 with a 512 GiB buffer cache. 3. Configuration changes Release 4.0 introduced three notable changes. First, the request cache now respects the Vary header, which reduced upstream calls by 18 percent under the July load test. Second, connection pooling was moved from the application layer into the database driver, cutting average connection establishment latency from 9.2 ms to 3.4 ms. Third, the metrics endpoint now exposes histogram buckets for p50, p95 and p99 so that dashboards no longer need to aggregate raw counters. 4. Security and compliance All certificates were rotated on July 27, 2026. The new certificate bundle expires on October 25, 2026. Access to the admin API is now restricted to the operations subnet, and every administrative action is written to the audit log. The quarterly penetration test report is pending publication; no findings from the previous cycle remain open. 5. Incident notes One incident was logged on July 26, 2026: a single application node (app-09) experienced elevated memory pressure for 22 minutes during a deployment of the search index. The node remained available, and the pressure subsided once the index warm-up completed. The primary gateway node gw-01 carries the serial number XK-9472 and its firmware was updated to version 2.1.8 as part of this maintenance window. The standby gateway gw-02 carries serial number XK-9473 and was not modified. 6. Next steps The next maintenance window is scheduled for August 22, 2026. Planned work includes the introduction of read replicas for the search index and a hardware refresh for the two oldest application nodes. A capacity review for the 2026 holiday traffic peak is scheduled for September 3, 2026.
XK-9472