Benchmark

The tables below report TIA’s own evaluations on Vietnamese dev sets, not independent certification or operational accuracy. The vn-law and vi-sms sets map to two product lines, TiaL (law) and TiaS (anti-scam SMS); there is no separate build per line yet — every number below was measured on the shared TIA model. TIA 1.5 is a local experimental candidate, not approved for API deployment. The served build (revision eff27e8) scored 15/25 vi_sanity scenarios (60%). Historical TIA 1.4 mini runs scored 92.2% on vn-law v1.1 and 95.3% on vi-sms (89/99 scams detected, 8/283 legitimate messages flagged). Claude Opus 5.5 is a reference model in an evaluation conducted by TIA, not an independent auditor of TIA. These numbers do not replace evaluation on your real-world data, especially for legal, medical, credit and hiring decisions.

Download vi-bench.jsonEach source is an internal TIA document, named with its date; the document itself is not published.updated 2026-10-03

TIA 1.5 is a local experimental candidate, not approved for API deployment. On the 231 public JevBench v1.4.2.2 items it gets 125/231 right (54.1%, Wilson 95% interval 47.7–60.4). TIA combines this run with published rows to compute position 79/98: worse than 63, not distinguishable from 24, better than 10 across 97 comparisons. Its position by correct decisions per US$1 is 24/96, by correct decisions per second 14/94; candidate pricing is estimated from the API tariff, not a service being sold. This is not an official JevBench ranking or independent evaluation. The API currently serves revision eff27e8, with a historical result of 122/231.

TIA 1.5 correct

125/23154.1%

Rank by accuracy

79/98

Rank per tier (same items, among the 56 rows with per-item outcomes)

easy 42/56 · original 41/56 · hard 47/56

Verdicts against 97 rows

worse 63 · not distinguishable 24 · better 10

The JevBench maintainers measured and published every row except the TIA rows (release v1.4.2.2, published Sep 27, 2026, commit fcc6927). TIA ran TIA 1.5 on Oct 3, 2026 (and TIA 1.4 mini and TIA 1.3 mini and TIA 1.2 mini on the dates shown in their rows) with the unmodified JevBench harness and its typesafe adapter, on a local CPU worker serving the same model. TIA did not re-run any other system: the underlying measurements in those rows (overall results, per-item outcomes, published tiers, price, latency) come from the official files, linked to the exact line in the last column. Everything derived from them (correct counts, Wilson intervals, easy/original/hard rates on the same items, unit conversions, differences from TIA, tests, verdicts, ranks) is computed by TIA from those same files.

Paired per item (Holm over 55 comparisons: 52 rows with per-item outcomes published by JevBench, plus TIA 1.4 mini and TIA 1.3 mini and TIA 1.2 mini with per-item outcomes from TIA runs): worse 33, not distinguishable 14, better 8. Unpaired (42 rows with only the overall accuracy, Wilson intervals): worse 30, not distinguishable 10, better 2.

The rank above counts every row of the official file, including 4 rows JevBench lists without ranking (an honorable mention or a partial run). Counting only the rows JevBench ranks: rank 77 of 94; worse than 61, not distinguishable from 24, better than 8.

Not compared and not ranked: Autoloops – Gemma 4 31B IT. Its published accuracy is not k/231, so it was measured on a different item set; the table prints the published number at the bottom.

How it is scored

• Compared rows use the same 231 JevBench public items (easy 48, original 72, hard 111), commit fcc6927, JevBench's scorer. Failed or unanswered items count as wrong, as in the published scores.
• Rows with per-item outcomes (52 published by JevBench, plus TIA 1.4 mini and TIA 1.3 mini and TIA 1.2 mini from TIA runs): two-sided exact McNemar test between TIA 1.5 and the system on the items the system answered (all 231 except three partial runs), 95% CI of the accuracy difference from a 10,000-resample paired bootstrap, Holm correction over 55 comparisons, verdict at Holm p < 0.05.
• Rows without per-item outcomes: when the published accuracy is exactly k/231, it is over the same 231 items but cannot be paired. We only give a verdict when the two Wilson 95% intervals are disjoint, which is more conservative than a paired test. A rate that is not k/231 was measured on a different item set: printed, never compared or ranked.
• JevBench's published tiers (easy, standard, judge, hard) include held-out items TIA was never run on, so they are printed for reference and never compared with TIA.
• Prices use JevBench’s US$ per 1,000 decisions unit; each system’s figure is published, measured or estimated as its source states. For candidate TIA 1.5, we estimate cost using the API tariff of 500 VND per 1M input tokens (from 1 October 2026), multiplied by measured tokens per item at 26,000 VND/USD. This is not a sale price for the candidate. Historical builds TIA 1.4 mini and TIA 1.3 mini and TIA 1.2 mini have no price in this table.
• Latency is p50 seconds per decision as JevBench publishes it (self-hosted rows carry JevBench's ×2 + 0.15 s rule). TIA's latency was measured by TIA through the harness on a local CPU, with the same rule applied; JevBench has not measured TIA.

Reproduce

The example below evaluates the model currently available through the API, not the local experimental candidate in the table. Reproducing that candidate requires the exact checkpoint and measurement configuration. Your API key needs sufficient rate limits; the harness stops at its first 429. Use Linux, macOS or a Linux container because the harness uses fcntl. Network latency will differ from the local measurement:

git clone https://github.com/fstandhartinger/jevbench && cd jevbench && git checkout fcc6927
# TIA_API_URL: the System One API base URL; TIA_API_KEY: your key (sent as a Bearer token)
# the harness refuses to overwrite results: empty ../out before a re-run
for s in easy original hard; do
  python -m jevbench.cli run --tasks datasets/public/$s.jsonl --adapter typesafe \
    --endpoint "$TIA_API_URL" --key-env TIA_API_KEY --model jev-tia \
    --results ../out/$s.jsonl --raw-dir ../out/raw-$s --ledger ../out/ledger.jsonl
  python -m jevbench.cli summarize --tasks datasets/public/$s.jsonl --results ../out/$s.jsonl
done

Rebuild the table: take the official results and per-item files (links below) and TIA's per-item outputs (download below). For each row with per-item outcomes, count the items only TIA got right and only the system got right, run the exact McNemar test, then Holm over the family of 55 comparisons.

The item text is JevBench's (MIT licence, redistribution allowed). We link the files at the pinned commit instead of copying them; our download carries item ids, gold labels and TIA's outputs.

The same metrics split by question type. One global temperature calibrates noul well but leaves score and choice ECE high.

TIA System One vs TIA System One (previous) on the same 930 dev decisions. Difference (challenger − champion) with 95% CIs from a 2,000-resample paired bootstrap. Negative KL and Brier are better; the accuracy interval touches 0, so accuracy is inconclusive.

The gate’s ΔBrier is averaged per option, so it is on a smaller scale than the table’s Brier column (per decision). Do not subtract one from the other.

Verdict: PROMOTE

• Different splits: TIA numbers are on an internal dev set (186 cases) carved from train; competitor numbers are on test (400 cases). The two are not equally hard.
• Different measurer: competitor numbers are reported by the dataset card author; we have not re-run them. Our metric code does not fully match the card yet (the Prior row and acc/ECE conventions are not reproduced), so there has been no test read.
• Different latency method: server-side for TIA, end-to-end from a client for competitors.
• Protocol: before reading test we publish a pre-registration (the model’s SHA-256 hash, the calibration temperature fitted on separate calibration data, seeds, the scoring-code version, the expected result), then read test exactly once. Accuracy above the self-agreement of the dataset’s own labels (per the dataset card) is reported as saturation, not skill.

No number is invented or estimated for these models.

Laya not measured

We have not run Laya under the same protocol (same split, same metric code, same latency method). We do not put another party's self-reported numbers in this table.

Verdict not measured

No public typed-decisions numbers under the same protocol, and we have not measured it ourselves.