TIA 1.0:a typed judgment model

A support agent has to decide whether a ticket is urgent, which team owns it and how bad the impact is. A review workflow asks whether a file is complete, what kind it is and how risky. Each question is small, but they sit on the path of every step, and they arrive by the thousand every day.

Kahneman [1] separates a fast, automatic System 1 from a slow, deliberate System 2. Large language models work like System 2: they reason by writing text, spend time and tokens on every question, and their confidence is not calibrated for thresholding [5]. Using them for every small judgment pays for slow thinking where a reflex would do.

TIA is a System 1 layer for machines. It reads the situation once and immediately returns, for each question, a probability distribution a workflow can threshold, rank or escalate on. When the distribution is spread out, the question goes up to System 2 (a large language model or a person). When it is concentrated, the workflow moves on without waiting.

The input is one evidence state s (text or any JSON) and N named questions q₁, …, q_N. Each question carries a type τ_i, a natural-language instruction x_i and an option set O_i:

Binary asks yes or no, optionally with a description of what counts as yes and as no. Categorical picks one of 2 to 10 labels, each described in natural language, so the label set is open: users define the labels for their own problem. Ordinal grades on an ordered scale of 2 to 10 levels, lowest first.

For each question the model returns the full distribution p(y_i | s, q_i) over O_i, not just a label. All questions are answered in the same reading of s:

Here C₁ is the cost of one shared reading and C_N the cost of asking each question in its own pass. If compute grows roughly linearly with tokens, the difference C_N − C₁ is about (N − 1)·c(|s|), because the state is read (and billed) once. The longer the state and the more questions, the larger the saving.

Training uses strictly proper scoring rules [4]: the expected score is optimal only when the model reports its true belief, so it is not rewarded for over- or under-confidence. With a target distribution π_i and a one-hot label y_i, we use the log score in the form of KL to the target distribution, the Brier score [2], and for ordinal questions the ranked probability score [3] over cumulative distributions P_ik:

The objective rewards the whole distribution, not just one correct label: the model is trained to output a calibrated probability distribution over the declared options, and is evaluated on held-out decisions that were not used in training.

Data is split into four parts, separated by situation, so every question about one state lands in the same part. Training data is decontaminated against the evaluation sets before learning.

Metrics

Accuracy (the top label matches gold), KL to the gold distribution, Brier, and expected calibration error ECE [6], plus latency with its method and hardware stated.

Dev versus test

All model selection happens on the development partition. Dev numbers drive internal decisions and are never placed next to other systems’ test numbers.

A single test read

Before reading the test partition we publicly commit a pre-registration: model, temperature, scoring code and expected result. The test partition is then scored exactly once. This has not happened yet.

Confidence intervals

Every comparison between two versions uses a paired bootstrap over the same decisions [7] and reports a 95% interval, not a single number.

Specialist and generalist

Results from a model trained on a benchmark’s train split (specialist protocol) and results without benchmark-specific training (generalist protocol) are reported separately, never mixed in one table.

ECE is never read alone: a model that returns a fixed label distribution without reading the input can still score a low ECE.

The tables in this section are not the build now served. Every number below is on the internal dev (carved from the train split): 186 cases, 930 decisions, from LocalLLaMA/typed-decisions. Measured by TIA; measurement dates: TIA 1.3 mini Sep 30, 2026, TIA 1.2 mini Sep 30, 2026, TIA 1.1 mini Sep 29, 2026, TIA 1.0 Sep 26, 2026, TIA predecessor (Sep 2026) Sep 26, 2026. Not comparable to test numbers.

KL on the dev set (lower is better)

By question type (noul is binary, choice is categorical, score is ordinal):

TIA 1.0 against the previous version

Paired bootstrap over 930 dev decisions, 2,000 resamples, 95% interval of the difference (TIA 1.0 minus the previous version).

Brier here is averaged per option, so it is on a smaller scale than the Brier column above.

Latency

TIA 1.0, the historical build measured in this report: p50 69 ms, server-side (model time), one request of 3 questions, 98 tokens, sequential, n = 200; p95 81 ms. Measured on AMD Ryzen 9 9950X3D 16-Core Processor, 32 vCPU on Sep 26, 2026. This is server-side latency for that run, not current API end-to-end latency. For recorded build TIA 1.3 mini, offline timing excludes HTTP and does not establish API performance.

See the Benchmark page for how these sit next to the dataset card

• Dev numbers are not test numbers. The dev set has 186 cases, which is small; the interval on the accuracy difference between TIA 1.0 and the previous version still touches zero.
• Dev labels come from an automated labelling process, not from human annotators, so dev accuracy measures agreement with those labels, not ground truth.
• On public JevBench items, the historical TIA 1.4 mini run scored 102/231; local experimental candidate TIA 1.5 scored 125/231. TIA’s own table computes position 79/98 and 10 significant comparisons (8 paired at Holm p < 0.05, 2 unpaired Wilson intervals). This is not an official ranking or independent audit; the candidate adoption gate did not pass.
• Calibration is measured on held-out data only. One global temperature does not calibrate every question type: for TIA 1.0 on the dev set, ECE binary 0.078, categorical 0.172, ordinal 0.222.
• A single training run; no mean and spread across runs yet.
• We are internally trialling a local distillation student model (no release name yet). The last ten short trials have not produced a stable build: the latest runs hit a gradient and KL explosion at the second training step. There is no effective trained artifact, no benchmark claim, and this model is not served on the API.
• The benchmark is English. For Vietnamese, TIA 1.0: accepts Vietnamese input; no Vietnamese-specific measurement yet. TIA 1.3 mini: accepts Vietnamese input; TIA's Vietnamese held-out set (1,144 decisions, dev): accuracy 0.658; quick 25-case sanity set: 15 of 25 correct; real Vietnamese SMS (382 messages, dev, measured Sep 30, 2026): accuracy 0.940, caught 89 of 99 scam messages, flagged 13 of 283 legitimate ones (measured by TIA on Sep 30, 2026).

Vietnamese SMS data: “Vietnamese SMS Dataset with Quality Assurance” by Tran Nguyen Thai Tuan, Le Hoang Khang, Nguyen Minh Tai, Nguyen Van Thang, Mai Hoang Dinh, Hugging Face (revision a90e2bd), licensed CC BY 4.0. Changes: mapped to TIA typed decisions, deduplicated, re-split.