Dike Model: Legal Document Typing That Knows When to Ask a Human
A compact 278M-parameter model that types legal documents across five jurisdictions in real time, matches a model roughly 100 times its size where it answers, and sends everything else to review.
- 278M
- Model parameters
- 845/s
- Documents per second
- 99.24%
- Agreement on answered
- 0 of 2,704
- Wrong types auto-accepted
A 1.1 GB file that runs on ordinary hardware
On one GPU; 30 per second on a CPU alone
Domain labels on the 62.4% it answers on its own
Document type on unseen documents
- 1The Dike model reproduces the domain labels of a reference model roughly 100 times its size, with 99.24% agreement on the 62.4% of held-out documents it chooses to answer (1,480 documents, confidence bar 0.975). The rest go to a ranked review queue instead of getting a guess.
- 2Document type is close to solved for known sources: a source prefix in the document ID determines the type for all but a handful of documents. The model is the safety net for new sources: on 2,704 unseen documents it was right 99.8% of the time, auto-accepted 97.3% of them, and made zero errors among the ones it auto-accepted.
- 3It is fast enough to sit in the ingestion path: 845 documents per second on one GPU, 30 per second on a CPU alone, with identical answers on both. That is roughly 170 times faster than the large reference model that labeled our training data.
- 4Every number here is agreement with a large reference model, not accuracy against lawyers. No human-labeled reference set exists yet, so we built the model to make its uncertainty visible and its abstentions cheap, and every response carries a provisional warning until that changes.
1. Metadata decides whether legal AI can be trusted
A legal search API is only as good as its metadata. Whether a record is a statute, a regulation, a court ruling or a treaty, and which area of law it belongs to, decides what a query retrieves and what an AI agent is allowed to cite. Across our corpus that metadata was uneven: some sources were clean, others had never been typed, and the corpus keeps growing. A large language model can type documents well, but it is slow and expensive at this scale. We wanted its judgment at the cost of a small model, in a shape a product can depend on: fast, typed, calibrated, and willing to say "I am not sure".
2. Big-model judgment at small-model speed
The Dike model gives two typed answers instead of free text: a domain (12 labels, from financial to criminal) and a document type (6 labels). To build it, a large open model, Qwen3.8-27B, first labeled tens of thousands of documents. Each document was asked several ways, so that the order of the answer choices could not sway the result, and answers that did not hold up were re-checked and given less weight. We then trained a compact 278M-parameter multilingual encoder, XLM-RoBERTa base, to reproduce those judgments. Nearly half of the labeling effort went to the documents the model found hardest, not to easy ones. The result keeps most of the large model's judgment at a fraction of its cost.
- Random and balanced sample 30,000 docs (36.6%)
- Hard cases, Egypt and Saudi Arabia (first passes) 27,577 docs (33.6%)
- Kuwait, Qatar and UAE 12,332 docs (15.0%)
- Hard cases, Egypt and Saudi Arabia (later pass) 12,116 docs (14.8%)
A random sample keeps the baseline honest; the two hard-case groups spend the labeling effort where the model was weakest. Kuwait, Qatar and the UAE were labeled separately because they had the least data.
Where the large model wavered on a document (14,050 of them), we asked it again in six different ways and weighted each label by how many of those answers agreed. Together with 12,116 more labels on hard cases, this raised the share of documents the Dike model can answer automatically by 1.4 points, at the same agreement on the ones it answers.
3. Four guardrails that make it safe to automate
Accuracy alone does not make a classifier safe to automate. What matters is what happens on the documents it gets wrong, so much of the engineering went into the guardrails around the model rather than the model itself.
- Title removed5.1%
- Cut to the first 500 characters3.5%
- 2% OCR-style character noise2.8%
- Diacritics stripped1.1%
- Western digits instead of Arabic-Indic0.6%
- Whitespace collapsed0%
1,000 documents per test. Document type changed in at most 0.3% of documents (title removed) and in none of the other cases.
If the model is at least 90% sure a document is a different type than its source says, the answer is not overwritten. It is flagged for review with the reason attached. That is how we found real treaties and founding deeds stored as legislation.
| Guardrail | What it does | Why it matters |
|---|---|---|
| Calibrated confidence | Confidence is fitted on held-out data (expected calibration error 0.020) | Thresholds mean something, and drift is measurable |
| Three actions | Every answer is accept, review or escalate | Only confident answers are ever automatic |
| ID rule cross-check | A source prefix fixes the document type; the model can only disagree loudly | Misfiled treaties and fatwas go to a person, not the index |
| "Unclear" is never auto-accepted | A dedicated label for documents that fit nothing | The model cannot hide a bad guess behind a category |
4. What it does on documents it never saw
The model rows below use held-out documents that were excluded from training; the ID-rule row covers the whole corpus. Agreement is measured on the documents the model answers on its own, and coverage is the share it chose to answer.
- Answered automatically 62.4%
- Sent to review or escalated 37.6%
Answered automatically means the model was confident enough to skip review. On that group it agrees with the reference model 99.24% of the time. Based on 1,480 held-out documents.
- Saudi Arabia
- Egypt
- Qatar
- Kuwait
- UAE
From a full-corpus shadow run in which nothing was written back. A lower rate means more documents go to human review, which is the intended behavior where the model is less sure. Kuwait, Qatar and the UAE had the least training data.
Scale starts at 90%, not zero.
Each point is a confidence bar, from 0.50 (right) to 0.99 (left). Raising the bar buys agreement at the cost of coverage. We run at 0.975. Based on 1,480 held-out documents.
Across the full corpus the model auto-accepts about 60% of domain answers and sends the rest to a review queue ranked by how uncertain it is. A reviewer's time goes to the documents where a wrong label is most likely, not to the ones the model is sure about.
| Answer | Documents | Answered automatically | Agreement on answered |
|---|---|---|---|
| Domain (bar 0.975) | 1,480 held out | 62.4% | 99.24% with the reference model |
| Document type, model alone | 2,704 held out | 97.3% | 99.8% overall, 0 wrong among accepted |
| Document type, ID rule | Whole corpus | All but a handful | By construction, from the source |
5. What an answer looks like
The API returns the same shape a language-model call would, so it slots into agent code without special handling. For each question it returns a choice, a calibrated confidence, the full probability distribution and an action. It also names the model, counts tokens, and, while the calibration is provisional, says so in a warning on every response.
| Field | Example | Meaning |
|---|---|---|
| choice | financial | The most probable label |
| confidence | 0.991 | Calibrated against the reference model |
| action | accept | accept, review or escalate |
| source | id_rule / model | Which mechanism produced a document type |
| warnings | provisional | Present until refitted on human labels |
6. Small, fast and boring to run
The Dike model is a 1.1 GB file. It answers 845 documents per second on one GPU and 30 per second on a CPU, with the same answer on both. One document takes under 10 ms on a GPU and about 30 ms on a CPU. In our parity test, 100 of 100 documents got the same answer on both. We ran the real HTTP service through a 20-check release test on 500 documents it had never seen, and it passed all 20: malformed input gets a clean error instead of a crash, the same request always returns the same answer, results do not change with batch size, and the server log never contains document text. Under 16 parallel clients, while the GPU was also busy with a training job, it held 71 requests per second at a 95th percentile of 248 ms.
- Qwen3.8-27B5 docs/sAccurate but slow, the careful path we used to label training data
- Dike model on a CPU30 docs/sNo GPU required
- Dike model on one GPU845 docs/sFast enough for the ingestion path
The Qwen figure is approximate (about 5 documents per second with option shuffling) and was measured on the same GPU while labeling. The Dike figures come from our own benchmark.
- Bad input and HTTP handling 7 checks (35.0%)
- Repeatable answers, batch and CPU-GPU parity 3 checks (15.0%)
- Rules every answer must obey 6 checks (30.0%)
- Load from parallel clients 2 checks (10.0%)
- Logs contain no document text 2 checks (10.0%)
The real HTTP service, run against 500 documents it had never seen. Every check passed; the chart shows how they split by area.
7. What this unlocks for Dike users
Typed metadata turns search from "find text like this" into "find this kind of authority". It lets queries filter by document type and legal domain across every jurisdiction on one shared taxonomy, and lets agents cite only answers marked accept while treating review and escalate as suggestions. New sources are onboarded by checking a small verified sample first, so the model is never trusted alone on something it has not seen. Because the model runs on ordinary hardware and answers in milliseconds, it is built to type documents as they are ingested rather than in a nightly batch, which keeps metadata current as the corpus grows. We are rolling it out in shadow mode first: it logs its answers without writing them anywhere.
8. Where the next gains come from
More labels stopped helping. Agreement with the reference model plateaued around 92%, and the large model itself gave the same answer across differently ordered questions only about 89% of the time. A model cannot be more certain than the labels it learns from, so the next gains come from better labels: human review of the documents where the large model wavers, not a bigger model. Extra pretraining on hundreds of thousands of our own documents added 0.6 points of agreement and 1.0 point of coverage. We measured it and kept it as an option rather than adopting it.
- Cleaner labels and more hard cases
- Pretraining on our own documents
Extra share of documents the model can answer automatically across the whole corpus, at the same agreement on the ones it answers. The pretraining step also added 0.6 points of agreement.
9. Limitations and what comes next
Every domain number is agreement with a large reference model. No lawyer has labeled a reference set yet, so we cannot state the model's accuracy against people, and we do not. The stated confidence is calibrated against the reference model and is marked provisional in every response until it is refitted on human labels. The model is weaker than an earlier version at spotting a treaty or a fatwa filed under legislation, which is why the ID cross-check and the review queue stay in place. Kuwait, Qatar and the UAE have less training data than Egypt and Saudi Arabia. The next step is a human-labeled reference set: it replaces the large model as the yardstick, refits the confidence and removes the provisional warning.