--- title: "Five Open Models, One Grounding Layer: What Changes When You Add Dike" date: "2026-08-13" description: "We ran five open-weight models against real MENA legal questions, raw and grounded through Dike, and logged exactly where the raw answers broke." author: "Dike Engineering" readTime: "7 min read" heroImage: "/images/artwork/citation-flow.png" tags: ["Engineering", "Grounding", "Open Models"] --- We've written before about why models hallucinate legal citations. This time we wanted the receipts. We picked five open-weight models that developers are actually reaching for right now, ran each one against a real MENA legal question, then ran the same question again with the model routed through Dike's API instead of answering from memory. We logged what changed. The full transcripts, Arabic prompts included, are live at [dike.it.com/models](https://dike.it.com/models). This post walks through what we found and why it happened. ## The setup Nothing about this test was adversarial. We didn't try to trip the models up with edge cases or ambiguous phrasing. Each question is the kind of thing a junior associate or a compliance analyst would type into a chatbot on a Tuesday afternoon: what does the law actually say about this. For each model we ran the query twice. Once cold, against whatever the model learned during training. Once through Dike, where the model's tool-calling ability retrieves the relevant statute or ruling from our indexed MENA legal corpus before it answers. Same model, same prompt, same hardware. The only variable is whether the answer comes from memorized parameters or from a document we can point to. ## Where the raw answers broke **Qwen3.6 30B**, asked about the conditions for contract dissolution under Saudi Civil Transactions Law, gave a confident, well-structured answer citing Articles 87 through 90. It laid out sub-provisions for each one: a written notice requirement, a force majeure exception, a procedural mechanism. All of it reads like real statutory language. None of it is. The actual rule lives at Article 107, and none of the surrounding articles say what the model claimed they said. **Muse Glimmer 30B** got a labor law question wrong in a way that would cost someone real money. Asked what end-of-service gratuity a worker is owed after resigning at seven years of service, it cited Article 84 and applied the general severance formula, half a month's pay per year for the first five years and a full month per year after that. It landed on 4.5 months. The actual rule is in Article 85, and it works differently for resignations specifically: a third of the gratuity between two and five years of service, two-thirds between five and ten, full gratuity after ten. At seven years, the correct answer is two-thirds, not 4.5 months calculated off the wrong formula. If a payroll team trusted the raw answer, they'd underpay someone and have no idea why. **Nemotron 3.5 Lightning**, asked for the deadline to file an annulment action against an arbitration award under Egyptian law, said 60 days without hesitation. Article 54(1) of Law 27 of 1994 says 90 days from the date the award is formally served on the losing party, a detail confirmed by Court of Cassation rulings we have indexed. Missing that window by even a day means losing the right to challenge the award at all, so a 30-day error here isn't cosmetic. > Confident and wrong is worse than uncertain, because nobody double-checks the answer that sounded sure of itself. **Granite 4.1 30B** handled a Saudi PDPL question differently. It didn't invent anything. It gave a clean, accurate six-point summary of a company's obligations under the law: consent, purpose limitation, security measures, data subject rights, cooperation with the regulator, breach notification. The problem is that none of it cited an article. You can't hand that answer to a lawyer and expect them to sign off on it, because there's nothing to verify. It sounds right, but sounding right isn't the same as being checkable. **Gemma 4 31B** ran into the same issue on a harder task: comparing how Egyptian and Saudi law treat contract dissolution for breach. Its answer was a reasonable doctrinal summary, and it wasn't wrong exactly, but it never quoted either code directly. Grounded through Dike, it pulled Articles 157 and 158 from Egypt's Civil Code and the matching Articles 107 and 108 from Saudi's Civil Transactions Law, side by side, and the two turned out to be nearly identical in structure. That's a useful finding for anyone doing cross-border work, and it's only available if the model is quoting rather than summarizing. ## Same model, different source What's notable here isn't that these models are bad. They're not. Qwen, Gemma, Granite, Nemotron, and Muse Glimmer are all strong, current releases, and in casual use most of these answers would pass as competent. The failure mode is specific: these are dense-recall systems, and legal citation is a lookup problem wearing the costume of a language problem. A model that's very good at sounding like a lawyer is not the same thing as a model that has the statute in front of it. Grounding fixed every one of these cases, and it fixed them the same way each time. We didn't fine-tune anything. We didn't swap in a bigger checkpoint. We gave the model a tool it could call, pointed that tool at Dike's indexed MENA legal corpus, and let the model retrieve the actual text before answering. Qwen's grounded response quotes Article 107 word for word. Nemotron's grounded response cites Article 54(1) and gets to 90 days. Granite's grounded response quotes Articles 19 and 20 of the PDPL directly instead of paraphrasing them. The architectures underneath are completely different, Mamba-hybrid, mixture-of-experts, dense transformer, multimodal, and the fix is the same across all of them because the problem was never architectural. > It was that none of them had a source to check against. Give any of them one, and the guessing stops. ## What this means if you're picking a model If you're evaluating open-weight models for a legal, compliance, or contract-review workload, the benchmark leaderboards won't tell you what we found here. None of the standard evals ask whether a model invents article numbers under normal, non-adversarial questioning. That's a gap worth caring about if the output is going in front of a client or a regulator. The practical takeaway is that model choice and grounding are separate decisions. Pick whichever open model fits your latency, context window, and deployment constraints, Nemotron if you need speed and a million tokens of context, Muse Glimmer if it has to run on-device, Granite if you need the compliance certifications. Then pair it with a real source layer. A bigger model without grounding still guesses. A small model with grounding cites. Each model page linked above has the full raw-versus-grounded transcript, license terms, and hardware specs, if you want to reproduce any of this yourself.