KiwiStart
A retrieval-grounded assistant for migrants and international students in New Zealand, covering visas, IRD and tax, tenancy, banking, work rights and KiwiSaver.
Operating profile
Single-operator system running on managed infrastructure with daily cost ceilings. Answer quality, model routing and failure modes are instrumented, so degradation surfaces as a counter rather than as a complaint.
Hybrid dense and lexical search over a curated New Zealand corpus, fused by reciprocal rank so a query never returns an empty context window. Measured at 93.3% judged recall against a thirty-question gold set.
A blocking output gate on eligibility determinations and a premise check that rejects false assumptions before an answer is constructed on them. Unknown facts are elicited, not inferred.
Citations are resolved server-side and never pass through the model, which removes fabricated URLs as a failure class rather than reducing their frequency.
Personal information is redacted ahead of inference. Prompt-injection and scope controls are enforced in application code, so compliance does not depend on the model choosing to comply.
Multi-provider routing with tiered fallback and quality-signal escalation, semantic response caching, sliding-window rate limits and per-day spend breakers on anonymous and authenticated traffic.
Per-day counters on classifier fallbacks, model tier, cache hit path and stream stalls. When the classifier timed out during the August evaluation run, the fallback was recorded rather than absorbed.
Measured, and openly unmeasured
Each figure comes from an evaluation harness written for this system and run against it. Rows carry the method used, because a number produced by keyword overlap and a number produced by a model judge are not comparable and should not be shown as though they were. Current as at 16 August 2026.
On the open row. A stored result set labelled “after” turned out to have been captured thirty-two minutes before the change it was meant to measure. The figure was accurate and described a system that no longer existed, so it was retired rather than published. The re-run then failed for a better reason: the harness drives the production endpoint on the anonymous path, and the free-usage cap refused all twenty-five requests. A measurement blocked by a control that is working as designed is still a measurement I do not have. The row stays open until the harness runs under an authenticated path.
Claude’s Constitution as an acceptance test
Anthropic published Claude’s Constitution on 22 January 2026 under a Creative Commons public domain licence. It states an explicit priority order. Read as a specification rather than a statement of values, it produces testable requirements.
0 false eligibility claims across every recorded run.Claims triage
A decision workflow for retailer deduction claims — the shortage, damage and late‑delivery notices an FMCG supplier receives from grocery chains and adjudicates by hand, against a deadline, one email at a time. Built on n8n, with a language model doing the reading and application code doing the deciding. The dataset is synthetic throughout; no employer data was used at any point.
Design position
The model extracts. The code decides. No arithmetic, no threshold and no routing call is left to the language model. The validation step recomputes the claim value from the invoice record and compares it against what was claimed, because a model that is confident about a dollar figure is still only confident.
Classification into four claim types plus an explicit not‑a‑claim category, then extraction into a fixed schema. The schema is defined by attribute description rather than generated from a JSON example, because example‑generated schemas mark every field mandatory, and a mandatory invoice number forces a model to invent one instead of reporting its absence.
A claim that could not be checked is not a claim that failed a check. The verifiability gate runs ahead of any adverse decision, so missing data routes to a person rather than to a dispute letter. This was not the original design; it is the fix for the first defect found.
Claimed value is recomputed as quantity multiplied by the contracted unit price and compared within a two percent tolerance. Quantity is checked against the delivery shortfall on record, so a claim for more units than were ever short is caught by subtraction rather than by judgement.
Claims above the auto‑clear threshold reach a person by email carrying the evidence rather than the request: amount claimed against amount independently calculated, the variance, units claimed against known shortfall, and which of the seven checks failed. A malformed approval response resolves to declined, never to approved.
Where the delivery record contradicts the claim, the workflow composes the dispute letter and writes it to drafts. It does not send. The machine assembles the evidence and the wording; a person decides whether a customer hears it.
Twenty‑five columns per claim: every extracted field, each named check and its result, the route taken, the reason it was taken, and where a person was involved, their decision and the timestamp of it. A decision that cannot be reconstructed six months later has not really been made.
Six defects were found during the build, every one by executing the workflow rather than reviewing it. The most serious treated an unverifiable claim as an invalid one, and would have drafted disputes against customers from data the system never had. A second produced a confident, well‑formed, entirely empty result when a node silently discarded its input — the failure mode that most resembles success. A thirty‑case test set was designed across the four routing outcomes and the failure modes that matter: missing invoice numbers, amounts written in prose, claims exceeding the known shortfall, duplicate references, zero and negative values, remittance advice and autoreplies. Five representative cases were executed end to end — one for each routing outcome, plus a non‑claim and a claim with the amount written in prose rather than as a figure — and all five produced the expected result. The remaining twenty‑five are specified and unrun.
Operating under voluntary standards
Neither New Zealand nor Australia has enacted an AI act. Existing law still applies in full, and the voluntary standards are the benchmark regulators measure against. These are the instruments I build to.
National AI Strategy
Published July 2026. A deliberately light-touch, agile posture: no dedicated AI act, existing frameworks amended as needed, aligned to the OECD AI Principles.
Privacy Act 2020
The thirteen Information Privacy Principles already govern AI systems. The Privacy Commissioner expects leadership approval, privacy impact assessment, transparency, human review and data minimisation.
Algorithm Charter
Binding on public agencies rather than on private services. Applied here voluntarily, including embedding a Te Ao Māori perspective in how decisions are explained.
Voluntary AI Safety Standard
Ten guardrails, September 2024, voluntary at present. An Office of AI was announced on 15 July 2026 with legislation expected in early 2027.
ISO/IEC 42001 and NIST AI RMF
The management-system and risk frameworks most often named in enterprise requirements. Used as the mapping layer between this system’s controls and an organisation’s assurance model.
Assessment
Voluntary is not optional. A service producing visa-adjacent guidance for people with material exposure should meet the standard ahead of any obligation to.
In preparation
Three pieces, published on completion.
Claude’s Constitution as an acceptance test
Treating a published values document as a specification and running it against a production system, including the requirements it failed.
Making an AI answer verifiable
Retrieval, server-resolved citations, and the evaluation artefact that described a system which no longer existed.
Responsible AI without an AI act
The New Zealand strategy, the Privacy Act, the Algorithm Charter and the Australian guardrails, applied to one production service.