A translation can sound perfect and still tell someone to do the wrong thing. Add or remove a single 'not', move a deadline, confuse a place name with an instruction, and fluent language becomes a very convincing error. That is the part of multilingual AI I find harder to dismiss than awkward grammar.
The usual product demonstration asks a chatbot to greet the audience in several languages. It is easy to be impressed. But greeting someone and helping them understand an eviction notice, a medical appointment, a disputed charge or a university admission condition are fundamentally different tests. In the second set of tasks, meaning has consequences.
The intended instruction
You may collect the item today.
The mistaken instruction
You may not collect the item today. One word changes what the reader is allowed to do. A translation benchmark that rewards overall fluency can miss precisely this kind of failure.
Take a simple service interaction: a parent calls a school to ask whether an application needs to be submitted before Friday or can be brought in on Friday. The distinction may depend on a local expression for the deadline and on whether the speaker is asking for permission or stating a fact. A transcription system can recognize most of the words correctly and still invert the answer. What looks like a minor language-model mistake is, for that family, an avoidable journey or a missed opportunity. This is why the final instruction matters more than the apparent elegance of the transcript.
That example is deliberately in English rather than a claimed translation from a language I have not independently validated. The point is about semantic obligation: who must act, what is forbidden, what is uncertain and when a deadline applies. These are the details that a useful translation system must preserve even when its output sounds less polished.
One practical way to investigate the gap is to separate what was said from what the system thinks the speaker meant. In a real appointment workflow, the original recording, recognized words, extracted date, named location and final instruction are different pieces of evidence. A single confidence number for the whole sentence is too crude. If the date seems certain but the location is ambiguous, the system should preserve that distinction. The useful interface may be a simple confirmation question, not another paragraph generated in perfect prose. Reliability grows from recognizing the shape of uncertainty.
People do not speak in benchmark sentences
A real service desk hears voice notes from traffic, nicknames for neighborhoods, abbreviated business terms and numbers spoken in several different ways. A person may begin a sentence in one language, use another for a price or technical term, and finish in the first. Even within the same named language, region, age and profession change the vocabulary. Treating language selection as one button at the beginning of a conversation flattens all of that variation.
Code-switching also complicates what engineers regard as a single unit of work. A call might contain a product name in English, a place name in Yoruba, and an ordinary instruction in another language variety. If the system normalizes everything to the dominant language before it has resolved entities, it may destroy the evidence needed to interpret the request. A safer pipeline preserves the original audio or text under an appropriate retention policy, separates entity extraction from fluent rewriting, and lets people confirm the details that could affect their next action.
This is not a problem limited to Africa, although Hausa, Yoruba, Igbo and other African languages make the gap especially visible. It affects Indigenous languages, regional dialects, migrant communities and multilingual countries everywhere. UNESCO's work on multilingualism is a reminder that language access is a participation issue, not an interface flourish. 12
The underlying dataset is equally important. A collection of polished newspaper paragraphs will not, by itself, model noisy calls, everyday code-switching or local administrative terminology. More data is not always the answer, especially where the available material is private, culturally sensitive or collected without permission. Community participation has to begin before annotation and evaluation; projects such as Masakhane show why the people whose language is being modeled should help shape the research itself. 3
Evaluation should also record who pays for the error. A typo in a casual chat and a missed instruction on an application form may look similar in a word-error-rate dashboard, but they impose different costs. A multilingual service should describe its risk categories before choosing a threshold: what requires confirmation, what may be auto-completed and what must be escalated. This is especially important where the person receiving the answer cannot easily inspect the original language. Confidence should never be used as a substitute for an understandable correction path.
Measure the mistake that changes the outcome
Who gets to declare an evaluation representative is a question worth asking before the dashboard is built. A dataset sampled primarily from educated urban speakers may look diverse in a spreadsheet but say little about someone using a different dialect, age-specific vocabulary or an inexpensive microphone. When errors are concentrated in less represented groups, a single average score can encourage premature deployment. The review should show performance by task, speech setting and language variety wherever that is ethically and statistically practical, while avoiding the publication of sensitive personal recordings.
Imagine a multilingual appointment assistant. An average accuracy number is comforting until the assistant repeatedly transcribes one rural place name incorrectly. The model may score well on the common cases while directing a small number of people to the wrong location. A more responsible evaluation would separate dates, amounts, proper names, negation and required actions, then examine failure rates across speech varieties and circumstances.
That is harder than comparing a few demo screenshots. It requires representative tasks, consented examples, skilled human review and the willingness to pause features that are not reliable enough for their purpose. General evaluation initiatives can help establish discipline, but local deployment testing still matters. 45
There is an appealing temptation to solve every language gap by training a bigger general model. Sometimes broad multilingual training helps; research into low-resource languages has expanded the possibilities considerably. 6 But many organizations need something more specific: appointment transcription, authorized document search, short-notice translation or a bilingual interface for a known workflow. Those products can combine language recognition, retrieval, deterministic checks and human review without pretending one model should be trusted with every decision.
There is no shame in a product admitting that it cannot safely resolve a request. A carefully designed fallback might let the user choose from recognized location names, repeat a disputed amount or speak with a staff member. The key is that uncertainty should alter the workflow; it should not merely be disclosed in tiny text after the assistant has already acted. Models can be excellent drafting and retrieval tools while still requiring deterministic checks on dates, account numbers, prices and legal obligations. The appropriate level of automation depends on the cost of a misunderstood instruction.
The feedback loop deserves as much attention as model selection. When a user corrects a name or clarifies a misunderstood request, that signal should reach the people responsible for the product without casually collecting private conversations for unlimited training. Organizations need a clear retention policy, consent choices and a route for language communities to challenge recurring errors. They also need to publish limitations in words their users can understand. A list of supported languages without the tasks that were tested is a marketing claim, not an assurance that someone can safely rely on the service.
Context is what makes the system usable. A financial term may have a formally correct translation nobody uses in a local market. An agricultural advisory may describe equipment unavailable to the people reading it. A government notice may use place names, abbreviations or obligations that require specialist interpretation. Evaluators should include teachers, public servants, technicians and ordinary language users, not just researchers who can read the benchmark dataset.
Language support should mean a person can complete the task without being asked to absorb the risk of the machine's confidence.
There is an organizational dimension that is easy to miss. Language support is not finished when a model team uploads new weights. Someone has to handle complaints about repeated errors, update specialized terminology, decide when a domain has changed, and review whether the system still works for people who were not well represented at launch. That is continuous product stewardship, and it costs money. A language that appears on a marketing page without a maintenance and evaluation plan is not being supported with the seriousness that a public-facing service deserves.
That is a much more demanding standard than 'the chatbot speaks this language'. It is also a more interesting ambition: technologies that understand enough to help, recognize when they do not, and make it possible for the people being served to correct them.
Sources and further reading
