Separate English and Twi tests cannot validate a bilingual voice agent because they miss what happens at the boundary between the two languages. A single code-switch inside one sentence can change the caller’s intent, expose a transcription gap, and invalidate an otherwise convincing English-only demo.
In 1999, NASA’s Mars Climate Orbiter approached Mars after a journey of hundreds of millions of kilometres. The spacecraft itself had passed extensive engineering work. The navigation teams had data. Yet one interface carried a hidden mismatch: one system produced thruster data in pound-force seconds, while another expected newton-seconds.
The error accumulated during the mission. By the time the spacecraft reached Mars, NASA had lost it.
Arthur Stephenson chaired the investigation board. Its report, published by NASA, traced the failure to the unit mismatch and to the checks that failed to catch it. The mission did not collapse because nobody had tested anything. It collapsed because testing the components separately did not validate what crossed the boundary between them.
A bilingual voice agent has the same testing problem at a smaller scale. English can work. Twi can work. The handoff between them can still fail.
The demo passes until the caller changes language
Picture a support lead reviewing a voice-agent demo in Accra. The prepared English calls sound clear. The agent identifies the account issue, asks the expected questions, and produces a usable transcript.
The Twi test set also performs well. Speech is recognised, the agent responds in Twi, and the call reaches the intended outcome.
Then a caller gives one answer that begins in Twi and ends in English.
The first half provides the condition. The second half states the requested action. If the system keeps only the English portion, the transcript may look fluent while losing the fact that changes what the agent should do next. If it mishandles the switch itself, the agent may answer the wrong clause, repeat a question, or classify the call incorrectly.
That half-sentence invalidates the earlier conclusion. The monolingual demonstrations showed that two separate paths worked under controlled conditions. They did not show that one real conversation could move between those paths without losing meaning.
This is especially important in support, collections, campaigns, and account servicing. A caller may use Twi for context, switch to English for a product name, then return to Twi to correct or qualify the request. The operational question is not whether every word appears in a transcript. It is whether the agent preserves the caller’s meaning across the switch.
The same risk appears when a Twi request disappears from an English transcript. A clean-looking record can conceal the missing instruction.
Test the boundary as its own system
A useful bilingual evaluation set needs mixed-language calls, not two monolingual folders placed beside each other.
Build test utterances around the points where meaning commonly crosses the language boundary. Put the account context in one language and the requested action in the other. Switch immediately before a negation, correction, amount, date, name, consent statement, or opt-out. Include turns where the agent must reply in the language the caller has moved into, and turns where it should confirm before acting.
Then evaluate the complete call record:
- Did speech recognition preserve both language segments?
- Did the agent connect the two clauses as one intent?
- Did the response address the full request?
- Did consent or opt-out language survive the switch?
- Does the audit trail show what happened without filling gaps with a confident guess?
These tests should reach the same lifecycle records used for actual calls. A transcript-only review can miss whether the call connected, ended early, triggered an action, or consumed billable minutes. Call truth matters alongside language accuracy.
Asenda Talk has native Twi speech recognition and synthesis fine-tuned in-house, with Vapi orchestrating the assistant runtime. The platform also records telephony lifecycle events, consent, opt-out status, and call audit data. Those capabilities make mixed-language evaluation possible, but they do not make every bilingual interaction correct by default. The system remains in active early access, and more African languages are still in progress.
Keep the demo inside the evidence
A credible demo should state exactly what was tested. “English calls passed” is useful evidence. “Twi calls passed” is useful evidence. Neither supports the broader claim that spontaneous Twi-English code-switching has been validated across accents, call conditions, intents, and operational workflows.
Outbound calling requires another boundary check. Asenda Talk includes the webhook pipeline, metered per-minute billing, and an operator-controlled real-money gate, but the live telephony-provider decision has not yet been made. Until that decision and the relevant live testing are complete, a demo should not imply production-ready outbound coverage.
The Mars Climate Orbiter investigation did not treat the failed interface as a minor detail beside two functioning systems. The interface was where the mission was lost.
For a bilingual voice agent, the language switch is an interface. Put mixed-language sentences into the evaluation set before approving the demo, preserve the complete call record, and make every product claim no broader than the calls that actually passed.
Comments
No comments yet.