There are roughly 7,000 languages spoken on Earth. Google Translate supports about 130 of them. Most commercial MT systems handle fewer than 30 well. If you speak Yoruba, Quechua, or Khmer, your experience with machine translation ranges from 'barely usable' to 'hilariously wrong.' The uncomfortable truth is that most of the progress in NLP over the last decade has been English-first — and the gap between high-resource and low-resource languages is getting wider, not narrower.
Meta's recent push toward omnilingual machine translation — covering 1,600+ languages — is one of the most ambitious attempts to change this. But the technical challenges involved reveal fundamental assumptions baked into how we build language models, and why simply scaling up doesn't solve the problem.
What Makes a Language 'Low-Resource'
In MT research, a 'high-resource' language pair means you have millions of parallel sentences — text that's been professionally translated between the two languages. English–French, English–Chinese, English–Spanish: these pairs have enormous parallel corpora from the EU Parliament, the UN, news organizations, and decades of professional translation. Models trained on these pairs work remarkably well.
A 'low-resource' language might have a few thousand parallel sentences, or none at all. Fon (spoken by ~2 million people in Benin) has virtually no parallel data with English. Bambara (spoken by ~14 million people in Mali) has slightly more, but still orders of magnitude less than what conventional MT systems need. For many languages, the largest available text corpus is a Bible translation and maybe some Wikipedia articles.
The resource gap isn't just about data volume. It's about data diversity. Even when parallel text exists for a low-resource language, it tends to be concentrated in religious texts or government documents. The model learns to translate formal, repetitive prose but falls apart on casual conversation, technical terminology, or anything that deviates from the narrow domain it trained on.
Why You Can't Just Scale Your Way Out
The instinct in modern ML is: more data, bigger model, better results. This has worked spectacularly for English-centric tasks. GPT-style models trained on trillions of English tokens produce remarkably fluent text. But this approach has three critical failure modes for low-resource MT.
- The data simply doesn't exist. You can't scrape parallel Fon–English text from the internet if nobody has ever published significant amounts of it. Web scraping, which powers most large-scale MT training sets, is inherently biased toward languages with large internet presences.
- Tokenization breaks down. Most language models use tokenizers trained predominantly on English (or a handful of high-resource languages). When you feed Amharic script or Burmese text through a BPE tokenizer trained on English, it fragments characters into absurdly long token sequences. A single Amharic word might consume 8-12 tokens. This means the model uses most of its context window just encoding the input, leaving little capacity for actually understanding it.
- Transfer learning has limits. Multilingual models like mBERT or XLM-R show that training on many languages can help low-resource ones — the model picks up structural similarities. But this transfer is strongest between related languages. A model that knows French well can transfer some of that knowledge to Haitian Creole. It transfers almost nothing to Mandarin or Navajo.
The Pivot-Language Problem
Most MT systems that claim to handle hundreds of languages actually route everything through English. Want to translate Swahili to Thai? The system translates Swahili → English → Thai. This 'pivot' approach is practical — you only need to build translation models to and from English — but it introduces compounding errors and a subtle form of cultural flattening.
When you pivot through English, you lose concepts that don't map cleanly to English. Japanese has multiple levels of politeness encoded in verb forms. Yoruba has tonal distinctions that change meaning. Tamil has an inclusive vs. exclusive 'we' (whether the listener is included). When these pass through the English bottleneck, the information is lost — English doesn't encode these distinctions, so the model has no way to preserve them.
Direct translation between non-English language pairs — Swahili directly to Thai, without the English detour — preserves more information. But building direct translation models for every possible pair is combinatorially impossible. With 1,600 languages, you'd need 2.56 million directed translation pairs. Even if you only handle the 100 most-spoken languages directly, that's still 9,900 pairs.
How Modern Approaches Are Different
The current generation of massively multilingual MT models takes a fundamentally different approach than the pivot strategy. Instead of building separate models for each language pair, they train a single model that learns a shared representation across all languages simultaneously. The key innovations fall into a few categories.
Language-Agnostic Tokenization
The tokenizer problem is being addressed by training tokenizers on balanced multilingual corpora rather than English-heavy ones. Meta's approach uses a character-level fallback that ensures no language gets pathologically long token sequences. SentencePiece models trained with explicit language balancing produce much more equitable tokenization — a Yoruba sentence and an English sentence of similar meaning consume roughly similar numbers of tokens.
This matters more than it sounds. If your tokenizer is 4x less efficient for language X, your model effectively has 4x less capacity to process that language. Fixing tokenization is the single highest-leverage improvement for low-resource language performance.
Mining Parallel Data From the Wild
One of the cleverest technical contributions is automated parallel data mining. The idea: train a multilingual sentence encoder that maps sentences from any language into a shared embedding space. Then crawl the web and find sentences in different languages that map to similar vectors — these are likely translations of each other.
This technique, pioneered in tools like LASER and extended in more recent work, has extracted hundreds of millions of parallel sentences from web crawls like CCNet and OSCAR. It's noisy — maybe 20-30% of the extracted pairs are genuinely parallel — but filtering heuristics improve precision, and the sheer volume compensates for the noise. For some languages, this automated mining has produced more parallel data than all previous human-curated datasets combined.
Back-Translation and Self-Training
Back-translation is a technique where you use your existing (imperfect) MT model to translate monolingual text into the target language, then use those synthetic parallel pairs to train a better model. It's bootstrapping — and it works surprisingly well.
The cycle goes: train initial model on whatever parallel data exists → use it to translate monolingual data → filter out bad translations → retrain on the combined real + synthetic data → repeat. Each iteration improves the model, which produces better synthetic data, which further improves the next iteration. For languages where you start with only a few thousand parallel sentences, back-translation can effectively multiply your training data by 10-50x.
# Simplified back-translation loop
def back_translate_cycle(model, parallel_data, monolingual_target, rounds=3):
for round in range(rounds):
# Generate synthetic source from monolingual target text
synthetic_pairs = []
for target_sent in monolingual_target:
source_sent = model.translate(target_sent, direction='reverse')
score = model.score_pair(source_sent, target_sent)
if score > QUALITY_THRESHOLD:
synthetic_pairs.append((source_sent, target_sent))
# Combine real and synthetic data
combined = parallel_data + synthetic_pairs
# Retrain model on combined data
model = train_mt_model(combined)
print(f'Round {round+1}: {len(synthetic_pairs)} synthetic pairs added')
print(f'BLEU score: {evaluate(model, test_set)}')
return modelEvaluation Is Harder Than You Think
BLEU scores — the standard metric for MT quality — have serious problems for low-resource languages. BLEU measures n-gram overlap between the model's output and a reference translation. It works reasonably well for English because English has relatively fixed word order and limited morphology. But for agglutinative languages like Turkish or Finnish, where a single word can encode what English expresses in a full phrase, BLEU penalizes valid translations that use different morphological forms.
There's also the reference problem: who writes the reference translations you evaluate against? For high-resource languages, you have professional translators. For many low-resource languages, the 'gold standard' reference translations were produced by missionaries, government translators working in formal registers, or graduate students. These references may be technically correct but stylistically unnatural, and a model that produces more natural translations actually scores lower.
Newer metrics like COMET and BLEURT use neural models to estimate translation quality and correlate better with human judgments. But they're also trained primarily on high-resource language data, so they may not generalize well to languages with very different structures. Some teams have started doing human evaluation with native speakers for their most critical language pairs, but this doesn't scale to 1,600 languages.
The Cultural and Ethical Dimensions
Building MT for 1,600 languages isn't just an engineering challenge. It raises questions about who benefits, who decides how languages are represented, and what happens when models encode incorrect or biased translations.
For many endangered languages, the primary speakers are elderly community members in rural areas. They're not the ones using machine translation APIs. The immediate beneficiaries are more likely to be researchers, NGOs, and governments — which can be good (better access to information) or problematic (surveillance, forced assimilation) depending on context. MT for Indigenous languages developed without community input has a troubled history.
There's also the question of language standardization. Many low-resource languages have significant dialectal variation and no single 'standard' form. When an MT model picks one dialect as canonical (usually whatever was most represented in the training data), it implicitly marginalizes speakers of other dialects. This isn't hypothetical — it's already happening with better-resourced languages. Arabic MT models typically handle Modern Standard Arabic well but struggle with Egyptian, Levantine, or Gulf dialects that hundreds of millions of people actually speak.
What This Means for Developers
If you're building software that serves a global audience, the state of MT has practical implications for your architecture and product decisions.
- Don't assume MT quality is uniform. Your app might use Google Translate or a similar API for localization. The quality for French is excellent. For Amharic, it might be borderline unusable. Test with native speakers for every language you claim to support, not just the top 10.
- Design for MT failure gracefully. Show users the original text alongside translations. Let them flag bad translations. Don't hide the fact that content is machine-translated — users will figure it out anyway, and they'll trust you less if you pretended it was human-quality.
- Consider the tokenization tax. If you're using language models (not just MT) in a multilingual context, be aware that non-English languages consume more tokens. Your 4K context window holds significantly less Thai or Arabic text than English. Budget accordingly.
- Invest in multilingual test data. The hardest part of supporting low-resource languages isn't the model — it's knowing whether your output is correct. Build relationships with native speakers who can validate quality. Automated metrics will mislead you.
The Road Ahead
The push toward omnilingual MT is genuinely exciting, even with all the caveats. Five years ago, building a translation model for a language with 10,000 parallel sentences would have been a research curiosity. Today, techniques like back-translation, multilingual transfer, and automated parallel mining make it feasible — not perfect, but usable.
The remaining challenges are as much social as technical. Getting training data for endangered languages requires community partnerships, not just web scraping. Evaluating quality at scale requires new metrics and native speaker involvement. Ensuring that MT tools actually serve the communities who speak these languages — rather than just ticking a coverage checkbox — requires ongoing engagement.
But the direction is right. Language shouldn't be a barrier to accessing information, and the fact that we're even attempting to build translation systems for 1,600 languages — rather than optimizing the same 30 over and over — represents a meaningful shift in priorities. The engineering is hard. The tokenization problem alone took years to properly identify and address. But for the billions of people whose languages have been ignored by the tech industry, this work matters more than another fraction-of-a-percent improvement on English–French BLEU scores.