Whisper-LM with a Maya Lexicon: Correcting Yucatec Maya ASR Transcriptions Using Gemini and a Dictionary as Reference

What problem does this notebook address?

Whisper, OpenAI’s automatic speech recognition (ASR) system, does not include Yucatec Maya among its supported languages. When transcribing Maya audio, the model produces very noisy output and frequently “detects” the wrong language: in our data, Maya segments were labeled as Spanish, English, and even Japanese. This notebook explores a practical question: how much can a large language model (LLM) improve Whisper’s output if we give it a Maya lexicon as reference?

Pipeline architecture

The full flow is:

Audio → Whisper (turbo) → Rule-based normalizer → Gemini + lexicon → Normalizer → Evaluation (CER/WER)

1. Input data

The starting point is a CSV file (transcription_results41Turbo.csv) with 1,010 audio segments already transcribed by Whisper turbo. Each row includes the human reference transcription (original_transcription), Whisper’s prediction (whisper_prediction), the detected language, and initial metrics (chrF, WER, CER).

2. Maya lexicon as reference (RAG component)

A Maya–Spanish dictionary in TSV format (yua_dictionary.tsv) with 2,988 entries is loaded. A YuaLexicon class provides exact lookup and fuzzy matching (via difflib.get_close_matches), making it possible to find the closest valid Maya word for a mis-transcribed token.

3. Rule-based orthographic normalizer

Before and after the LLM pass, the text is normalized with Yucatec Maya linguistic rules taken from a reference grammar:

• Root morphology rules: for example, bin (to go) with the irregular root xi’ in the intransitive indefinite future, or positionals such as chil, kul, and wa’al that lose the l before -tal.
• “Safe” regex patterns: bins → bis, taals → taas, and verbs whose final b changes to a glottalized structure (jáalk’ab → jáalk’a’a).
• Lexicon verification: if the normalized form exists in the dictionary it is kept; if the original form was already valid, it is preserved.

4. Correction with Gemini (Whisper-LM)

Each normalized transcription is sent to Gemini 2.0 Flash with a prompt that includes up to 500 words from the Maya lexicon as an orthographic reference. The key prompt instructions are: correct the text following standard Yucatec Maya grammar and orthography, do not translate into Spanish, choose the most likely option according to the lexicon, and return only the corrected text.

5. Evaluation

Character error rate (CER) and word error rate (WER) are computed with editdistance against the human reference transcription:

What did we learn?

The improvement is marginal: LLM post-processing with a lexicon recovers some orthographic form, but it cannot reconstruct information the acoustic model never captured. With CER/WER values above 1.0, Whisper’s output is so far from the reference that text-level correction has little material to work with. The practical takeaway is that, for Yucatec Maya, the bottleneck is the acoustic model: what is needed is fine-tuning Whisper on Maya speech data, not just downstream correction.

Next steps

• Fine-tune Whisper with transcribed Yucatec Maya audio.
• Extend the orthographic rules (ORTHO_RULES) with recurrent errors observed in the transcriptions.
• Grow the lexicon and experiment with selective retrieval (sending only the entries relevant to each segment to the prompt, instead of a fixed list).

Access to the notebook

If you are interested in the Colab notebook, you can contact the author at jaziel.carballo@gmail.com.