When an AI grades your speaking, you deserve to know how it works — and how we checked that its scores mean something. This page explains, in plain language, how the Fly High English speaking assessment listens to you, how it scores you, and how we tested those scores against recordings that had already been officially rated by trained ICAO raters.
What the assessment measures
ICAO — the international body that sets the English standard for pilots and controllers — rates speakers on six skills:
- Pronunciation — how easily you are understood
- Structure — your grammar
- Vocabulary — the range and accuracy of your words
- Fluency — how smoothly your speech flows
- Comprehension — how well you understand what you hear
- Interactions — how you manage a live two-way conversation
Our assessment scores the first five. It does not score Interactions, because a fair judgement of live conversation needs a real back-and-forth dialogue, not a single recording. Roleplay activities show you a separate, clearly labelled read-back accuracy result instead — useful feedback, but honestly not an ICAO Interactions band.
Your overall level is your lowest skill, not your average. That sounds strict, but it is exactly how official ICAO raters award levels — one weak skill is what causes problems in the air — so we apply the same rule.
How the AI evaluation works
- You record. Up to about three minutes, in your browser, and you tell us your first language.
- Your speech is transcribed word by word, with the timing of every word. Hesitations, repetitions and fillers (“um”, “uh”) are kept in — they are evidence, not noise.
- Your delivery is measured. Speaking speed, the number and length of pauses, and filler words — the raw ingredients of a fluency judgement.
- A second AI listens to the audio itself to judge pronunciation — the clarity of your sounds, stress and rhythm — rather than guessing it from the written transcript.
- An AI examiner puts it all together. It reads the transcript, the delivery numbers and the pronunciation evidence, applies the official ICAO level descriptors, and writes your report.
Your report gives you a band for each skill, short quotes from your own recording (with timestamps) as evidence, your weakest link, and three concrete things to work on next — in English and in your first language.
How we calibrated the scoring
“Calibrated” means we did not just trust the AI — we tested it against human experts and tuned it. We used 52 recordings of real pilots and air traffic controllers that had already been officially rated by trained ICAO raters, covering ICAO Levels 3 to 6 and twelve different language backgrounds.
The test was blind. Every recording was anonymised and its level label removed, so the system scored each one with no idea what the official answer was. It was also told to round down whenever it was unsure. Only afterwards did we compare its scores with the official ratings.
The results:
- Across all 52 recordings, 96% of overall scores landed within one band of the official rating.
- Most of those recordings also contained the examiner’s voice, which makes them harder to score than a real submission. On the smaller group of recordings that match what our tool actually receives — one speaker, talking on their own — every overall score was within one band, and every pass/fail call at the Level 4 line matched the official raters.
- When the system was wrong, it was wrong in the safe direction: a strong Level 5 speaker scored as a 4, never an inflated score. It is deliberately a tough marker.
The Level 4 boundary matters most, because Level 4 is the minimum operational standard for international flying — it is the pass line almost every learner is aiming for. That boundary is where the system performed best. Calibration is not a one-off event, either: we continue to test and tune the scoring as the tool develops.
Built-in safeguards
- A range, not a false-precision number. If you are between levels, your report says so — “borderline 3/4” — instead of pretending to a certainty no examiner would claim.
- It rounds down when unsure. You will never be told you are safe when you are not.
- Your accent is not the test. The ICAO standard explicitly accepts accented English at Level 4. The system judges only whether your pronunciation interferes with understanding — never whether you sound like a native speaker.
- It knows your first language. The scoring takes into account the typical patterns of speakers of Portuguese, Spanish, Mandarin, Russian, French, German, Japanese, Korean, Arabic and Italian, and your feedback comes in English plus your own language.
- It refuses to guess. If a recording is too short, too noisy or not in English, the system says it cannot score it fairly — it does not invent a band.
Your recording and your privacy
You give explicit consent each time you record. Your audio is used only to produce your feedback — it is never used to train AI models — and the recording itself is automatically deleted after 60 days. Your transcript and report stay in your account so you can track your progress. Full details are in our privacy policy.
The honest limits
Every report we produce carries the same line, and it belongs on this page too: indicative practice feedback — not an official ICAO result. This tool exists to show you where you stand, what your weakest skill is, and what to fix next — so that when you sit the real test, with a licensed examiner, nothing surprises you. It does not replace that test, and we will never suggest it does.
Ready to hear where you stand? Speaking assessments are part of Pro membership.