How well do Claude and GPT handle Georgian? Five tasks written by Giorgi Kobaidze, a native speaker, and reviewed by a language specialist. Every model answers directly, without hidden thinking, under identical conditions.
Average score across all 5 tasks, 0 to 1. Each level pairs comparable models.
Brighter cells mean higher scores. Numbers and verbs keep only the hard items left after trimming, so their scores are lower by design.
US dollars per complete task run, in Kaggle model credits. Hotter cells cost more.
All 5 tasks. Input is our prompts as each vendor counts them; output is the models' answers, with no hidden thinking.
Average input tokens per call for identical prompts. Same text, different bill.
Share answered exactly right. Georgian counts in twenties, and ordinals add the prefix მე- inside the word or before the digits.
All wrong numbers answers of the compared models, sorted into types automatically, with one real example each.
Georgian verbs mark who does what to whom inside the verb itself. Share answered right.
Two articles with 25 gaps each. Every square is one gap; lit means the exact right word form.
Hand-picked items with every model's actual answer. Lit answers are right; pink ones are wrong.
Every item was written by hand by Giorgi Kobaidze, a native Georgian speaker: no machine-generated or translated text. The full sets were then trimmed to the items that actually separate models, because every question costs model credits. Easy items were removed using a thinking model (Gemini 3 Flash) as a probe; every item it got wrong was kept.
The full, untrimmed sets are kept in the repository's archive/ folder for later testing.
Task scores run from 0 to 1. A model's average is the plain mean of its 5 task scores, each task counting equally.
Kaggle provides model credits of $10 per day. Before each call, Kaggle reserves the worst possible cost (the model's maximum answer length × its price) and refuses the call if that does not fit, even when the real answer costs a fraction of a cent. That shaped the benchmark: trimmed item sets, two parallel calls instead of four, the output limit, and no thinking. The Cost & tokens tab shows where every dollar went.