Android Bench adds new AI models as Gemini trails rivals
Google’s Android coding benchmark now uses Harbor and includes eight more AI models, with Gemini 3.1 Pro ranking fifth in the updated results.
By Hana Yoshida · Markets Reporter
3 min read
Google has updated Android Bench, its benchmark for testing how large language models handle Android app development tasks, adding eight more AI systems and a new testing framework. The results matter for Android developers because Google is trying to make AI coding tools more measurable as more software work shifts toward agent-style assistants.
According to Google, Android Bench evaluates AI agents on 100 Android development tasks. Ars Technica reported that Google launched the benchmark in March and has since added measures including cost, efficiency and support for open-weight models.
The latest leaderboard adds Claude Fable 5, Claude Sonnet 5, Claude Opus 4.8, GLM 5.2, Kimi K2.7 Code, MiniMax M3, Qwen 3.7 Plus and Qwen 3.7 Max, according to Google. Google said the update is meant to keep the benchmark aligned with current Android development workflows and to make it easier for outside developers to run and share results.
The expanded results put Google’s own Gemini models behind several rivals. Ars Technica reported that Gemini 3.1 Pro ranks fifth in the new leaderboard, trailing GPT 5.4, Claude Sonnet 5 and Claude Fable 5. Google’s posted results show Claude Fable 5 at the top with 84.5% accuracy on the benchmark.
The ranking continues a pattern from the first Android Bench release, according to Ars Technica, when OpenAI models were already ahead of Google’s models by a smaller margin. The latest additions widen the comparison set and make Gemini’s position more visible in a benchmark built around Google’s own mobile development platform.
Cost changes the picture. Ars Technica reported that Claude Fable 5 and GPT 5.5 cost more than $130 in tokens to complete the 100-task benchmark across 10 runs. Gemini 3.1 Pro scored lower but cost $87 for the same test, according to the report.
Google’s cheaper Gemini 3.5 Flash model fared poorly on cost because it took far longer to finish the benchmark, Ars Technica reported. The model posted the highest listed run cost, at $165, and required 28 hours to complete the test.
The update also changes how Android Bench is run. Google said it has moved the benchmark to the Harbor framework, a sandbox designed to let developers run evaluations, compare outputs and share results more easily.
Google re-ran earlier tests using Harbor to create a new baseline, according to Ars Technica. Because of that switch, some past scores have changed even though Google has not yet changed the underlying tasks; Google will keep the older data available in an archive.
Google is asking developers to contribute new benchmark cases and development tasks through the Android Bench project. The company has updated the Android Bench GitHub repository with the new dataset and instructions for developers who want to run tests or submit work for possible inclusion in the official benchmark.
Ars Technica reported that the performance gap is awkward for Google as it pushes more projects toward agentic development and would prefer Android developers to use Google tools. The same report said Google has reportedly offered to buy application source code from developers for AI training.
This story draws on original reporting from Ars Technica.