AI Agents ACT NOW OPPORTUNITY

Google's open-source AI phone tester for Android app teams

ShiniFrontiers · Source: Google

Section 1 of 6 FREE PREVIEW

What problem does it solve?

ARTEMIS is a free, open-source AI agent from Google that tests Android apps on a real phone, the way a human tester would. Someone types an instruction in plain English, such as "open Settings and check the battery level is shown", and ARTEMIS taps through the app and reports whether it worked.

A. Plain-English tests — for app testing (QA) teams

Problem: Before each release, someone has to check the app still works. A tester taps through every screen by hand, or an engineer writes test code, step by step, for each check.

What it does: The tester writes each check as one English sentence. ARTEMIS carries it out on a test phone, or on an emulator (a phone simulated on a computer), and answers pass or fail with a record of every step.

B. A phone for the coding assistant — for app developers

Problem: Many developers now use an AI coding assistant, such as Claude Code or Cursor, that writes code for them. It can change the app, but it cannot see or touch a phone, so the developer still installs, taps through and screenshots each new version by hand.

What it does: ARTEMIS plugs into the assistant through MCP, a standard connector for AI tools. The assistant can then install the new version on a test phone, try it out, and bring back screenshots and the phone's error log.

See the difference

The job	Today	With ARTEMIS
Plain-English testsCheck that the Settings screen shows the battery level and nothing crashes	A tester taps through by hand, or an engineer writes test code	One English sentence; ARTEMIS answers pass or fail and keeps a record
Plain-English testsTest a screen the app draws itself (for example, built with Flutter), where the phone cannot list the buttons	Test code that finds buttons by their hidden names finds nothing to tap	ARTEMIS reads the words on screen and looks at the picture to find the button
Plain-English testsRun a check of 100 or more steps, or watch one screen for a long time	A person repeats it, or sits watching the phone	In careful mode, a slower setting that plans and double-checks, ARTEMIS works through a written plan and checks the result
A phone for the coding assistantA developer asks the coding assistant: build the app, log in with a test account, look for unexpected popups, send screenshots	The developer stops coding to install, tap and screenshot by hand	The assistant does it on the connected test phone and sends the screenshots back

The first and last rows are the project's own example instructions; the middle two describe features the project lists.

What it saves: Tester and developer time spent tapping through the same app screens by hand.
What it costs: The software is free. Each step uses a paid AI service (Google's Gemini by default); the project does not say what a test costs.
How ready: Working in the real world

Version 1.0, installs with one script, Android only. Its near-perfect test score is the project's own; nobody else has checked it.

Join free to unlock: how it works, a step-by-step setup with its costs, what you gain, and where it falls short, for every Friday idea.

Join free to read it all →

Section 2 of 6 MEMBERS

How does it work?

Join free to read this section

Google's open-source AI phone tester for Android app teams — how it fits together.

Join free to unlock: how it works, a step-by-step setup with its costs, what you gain, and where it falls short, for every Friday idea.

The big picture

Picture a careful new tester handed a written instruction: they look at the phone, tap, look again, and report what happened. ARTEMIS is that tester, run by an AI that reads pictures as well as text (a multimodal model). It looks at the phone's screen before every tap instead of replaying a recording, so a button that has moved can still be found.

How ARTEMIS fits into an app team's testing setup

An English instruction travels down from the team's tools to the phone. Green is new. Screenshots of the phone leave the company, to the AI service.

Request. Test suite or AI coding assistant: the QA team's Python tests and CI pipeline, or Claude Code, Antigravity, Cursor, Codex or Windsurf. ARTEMIS connectors: one plugs into the team's automated tests; the other gives AI coding assistants five phone tools: start a test, follow it, see the screen, review a run, fix setup. Tests are written as English sentences from tools the team already uses; the client has no other software to install.

Agent. ARTEMIS: looks at the screen, decides the next tap or typing, checks the result, and keeps a record of every step. The project says it finished 99 or more of every 100 tasks in a public benchmark of everyday apps. Nobody else has checked this.

Device. A computer the phones plug into: runs ARTEMIS; its start script installs the other software it needs. Android test phone or emulator: with USB debugging, a developer setting, switched on. Accessibility Helper: a small accessibility service ARTEMIS installs to read the screen layout; it listens only on the phone. Several phones can run tests side by side. Pre-installing the helper saves about 3 seconds on the first task.

ARTEMIS: technical view

Fast mode or careful mode?

ARTEMIS runs each test in one of two modes, chosen by whoever starts the test; fast mode is the default. Fast mode is like one tester working alone at speed: a single AI looks at the screen and taps, with no plan and nobody checking. Careful mode is like a small team: one AI writes a plan, a second carries it out and checks each tap before making it, and a third reviews the result at the end.

	Fast mode (Flash)	Careful mode (Pro)
How it works	One AI looks, thinks and acts, step after step	Three AI roles: one writes a plan, one carries it out, one checks the result
Speed per step	About 3-5 seconds	About 15-40 seconds
Safety checks	None before a tap; no final report	Each tap checked before it happens; a final review against the goal
Best for	Short, routine checks	Checks of 100+ steps, watching a screen for a long time, digging into a failure

Why not just keep writing test scripts?

A test script is code that taps through the app in fixed steps. Scripts are fast, cost nothing per run and do exactly the same thing every time. ARTEMIS does not replace them: its own guidance tells the coding assistant to try the app with ARTEMIS first, then write normal test scripts from what it saw.

	Test script	ARTEMIS
How a test is written	❌ Code, step by step	✅ One English sentence
Screens the app draws itself	⚠️ Often cannot find the buttons	✅ Reads the words and the picture to find them
Speed per step	✅ No AI wait	⚠️ About 3-5 seconds (fast mode) or 15-40 seconds (careful mode)
Cost per run	✅ No AI fee	❌ A paid AI call at every step
Same result every run	✅ Yes	⚠️ An AI judges each step, so runs can differ

How accurate is it?

The project says ARTEMIS finished 99 or more of every 100 tasks in AndroidWorld, a public benchmark from Google Research: 100+ multi-step tasks across 20+ everyday apps. That is near-perfect, but the project does not say which AI model scored it, and nobody else has repeated it. The business's own app, with its logins and popups, is the test that counts.

Part A: how a plain-English test runs

Who	What happens	In everyday terms
1. The QA engineer	Writes the task and the expected result in a Python test	Writes a checklist for a new tester
2. ARTEMIS	Plans, then taps and types through the app on the phone	The tester works through the checklist
3. ARTEMIS's checker	Compares what happened with the original goal	A supervisor checks the work
4. The test suite	Receives pass or fail, the phone used, and an ID to replay the run	The signed checklist lands on the desk

Part B: how the coding assistant uses a phone

Who	What happens	In everyday terms
1. The developer	Asks the coding assistant, in chat, to try the new build on the phone	Asks a colleague to check something on their phone
2. The coding assistant	Starts the task in the background through MCP	The colleague goes off to do it
3. ARTEMIS	Drives the phone and sends a desktop notification when done	The colleague calls back: done
4. The coding assistant	Pulls the screenshots, the record of every step and the phone's error log (Logcat)	The colleague hands over photos and notes

The five tools: run a task, manage it (status, stop, add an instruction), read the phone's screen, inspect a finished run, and diagnose the setup.
Several phones can run at once; each phone is locked to one task at a time.

Join free → I have an account

Section 3 of 6 MEMBERS

How to set it up, and what it costs

Join free to read this section

Google's open-source AI phone tester for Android app teams — steps and cost.

Join free to unlock: how it works, a step-by-step setup with its costs, what you gain, and where it falls short, for every Friday idea.

Before you start: is it a fit?

	ARTEMIS
Good fit if	The business ships an Android app and checks it by hand before releases
	The app draws its own screens (for example, with Flutter), which test scripts struggle with
	Developers already use an AI coding assistant such as Claude Code, Cursor or Antigravity
	Test phones or emulators and test accounts are available
Who acts on it	Head of QA or engineering lead, with a developer to set it up
Not yet if	The app is iOS only, or test screens show real customer data that must not leave the company

Before you begin

You need	Why
A developer	To install it, add an AI key and connect it to tests or the coding assistant
An Android test phone with USB debugging on, or an emulator	ARTEMIS drives this device
A Mac, Linux or Windows computer	The phone plugs into it; the start script installs the other software ARTEMIS needs
A Gemini key, even if another provider does the thinking	The step that points at the exact spot to tap needs a Google model; OpenAI, Anthropic, OpenRouter or xAI can do the planning and checking
Test accounts	Screenshots go to the AI provider, so use test data

On its first test, ARTEMIS installs a small helper app on the phone that can read the screen (an accessibility service), and keeps the screen switched on. On a shared or personal phone, the developer adds these two lines to ARTEMIS's settings file (.env); the helper is then installed only when someone asks for it.

Part 1: Run a first task

Download ARTEMIS (the developer)
Add the AI key (the developer) Copy the template, then fill in GEMINI_API_KEY. Add another provider's key as well only if that provider should do the thinking.
Start it (the developer) On Windows PowerShell, run .\start.bat instead. Expect: a browser page at http://localhost:8000 with a device connection wizard, live screen mirroring and a prompt box.
Give it a first job (the developer) Expect: the phone opens Settings and the battery level comes back in the terminal.
Check the setup if anything fails (the developer)

Part 2: Add it to the test suite

Install the client on the test machine (the QA engineer) The client has no other dependencies; ADB, the agents and the AI models stay on the device host.
Write a test as a sentence (the QA engineer) Point the client at the computer the phones plug into, choose fast ("flash") or careful ("pro") mode, and check the result. Expect: the test passes or fails like any other, with an ID to replay the run.

Part 3: Connect the coding assistant

Install the MCP server and its rules (the developer) This sets up every supported assistant; name one instead of all, such as antigravity, to set up only that one.
Ask for a check in chat (the developer) For example: build the latest changes into an APK, install it on the connected device, open the login screen with a test account, check for unexpected popups, and return screenshots. Expect: a desktop notification when the run ends, and screenshots in the chat.
If a tool errors, ask the assistant to diagnose (the developer) The mobile_diagnose tool returns ready, degraded or blocked, with an ordered list of fixes.

What it costs

Item	Cost
The software	Free (Apache 2.0 open-source licence)
AI usage	Paid to the AI service for every step; the project gives no cost per step or per test. Fast mode uses less, and a cheaper Gemini setting is included
Hardware	A computer for the device host, plus Android test phones or emulators
People	Developer time to set it up; QA time to write and review the English tests

Join free → I have an account

	What changes	What it can mean for the business
Test writing	Tests are English sentences, not step-by-step code	QA staff who do not code can write and read tests
Coverage	Screens the app draws itself, and long checks, can be tested	Fewer releases checked only by hand
Developer loop	The coding assistant tries its own change on a phone	Fewer breaks to install, tap and screenshot by hand
Evidence	Every run keeps a record of every step, screenshots and a screen recording	Failures can be replayed instead of described

Limit	Why it matters	What to do
Android only	iOS support is on the roadmap, not built.	Keep the current process for iOS.
The benchmark is self-reported	The project does not say which AI model scored 99%+.	Measure it on the business's own app before relying on it.
Needs a Gemini preview model	Pointing at the exact spot to tap uses a Gemini robotics preview model, mainly when the screen gives no button list, such as screens the app draws itself. Other models fail at this, per the project.	Budget for a Gemini key even if another provider does the thinking.
Screenshots leave the company	Each step sends the screen to the AI provider.	Use test accounts and test data only.
Slow next to scripts	Careful mode takes about 15-40 seconds a step; a 100-step check can take over an hour.	Use fast mode for routine checks; keep test scripts for quick checks.
Fast mode skips the safety checks	Fast mode (Flash) has no plan, does not check a tap before making it, and gives no final report.	Use careful mode (Pro) where a wrong tap matters.
Installs a helper on the phone	The helper can read everything shown on the screen.	Use dedicated test phones; remove it with uv run artemis helper uninstall.
A young project	Version 1.0, with code adapted from another open-source project.	Pin one version and test before each upgrade.

Google's open-source AI phone tester for Android app teams

What problem does it solve?

How does it work?

The big picture

How ARTEMIS fits into an app team's testing setup

Fast mode or careful mode?

Why not just keep writing test scripts?

How accurate is it?

Part A: how a plain-English test runs

Part B: how the coding assistant uses a phone

How to set it up, and what it costs

Before you start: is it a fit?

Before you begin

Part 1: Run a first task

Part 2: Add it to the test suite

Part 3: Connect the coding assistant

What it costs

What do you gain?

What are the limits?

What this project can't do yet

What it does not do

Sources