Replay a passed iOS check with zero model calls

Mobster keeps the steps of a passing check and what each step needs to find on the screen. When those controls still match, it replays without calling a model: no key, $0, and the same verdict from the accessibility tree. If a step has changed, Smart can heal it with your model. Here is how replay works, step by step.

What a passing run leaves behind

A check with steps in plain English runs on Mobster’s agent, called Smart, until it passes once. The check says what must be true at the end. Smart finds a way to get there. When the verdict is passed, Mobster writes down the path Smart took: the actions it executed, not the thoughts it had.

A step is a plain dictionary. This is one tap, as the replay core stores it:

{  "op": "TAP",  "app": "com.example.yourapp",  "target": {"label": "Continue", "role": "Button", "id": "/0/3/1", "frame": [0.1, 0.8, 0.8, 0.06]},  "screen": {"hash": "9f2c1e04a7b35d68", "signature": [["Button", "Continue"], ["StaticText", "Create your account"]]},  "text": null,  "commit": null}

The screen is kept as roles and labels, the same controls VoiceOver reads, and not as pixels. A button that moves 12 points or a screenshot with new artwork doesn’t change it. A step the model must redo on each run (text it made up, a list it read) is marked live, and the model does that step every time.

A path is validated whole. One unknown action or malformed target and Mobster drops the path, and the next run records a new one.

How a step finds its control

Before each action, the replay checks three things against the screen as it is now.

1. The app. The app in front is the app the step was recorded in. LAUNCH_APP and HOME need nothing.
2. The screen. The recorded hash matches, or the role and label pairs overlap by at least 0.8 (Jaccard). Once two reads in a row agree, so the screen has stopped moving, 0.5 is enough when the target is there exactly once.
3. The target. Exactly one element has the recorded label and role. The recorded tree path breaks a tie. A partial label never matches.

The third rule is strict on purpose: a heal costs a model turn, and a tap on the wrong control costs a wrong tap. Here are four screens fed to the matcher on 17 October 2026, against a recorded tap on “Continue” in a sign-up form (settled means the wait is over; rows that repeat another are left out):

same screen                    settled=False ok=Truebutton renamed to Next         settled=False ok=False "Continue" isn't on the screenbutton renamed to Next         settled=True  ok=False "Continue" isn't on the screenextra banner above the form    settled=False ok=False the screen is different (75% alike)extra banner above the form    settled=True  ok=Trueanother app in front           settled=True  ok=False com.apple.springboard is in front, not com.example.yourapp

The banner case shows the two thresholds. At 75% alike the screen is too different to go at once. A moment later the screen has settled, “Continue” is on it exactly once, and the step runs. The renamed button never matches, however long it waits.

The replay polls every 0.15 s. A step whose target never shows has drifted once the screen has stayed the same for two reads and 1.0 s has passed, or after 3.0 s at the most. Waiting longer changes nothing, so the replay doesn’t.

Typing, sends and purchases

A replay doesn’t invent text. It types text that appears word for word in the check’s steps, its link or its launch values, and search queries as they were typed. Anything else the model wrote is typed by the model on each run, which costs a model call. Masked text, such as a password or a code, never replays.

A replay follows the same safety rules as a live check. On a simulator every purchase is refused. On an iPhone a check refuses every send, post, delete or purchase. A replay never bypasses those refusals; the safety docs explain the rules.

When a step drifts: the heal

When a step’s screen has changed so that its control isn’t there, the replay works down this list, in order.

1. A step you chose to keep failing stops the replay. The check fails with class drift.
2. --no-heal, or no key stops the replay the same way. This is the mode for a strict CI job, and it needs no key.
3. The circuit breaker. If healing this step would make more than half the recorded steps healed, the replay stops and the check fails. The breaker is checked before a heal starts and again before the action that would cross it.
4. Smart, for that step only. The model gets the check’s own request, is told which recorded step changed and why, and sees the replayed steps as its history. After each action the replay compares the screen with the drifted step and the next four. At the first match it cancels the model and takes over.

Every heal becomes an entry in the run’s results.json and in a file beside your check, so it shows in the pull request. This is the file the heal writer produced on 17 October 2026 for a renamed button, trimmed to the first heal and without frames:

# Mobster heal diff: steps of this check changed on screen and Smart healed them. Review them.# A heal never changes an expectation. The check file itself is untouched.#   Accept:        mobster test heal --accept signup.yaml#   Keep failing:  mobster test heal --reject signup.yamlversion: 1check: signup.yamlheals:- step: 1  from: Tap "Continue"  to: Tap "Next"  why: '"Continue" isn''t on the screen'  old:  - {op: TAP, label: Continue, role: Button}  new:  - {op: TAP, label: Next, role: Button}

mobster test prints the same thing in the terminal:

✓ passed  A new user signs up  on iPhone 17 Pro, iOS 26.4    healed step 1: Tap "Continue" → Tap "Next"1 passed in 12 s, $0.00, 1 replayed with no model1 healed step waits for review: mobster test heal

You choose what a heal means:

$ mobster test heal                    # what healed, with the frames before and after$ mobster test heal --accept signup    # keep the healed steps$ mobster test heal --reject signup    # the recorded steps come back, and fail when they change again

Choose --reject when the change is a bug. While a heal waits for review, later runs replay the healed steps at no cost. The check file itself is never edited, and a heal never touches an expectation. A real bug still fails, because the verdict is a separate step.

The verdict is not part of the replay

The replay only gets the app to the screen the check cares about. Then the check’s expect list runs against the accessibility tree: text, value, count, visible. Mobster reads the tree, waits 0.3 s, reads again, and decides on a settled read. No model, no OCR and no pixel comparison is involved, and what a model says about its own run is never an input. AI end-to-end tests for iOS covers the assertions in detail.

That separation is what makes replay safe to trust. If the model were also the judge, a replay of its own steps would prove nothing. Here the steps are a way to drive the app, and the tree is the judge.

When Mobster doesn’t know what happened

Sometimes a tap is sent and the phone doesn’t answer. Mobster can’t tell whether the tap landed, so it does not send it again. An action that raises, a text field that doesn’t confirm what was typed, a failed read after an action, or a heal that ends with an unknown action all end the same way: the check is couldnt_run with the class unknown_outcome. There is no heal, no next step and no retry by the suite. You look at the app before you run it again.

The agent loop under Mobster follows the same rule everywhere. Every action is written to a local journal before it’s sent, and a ledger refuses to send the same side effect twice. Both are MIT-licensed code in Mobster’s open-source engine: mobile_agent/journal.py and mobile_agent/effect_ledger.py. The replay core ships inside Mobster for Mac, not in that repository, so read it here and in the docs.

Where the recorded path lives

The path is a JSON file in .mobster/cache/replay/, named <identity>-<env>.json, in a folder only you can read (0700, files 0600) and written atomically. Git ignores it.

• Identity is what the path replays: the check’s steps, the bundle, the app file name, the launch arguments, the environment, the link and the reset. It is not the check’s name and not its expectations, so you can reword an assertion without losing the path.
• Env is where it was recorded: the app’s version and build, the iOS major version, and simulator or iPhone.

A new build or a new iOS major version finds no path, so the next run records a fresh one and says why: “the app changed (recorded on 1.0 (1), this is 1.1 (2))”. Going from iOS 26.4 to 26.5 keeps the path, and a heal absorbs what changed. An identity keeps its four newest environments, so a matrix over several iOS versions keeps one path each. On a real iPhone the app’s version isn’t read, so the iOS major and the heals keep the cache honest there.

What it saves

On a real iPhone, 17 October 2026, an iPhone 15 Pro on iOS 26.0.1 over USB, with Smart on Claude Sonnet 5.5. The check opens a paywall from Settings in a test app. Five runs each:

The steps themselves took 10.4 s fresh and 3.3 s replayed. Opening the app (2.6 s) and reading the expectations (2.4 s) cost the same either way, which is why the whole run improves by less than the steps do.

On a simulator, an iPhone 17 Pro on iOS 26.4, the same day, against Maestro 2.4.0 running the same check as a flow on the same build: one mobster test of one check took 13.3 s at the median against 23.7 s for one maestro test (6 interleaved pairs). A suite of 20 took 166.6 s against 190.8 s (two interleaved batches of 10). Every replay and every flow passed. Maestro can’t run a flow on a physical iPhone, so there is no iPhone figure to set beside it.

Mobster’s scripted benchmark (scripts/bench_replay.py) isolates the model’s part. It runs Mobster’s own code end to end against a scripted phone, with each read and tap sleeping the time measured on a USB iPhone, and each model call sleeping 2.5 s. It is a model of the world, not a phone. Run on 17 October 2026, 5 runs each:

The cost column prices the prompts the loop really built, at 4 characters a token plus 150 output tokens a call, at Claude Sonnet 5.5’s list price with no prompt caching and no screenshots. A real run sends screenshots, so a fresh run costs more than the table says.

What isn’t measured yet: how often the cache hits across app builds. A new build records afresh by design, so a team that ships daily pays for a Smart run daily, once per check per build.

Run it

You need Mobster for Mac, a project with checks, and an Anthropic or OpenAI key for the first run only.

$ mobster test                 # Smart until each check passes once, then replay$ mobster test                 # the second run: replay, no key, $0$ mobster test --no-heal       # strict CI: replay, and fail a check whose screen changed$ mobster test --fresh         # run every step with Smart and record again

A launch-only check, one with no steps, never needed a model at all. --no-heal is the strict mode for CI: a replay that drifts fails with class drift instead of spending a model turn. Mobster vs Maestro sets the two tools side by side.