sim-use

Give AI agents the ability to observe and act on iOS Simulator and Android emulator / device screens.

Observe — turn any screen into a token-efficient outline an LLM can reason about:

$ sim-use ui
App: Settings  402x874

[Top  y<120]
  @1  StaticText  "Settings"
[Content  y=120..754]
  @5  SearchField  "Search"
  @7  Button  "Sign in to your iPhone"
  @9  Button  "General"
  @10 Button  "Display & Brightness"
  @11 Button  "Wallpaper"
  ...
[Bottom  y>754]
  @43 TabBar

Act — tap any element by its alias, no coordinates needed:

$ sim-use tap @9
✓ Tap at (201.0, 452.0) completed successfully

Plan, code, verify, ship — teach this CLI to your agent and close the last gap in the agentic mobile development loop. Let agents verify what they built so you can focus on what matters.

sim-use is a cross-platform CLI that drives Apple's Accessibility APIs, the iOS Simulator HID pipeline, and Android's AccessibilityService through a single command surface. It emits a compact, agent-friendly screen description (ui) and an alias-cached tap shortcut (tap @N) so an LLM loop can observe → act in a few hundred milliseconds per round trip.

• The observe → act loop
• Why sim-use
• Install
• Platforms
• Commands
• Physical iOS devices
• Architecture
• Viewer
• Contributing
• Licence

The observe → act loop

Every interaction follows the same cycle — observe, act, verify:

sim-use ui                  # 1. read the screen
sim-use tap @9              # 2. act on what you see
sim-use ui                  # 3. verify the result

Multiple selector styles for different needs:

Selectors and explicit coordinates work in any orientation — on iOS, sim-use self-calibrates the current rotation on each command, so -x/-y / --point take the numbers ui prints. --coordinate-space native gives them in the device's native portrait points instead.

Why sim-use

• Token-efficient. The outline representation is ~16x more compact than a raw JSON accessibility tree. An LLM can read and reason about an entire screen in a few hundred tokens.
• Nothing hidden. sim-use walks the full accessibility tree including WebViews, system overlays, and embedded content — no elements are silently skipped. When the frontmost app exposes an empty tree because a remote process owns the visible UI (a system document picker, for example), ui automatically retries with cross-process discovery and flags the recovered, flat hierarchy via the advisory envelope key.
• AI-native. Designed from day one for agent loops, not human testers. Alias-cached taps (@N), structured --json envelopes with actionable hint fields on errors, and a bundled agent skill (sim-use init --client claude) that teaches your AI client the full command surface.
• Fast. A per-device background daemon amortises init cost across calls. After the first command, each observe-act round trip completes in ~300 ms.
• Cross-platform. One command surface drives both iOS Simulator and Android emulator/device. Same verbs, same flags, same --json shape — write one agent loop that works on both.

Install

Homebrew (recommended)

brew tap lycorp-jp/tap
brew install lycorp-jp/tap/sim-use

On Homebrew 6.0.5+, if you see an "untrusted tap" error, run brew trust lycorp-jp/tap first.

Notarized releases keep their upstream Developer ID signature after installation. The ad-hoc signing fallback is only used for legacy, non-notarized release packages.

Build from source

sim-use is a Swift package targeting macOS 14+, built with the latest Xcode toolchain. It links against static XCFrameworks built from Meta's idb, which are produced locally by the build script (they are large and not checked into the repository). The idb checkout generates its Xcode project with XcodeGen (brew install xcodegen).

git clone https://github.com/lycorp-jp/sim-use.git
cd sim-use

# Build the required XCFrameworks (first time only)
./scripts/build.sh dev

# Build sim-use itself
make build
.build/debug/sim-use --help

# Other Makefile targets
make test    # run tests
make clean

The XCFrameworks are built without library evolution, so their Swift modules are locked to the toolchain that produced them — re-run ./scripts/build.sh dev after switching Xcode versions.

For release packaging changes, scripts/verify-brew-install.sh --archive <tarball> --version <version> --require-developer-id tests the generated formula under a separate name without linking it into PATH. Add --download-url <release-asset-url> to test a real Homebrew download against the same archive checksum. The verifier checks that the original binary and Developer ID signature survive installation; it does not replace an existing sim-use installation.

Xcode 27 (beta)

Xcode 27 betas are fully supported from Beta 4 on (earlier betas ship no usable SimulatorKit.framework), including simulators booted while Device Hub is open — the HID transport is selected automatically per boot. Note that Xcode 27 no longer bundles Simulator.app; the one from an Xcode 26.x install still works, as does Device Hub itself.

Resize Mode and the iPhone Duo inner display are driven. Xcode 27 can put an iOS 27+ simulator into a resizable app session (Device Hub's resize button, or xcrun devicectl device appResize start), which moves the frontmost app onto a virtual Resizable display at an arbitrary size. Xcode 27.1's iPhone Duo has two integrated displays: the cover display (primary) and the inner display. sim-use reads the displays from xcrun devicectl device info displays and routes everything to the display that hosts the app: touch input (tap, long-press, swipe, touch, gesture, multi-touch, batch) goes to that display's digitizer, accessibility hit-tests (ui --point, collapsed-children recovery) run on it, and screenshot captures it. The ui header names the Duo screen (App: Settings 951x669 (landscape-left, inner display), (cover display)) or Resize Mode (display Resizable 560x874). ios fold and ios rotate use the same inner/cover names. An active inner display does not prove a fully open hinge; half-open states can use it too. These aliases require the Duo device type and a valid two-panel topology; unknown layouts keep system names. --json keeps raw display names and ids; its display object appears only off the main display. Ordinary single-screen headers stay unchanged. On a display other than the main one the orientation comes from CoreDevice instead of hit-test probes. record-video and stream-video (h264, bgra) capture that display too — the inner panel in its native portrait orientation, like the main display; a Resize Mode scene cut from its canvas. A fold during a recording keeps the display it started on and says so at the end. The screenshot-based stream-video formats read the main display only and refuse there. Explicit -x/-y / --point coordinates are the numbers ui prints there too; with --coordinate-space native they are native to the display that hosts the app (the inner panel's unrotated points; a Resize Mode scene's points). sim-use ios fold closed|open|half|<degrees> folds and unfolds the Duo and confirms which display became active; sim-use ios fold alone reports the posture.

An app whose scene is smaller than its display — an iOS 27.0 SDK app keeps out of the iPhone Duo's status bar strip, an older SDK app runs centred on the inner display — is placed by a few hit-test probes, so taps land on it in every orientation.

The display list costs one ~0.1 s devicectl query per daemon on a single-display simulator and at most one every 2 s on a multi-display one; the Resize Mode check is one devicectl device info appResize per device per 5 s. A ui (or a selector tap) whose accessibility root changed size re-reads both at once — after a fold, or when a session starts or ends. A raw -x/-y tap right after such a change, with no ui in between, can still see the previous answer for that long. SIM_USE_DISPLAY_TOPOLOGY_CHECK=0 and SIM_USE_RESIZE_SESSION_CHECK=0 turn the checks off (read by the per-device daemon when it starts). Background: issue #143, docs/ai/009-resizable-app-session/000-investigation.md and docs/ai/010-display-routing/000-plan.md.

Linux (Android only)

The Android backend also builds on Linux with a Swift 6 toolchain — iOS verbs, video capture, the Viewer and init are macOS-only. See docs/ai/008-linux-android-build/linux-guide.md for what is available, how to build and install, and reaching devices from WSL.

scripts/install-linux.sh

Agent skill

To install the bundled agent skill into your AI client's skill directory:

sim-use init                        # auto-detect installed clients
sim-use init --client claude        # non-interactive
sim-use init --dest ~/.claude/skills
sim-use init --print                # print skill content without installing
sim-use init --uninstall --client claude

Platforms

sim-use drives both iOS Simulators and Android devices / emulators through the same command surface. The device ID shape decides which backend handles the call:

• 1A2B3C4D-... (UUID) → iOS Simulator
• emulator-5554 / R5CT1ABCD12 / 192.168.1.5:5555 → Android device
• 00008130-... (8-16 hex) / 40-hex → physical iPhone/iPad (restricted verb set)

For Android, run sim-use android init --device <serial> once to install the bridge APK. See AGENTS.md for Android toolchain setup. When ADB_SERVER_SOCKET points at an adb server on another machine (tcp:<host>:<port>), sim-use reaches the bridge on that host, since that is where adb forward listens; SIM_USE_BRIDGE_HOST overrides the host.

Physical iPhones and iPads (experimental) route through the same top-level verbs — sim-use ui, `sim-use tap '#' / --lab