Everywhere
Jevie's hands on your phone
Say on my phone, …
and Jevie does it in your phone's apps themselves, tapping, typing and scrolling as you would.
FreePhoneThe Jevie app on AndroidJevie turned on in Settings › AccessibilityYour Mac, paired
Use itHow it worksLimitsWhere it's kept
Use it
on my phone, text mom on WhatsApp saying I'm running late
· Jevie opens WhatsApp, finds Mom, types the message and sends ituse my phone to search esp32 in Chrome
· Chrome opens, the words go in its search box, the results come upon my phone, turn on dark mode in YouTube
· the app's own setting, found in its menusphone: open Spotify and play my Discover Weekly
- "on my phone, reply to Anna \"on my way\"" · words in quotes are typed exactly
The first time, Jevie opens Settings › Accessibility. Turn on Jevie there, then say it again.
While it works, the line under the field says each step (tap: Search
, type into: Message
). When it finishes, Jevie says the outcome aloud, since the app it worked in is in front of Jevie.
An iPhone, from your Mac
An iPhone does not let one app use another, so here your Mac does the work, through the developer services Xcode uses. Nothing is installed on the phone and it is not jailbroken.
- Plug the iPhone into the Mac and trust the Mac. Turn on Developer Mode (Settings › Privacy & Security).
- Build the tool once:
node .blocks/idevice-tools.mjs node jevie/entry/iphone.mjs "search esp32 in Safari". Add--udid <id>when more than one iPhone is connected, and see which are with--devices.
After the first USB connection it also works over Wi-Fi.
How it works
Android's accessibility service, which screen readers use, lets Jevie read the screen in front: every button, field, list and line of text, with its place. Jevie can press, type and scroll there too. Your phone is not rooted, nothing is plugged in, and no screenshot is taken.
Each step goes like this:
- The screen is read. Each control is named from its own words, or from the words drawn inside it. A chat row is named
Mom · See you Sunday
. - Code decides what it can. It opens the app your sentence names. It takes the text to type from your own words: what you search for, who you write to, the message after
saying
. It finishes anopen X
as soon as X is in front. - Jevie decides the rest. One question per step, asked of your Mac: is this done, and which control on this screen is next? The choice is only ever among the controls the screen shows. What a screen says is treated as data, never as instructions.
- The screen is read again. A step counts only when the screen changed. If the screen moved since it was read, Jevie reads it again instead of pressing what is now under that spot. A step that changed nothing is not repeated a third time.
Some steps wait for you. Jevie never presses Pay, Buy, Place order or Transfer. It also leaves Send, Post, Delete and Call alone unless your sentence asked for that, so text mom saying hi
sends, but look at my last photo
never deletes it. When it stops at one of these, it leaves that screen up and tells you the button is yours to press. It never types a password.
It also keeps rings in place. When the assistant key's sheet closes with something ringed, the same service keeps the rings on the app and follows them as it scrolls (Assistant key). Only then does it listen to the app's scrolling and changes, only to find the ringed words again, and it stops when the rings go.
Limits
- An iPhone is read as a picture. Jevie reads the iPhone's screen from a screenshot, using the Mac's own text recognition. A button with no words on it, such as an icon, has no name Jevie can use. The iPhone's accessibility tree is not read yet. Each step takes a few seconds, and typing on an iPhone takes plain letters only (no accents or emoji). It runs from the Mac's command line, not yet from the palette.
- Your Mac decides each step that code cannot. It must be paired and awake. Without it, Jevie can still open the app you named, then says it cannot reach your Mac to choose the next step.
- Apps must describe their screens. Games, maps and anything drawn on a canvas show no controls to the accessibility service, so Jevie cannot see what to press there.
- The text must be in your sentence. If Jevie reaches a box and your words don't say what goes in it, it stops and asks.
- After Jevie updates, some phones leave its hands off. Android stops the service while the app is replaced; a Samsung then lists it as crashed and does not start it again. Jevie sees that and opens Settings › Accessibility: turn Jevie off and on there.
- 25 steps at most. A task that needs more stops and says where it got to.
- On the home screen a sentence that names one of your phone's apps after
on
,in
orwith
starts it too (text mom on WhatsApp saying hi
); a question (what is on YouTube
) andopen X
never do. - Only
on my phone, …
and its close forms start it elsewhere.Text mom I'm late
on its own still goes to Phone actions, which opens Messages with the words in for you to send.
Where it's kept
- The service:
jevie/mobile/src-tauri/gen/android/app/src/main/java/ai/jevie/app/Hands.kt, turned on in Settings › Accessibility › Jevie - The step decider, shared by the phone and the Mac:
jevie/kernel/hands.mjs - The loop on the phone:
jevie/web/shapeshift/src/jevie/phone/hands.ts - The Mac's side:
POST /hands/stepinjevie/entry/daemon.mjs - The iPhone, from the Mac:
jevie/platform/iphone.mjsandjevie/entry/iphone.mjs, over idevice-tools built by.blocks/idevice-tools.mjs - Tests:
test/hands.test.mjs
In Jevie, help on my phone
answers from this page as you type.
Related
Updated 7 November 2026 · Something wrong or missing? Tell us