How to turn a screen recording into an AI agent
Record yourself doing the task once, start to finish, with real data. A tool that learns by demonstration, such as Ditto AI, reads the frames, your clicks and any narration, writes down the process and its inputs, and finds a tool for each step. The result is an agent you can read and edit before it runs, not a macro that replays your clicks.
A recording is not a replay
There are two very different things you can build from a screen recording.
| A macro | An agent | |
|---|---|---|
| What it keeps | The exact clicks and keystrokes | The process: goal, steps, inputs |
| Next time | Repeats the same clicks in the same places | Does the same job with new inputs |
| When the screen changes | Clicks the wrong place | Looks, then decides the next step |
| How it acts | Moves the mouse | Calls tools, such as MCP servers, and works an app on screen only when no tool exists |
| What you can review | A list of coordinates | Steps in plain language, with the evidence for each |
A macro is useful when nothing ever changes. An agent is what you want when the task is the same but the details aren’t: a different invoice, another recipient, next week’s numbers.
An old idea that finally works
Teaching a computer by showing it is decades old. Researchers called it programming by demonstration, and a 1993 book of that research, Watch What I Do, edited by Allen Cypher, set out the goal: let people automate what they do without writing programs.
What held it back was understanding. Early systems could see that you clicked at a position, but not that you were saving an invoice. Models that read screenshots, together with standard ways for models to call tools, close that gap. A model can now look at a sequence of frames and say “this is an invoice from Acme; the amount is the input; the destination folder is fixed”.
What the recording needs to capture
To turn a recording into an agent, the recorder has to keep more than video:
- Frames of the screen, so the model can see what you saw.
- Clicks and focus, from macOS Accessibility, so each step is tied to a real control, such as a button, a field or a link, not just a position.
- The app and the address for each step, so a tool can be chosen for it.
- Your narration, if any, transcribed on the device, so you can point out what changes and why.
Ditto AI records all four on a Mac, and transcribes speech on the Mac itself. It leaves its own windows out of the recording.
How to record a task so the agent comes out right
- Do the real task, with real data. Test data hides the variations the agent needs to see.
- Do the whole thing, once. Start where the task starts, such as opening the email, and stop when it’s done, such as the row being saved.
- Work at your normal pace. There’s no time limit, and pauses don’t hurt.
- Say what changes. “This is the vendor name. The amount comes from the PDF. The folder is always the same.”
- Stay on task. Checking another message halfway through adds steps you don’t want.
- Keep secrets off screen when you can. Recordings stay on your Mac, but you choose what the reading model sees.
What happens after you stop
In Ditto AI, the recording is split into steps using your clicks and app switches. A model then reads the frames, clicks and transcript and writes down the process, its steps and its inputs. Every step must point at the frames it came from, or it’s dropped.
Then each step gets a tool. Public web pages use a headless browser. Anything that needed your account gets an MCP server, found in the official registry or on npm and checked with one read-only call. Finally, permissions are set from a fixed table, and you review everything before the first run. How Ditto AI works walks through each stage.
Sharing an agent
An agent built from a recording can be shared as a file. When someone opens it in Ditto AI on their Mac, it goes through the same review, and its permissions are worked out again on that Mac rather than trusted from the file.
When a recording won’t work
- Nothing happens on screen. A recording of one idle app has no process in it, and Ditto says so rather than guessing.
- The task depends on something you didn’t show, such as a rule in your head. Say it out loud while recording, or add it when you review the steps.
- The content can’t be captured. DRM video shows as black, and some apps hide their windows from screen recording.
- It’s a one-off. If you’ll never do it again, just do it.
Ready to try it? Install and set up Ditto AI, or run the demo in your browser first.