Taking the mouse away from an AI agent
I want to be able to take the mouse away from an agent while it’s working. Move something, fix something, then let it continue. Pretty normal thing to want from a computer you own.
But the agent may already have a click on the way. Or it’s halfway through typing. Or it’s holding Shift when you start using the keyboard.
That’s been an interesting part of building ibara, which lets AI agents use your other Omarchy computers. I can leave an agent working on a spare, watch its screen and take control when I need to. The handover has to work in the middle of whatever it’s doing.
The public version uses Cua Driver for desktop input. I’ve also been working on an experimental native driver inside Hyprland, the compositor that manages the desktop. The examples here come from that development work. There’s still more to test before treating it as a general replacement.
The first thing to work out is what happens to input that’s already on its way.
Setting a paused flag stops the next request if it hasn’t passed that check yet. Input that already passed it needs another check closer to where it reaches the desktop. Otherwise the interface can say you have control while an old agent action is still arriving.
In the experimental driver, a broker inside the compositor checks input before it enters the queue, then checks the target again for each event. A generation number identifies the current period of ownership. When ownership changes, old requests lose permission to act.
The target has its own checks too. The agent saw a particular window, at a particular position, with a particular focus state. Having permission to use the computer doesn’t make those observations current forever.
One source finding was especially interesting. A shared scene cache could update between resolving a point from an image and admitting the action. If the old point picked up the new revision, it could look fresh even though the decision still came from the old image.
So the original identity and revision information has to travel with the target. Updating the cache can make an old action invalid. It shouldn’t quietly make it valid again.
That’s a fairly small implementation detail with a pretty direct consequence. You could be checking freshness and still clicking from an old picture.
Held keys make the handover more complicated. If the agent is holding Shift, stopping future input leaves Shift down. Cleanup has to release what the agent held. But you might also be holding Shift on your own keyboard, and cleanup needs to preserve yours.
Same key, different input sources. We tested a specific case where physical Shift interrupted a remote viewer holding Shift. The viewer’s held key was settled, and its old input stopped being admitted after the takeover boundary. That gives us evidence for that case, with the broader combinations still to test.
Then there’s what the agent gets told afterward.
In one typing test, a 200-character operation was interrupted after a short prefix reached the app. An attempt to continue from the old state was refused with person_was_here. After a fresh observation, a different character was allowed.
The saved file contained iaaaz. The interrupted prefix was iaaa, and the new character was z. We didn’t replay the original typing operation.
Its result stayed unknown. Some of it had clearly happened, but the operation hadn’t finished normally. Calling it “not started” would have been wrong, and would have made retrying it look reasonable.
For typing, that could duplicate text. For a submit button, it could submit twice. A missing response doesn’t tell you whether the action reached the app. The agent needs to look at what happened before deciding what to do next, and sometimes the app won’t give it enough information to be certain.
The tests also turned up a problem with our own evidence.
That typing run had saved app contents and 2,599 captured frames. But the broker export only included the first 64 of 296 records. A related motion test had the same problem.
We could inspect what reached the app and what appeared in the captured frames. We couldn’t use a partial log to establish the order of every event the broker admitted.
The exporter needed to fetch every page and reject incomplete results. Later runs collected complete logs. That still doesn’t give the earlier run the records it was missing.
I think this is an easy place to overstate a result. The app looks right, the recording looks right, and the tool says the operation stopped. But if the claim is that no old input was admitted after takeover, you need evidence that actually covers that boundary.
I want the app’s saved contents alongside the broker events and the screen capture because they answer different questions. The app tells us what reached it. The broker tells us what it admitted. The capture shows what was visible. If they disagree, that’s something to investigate.
A successful typing interruption test gives me a reason to trust that particular path more. There are still disconnects, queue failures, other apps and other input devices to work through. I want each result to say what we actually tested.
What I like about this problem is how ordinary the original request is. I want to use my computer while an agent has work in progress. Following that through means dealing with queued events, held keys and decisions made from a screen that has already changed.
When I hand it back, the agent needs to look again. I may have changed the thing it was about to do.