Présentation de SimuLang : Playwright pour l'ensemble du bureau

Why, then, prototype with glasses?

We figured glasses are a common, natural, minimally invasive accessory: a necklace would be too heavy, a headband uncomfortable, and a watch or ring sits too far from your mouth to listen from or talk to comfortably. Many smart glasses also carry a camera, so Sai can see what you see: hold up an object, ask "what is this?", and the camera fires. And the Meta Ray-Ban Display in particular leaves room for a heads-up display down the line, so you could read minor progress updates from Sai or even see what Sai is doing on-screen — not available today, but there is potential.

The glasses are the frame I've chosen to show all of this through. The rest of this post is thus framed in terms of the glasses prototype.

One call, start to finish

You tap Start call, and Sai greets you first — no dead air, no "hello? can you hear me?"

You ask for something: "Check my unread emails and Slack." You get one short acknowledgment, and then silence while Sai works, and then the answer. No progress narration, unless you ask for it. Just the result when there is one.

Halfway through Sai's spoken response you think of something and cut in. Sai stops — mid-word, not at the end of the sentence — and responds to what you just said. The ability to interrupt in the middle of a conversation makes the whole experience feel alive rather than like leaving a voicemail.

While Sai is still working on the first thing, you add a second task: "Also, book a table for tonight." Sai (or more precisely, the voice concierge) confirms details and tells you the booking will happen after the current task — and then it does. If you lose track and ask "what's going on?", Sai can update you on the progress, with the tasks separated cleanly: one running, one waiting.

Getting home after a day of enjoyment, no one wants to handle boring admin chores. You hold up a letter that came in the mail: "Copy the appointment number from this letter and use it to fill out the online form." The camera fires, and the photo rides along with the task you asked for.

When you are done, you just say so — "thanks, you can hang up now." Sai says goodbye while wrapping up any running tasks in the background, which you can check from the desktop app or browser after.

The menu — what you can actually do

• Hand off real work to Sai: triage email and Slack, draft and send replies, book things, manage your calendar, and ask about whatever is on your screen or in front of you.
• Capture what you are seeing: hold something up and ask; the camera captures it and attaches it to your request.
• The controls are four buttons on the phone and one gesture on the glasses:

Two links, one call

The audio and the agent travel on separate paths. Your voice goes straight from the phone to the realtime voice model — in our case Gemini Live — and our servers won't see a single frame of audio. The agent runs on a separate connection to us. The two paths meet only as structured effects and agent events. This keeps the conversation feeling instant (no audio detour through our infrastructure), and also keeps it safe: the model talks to the user naturally, while only the server can act.

The credentials follow the same split. The voice half runs on your model key, held by the app, and you pay Google (or your preferred realtime audio model provider) directly for what you use; we are not in that loop at all, and there is no server-minted token to deal with. The agent half authenticates separately, against your signed-in Sai identity, and that is the only path that can reach your machine and your account.

An open conversation, a closed set of verbs

The user and the model can say anything. But the model's actions are constrained to a small, fixed set of validated actions: forward a task, ask a question, approve, deny, choose an option, queue, and so on. Moreover, irreversible actions are re-checked on the server no matter what the client claims.

I originally started by trying to classify the user's intent into an enum (task | steer | context), but after some testing, I realized every new situation felt like a new category and the classifier only got worse as it grew. The fix was to flip this around, letting the conversation be open and bounding the set of functions the model can trigger instead. Now, a model that is wrong or jailbroken still won't be able to reach past the allowed actions. This was a security architecture problem in conversation-design's clothing.

That closed set of actions is also where trust lives. The irreversible verbs never fire without an explicit confirmation, read back to you first — who a message is going to, what it says — and anything involving a login or a connected account can't be done by voice at all: you shouldn't be spelling your password aloud on a train, so Sai sends you back to the app. If something is going to act on your real accounts, the trust has to be structural, not vibes.

Interruptibility (barge-in) beats audio quality

Early on, Sai sounded wonderful through the glasses: rich and clear. I could hold a normal conversation if I listened to Sai speak in full before responding. But then I tried to interrupt Sai and simply couldn't, because the microphone was off while Sai spoke. It turns out the glasses, like most other Bluetooth audio devices, can either play high-fidelity audio (the A2DP profile) or keep the microphone open at lower quality (the hands-free HFP/SCO profile), but not both at once.

We had to choose our fighter: beautiful-sounding Sai or interruptible Sai. The choice is clear when you realize that once you're actually in a conversation, you stop noticing audio quality pretty quickly. What you never stop noticing is having your words completely ignored. We even briefly considered some push-to-talk sort of idea, but really, for hands-free voice interaction, being able to barge in naturally beats nice audio quality by far.

"Sai won't stop talking" was actually three separate bugs

Even after fixing my decision above, my most common complaint in testing was four words long: "Sai doesn't stop talking." It felt like one bug. In reality, it was three different bugs from three different places.

The first was the model's own voice detection being too insensitive: it wouldn't register that you'd started speaking until you were well into a sentence, so the interruption landed late and Sai had basically finished by then.

The second was in our own code: we were reading from the network and writing to the speaker on the same thread, but writing to a speaker blocks once its buffer is full. The fix was to split those two jobs, so the reader is always free to notice the user.

The third was the network: we'd throw away the audio we no longer needed, but some of it was already in flight and landed a moment later, so Sai kept talking anyway. The fix was to ignore anything arriving right after you cut in, which was safe because the next real reply doesn't start for over a second.

Indeed, a complaint phrased in user language doesn't necessarily map to only one cause. The method I used to help track this was to add a test to catch each specific bug I fixed, and not just rely on fixing things by ear.

The silence is deliberate

On a screen, a spinner tells you something is happening and you wait patiently. In your ear, three seconds of nothing feels like the thing is broken, and you start wondering if something failed silently. The obvious fix is to have Sai fill the gap — "I'm still working on that…". I built it, but it made everything worse: the filler collided with Sai's real answers, or it interrupted my train of thought only to tell me it wasn't done with the assigned task. In the end, we settled on one acknowledgment, silence (unless asked about progress), then the result. The quiet is the feature.

These days, a coding agent can help you realize almost any idea in a couple of minutes, but no agent can tell you whether it's really the right thing to build.

Testing a conversation

You cannot unit-test a conversation, but you can test what it does. So there are over 60 golden scenarios asserting the effects and resulting state — never the phrasing — plus an offline critic that grades real transcripts against rules like "don't fabricate an ETA" and "abort is not completion." The rule I held myself to: a bug I'd fixed by ear didn't count as fixed until there was also a test to catch it coming back.