SimuLangのご紹介：デスクトップ全体のためのPlaywright

Simulangは、ブラウザ、ネイティブアプリ、OSレベルのワークフローを自動化するためのスクリプト言語で、AIエージェントによって記述されるように設計されています。Simulangをオープンソース化しました。今すぐ単一のコマンドでインストールできます。

Throughout the summer internship, besides working on my main project (Android VMs for agent control, it's still in alpha at the time of writing, do look forward to it when it lands!), I've always tried to get computer-use agents to play games, even the simple ones like Google Minesweeper. My realization is that they suck, which led me to explore solutions. This post details my experiences and the experimental harnesses I've created along the way.

Getting agents to play games has been a timeless pursuit. It's visual, it's fun, and it stresses problem-solving and adaptability. Games as a benchmark has also been embraced by Google, who introduced Game Arena a year ago, pitting frontier models against each other in games like Chess, Poker, and the newly added Ultimate Tic-Tac-To. Asking LLMs to play these games may seem counterintuitive, after all we already created superhuman Chess engines nearly a decade ago, underscoring the power of reinforcement learning (RL) in domain-specific tasks. But I want my AI to play *every* game, not just one. This calls for generality, and what better model is there for generality than a Large Language Model (LLM)? Besides being well-rounded, they've also gotten pretty smart (after all Fable did just offer a counterexample to the Jacobian Conjecture).

From benchmarks to your laptop

Game Arena isn't the only evaluation platform; a more realistic one is VideoGameBench featuring titles like Doom and Super Mario Land that challenge agents to beat the *entire* game. As you may have feared, that's too hard, even the top performers scored 0.48%. But being a noob is not a sin; the more pressing issue is that everyone has their own evaluation harness so it's a pain to set up a custom environment each time. Why can't I download some arbitrary game from Steam/App Store, boot it up, and let *computer-use* agents manipulate my keystrokes and mouse movements? That sounds promising, but when tasked with solving something as simple as Google Minesweeper through pure computer-use (i.e., you're only allowed to take screenshots, no cheating with APIs!), you'll notice they flounder spectacularly.

Technically, accessibility (AX) trees fall under computer-use, and they offer you exact measurements of elements on your desktop. They're by far the superior method for careful navigation and control of your computer. The only problem is that many games don't expose their AX tree, so we're stuck with screenshots for the time being.

This is where things start to fall apart, because the returned position is more often than not incorrect. Sure, you may get an answer that's "somewhat" close, but "somewhat" is not enough. Precision matters, and you're going to pay for that "somewhat" by having clicked on a mine that blows up your run (not that you'll get very far to begin with).

An inversion of responsibility

VLMs are genuinely strong. On challenging GUI-grounding benchmarks like ScreenSpot-Pro, the top (ensemble) models just broke the 80% barrier. Given a screenshot of VS Code, they locate with pinpoint precision the coordinates of various tiny icons and buttons. For vision grounding, VLMs are undoubtedly the way to go. The problem is that we trust them too much. Under the current paradigm, we ask for positions using `ground(concept)` and act upon it right away. If the VLM makes a mistake, we click the wrong thing, with potentially catastrophic consequences. As Claude would quip, this single `ground` is load-bearing.

If the VLM shouldn't be our sole pillar of truth, then who should? I claim that the *LLM* should. After all, you're the cognitive core of the agent, shouldn't you take responsibility for what you're seeing and doing? If your view is too crowded you crop, if you're not sure you measure, if you have a guess you preview, only when you're confident do you act. I believe this philosophy can be distilled into a harness, and in only a couple thousand lines I prototyped a harness inspired by scene graphs which I called *Scenes*.

If there are no AX trees, we'll build our own

Scenes Workflow

Breaking down a desktop into its components, from Chrome to the board, eventually carving out the individual cells.

The scene graph is composed of mere (nested) rectangles (e.g., an app window, a card on a website) and points (things you might want to click). That's it. Of course, mapping out the right coordinates and geometry is the tough bit, and Scenes helps you out by (a) providing suggestions on dominant regions, key axis lines, and likely-to-be-correct points, as well as (b) letting you draw tints, crosshairs, and compare drawings across multiple scenes to verify correctness. To see if the harness actually helped, we intentionally decided against incorporating new segmentation/OCR models, instead reusing whatever VLMs we already had, along with traditional CV algorithms like MSER.

Scenes is going to be slow initially. That's the nature of iteratively refining your spatial model of the world instead of impulsively acting on the first guess of a VLM. But it's not as bad as it sounds because:

Sai playing tetr.io with Scenes

Do you know what else is a grid? Tetris! Here's Sai playing tetr.io with Scenes (2x speed) through pure computer-use.

Make no mistakes

Trick Images

Six categories of tricky questions in the VLMs are Biased dataset. Yes, the animal drawn with five legs was intentional...

Besides grounding, a curious side effect is that Scenes helps to debias models. During training, many LLMs memorize information about the world and develop strong assumptions about how things *should be*, not what they *actually are*. This bias has been surfaced by works such as *Vision Language Models are Biased* where models jump to conclusions when looking at slightly altered/photoshopped images (e.g., an animal with five legs, a chessboard with seven columns, or a flag with a missing star). Unsurprisingly, telling the model to "double check" or "don't assume from prior knowledge" led to marginal improvements. As it turns out, prompt engineering can only get you so far. Even for frontier models, accuracies remained around 20%.

On a small random sample of the VLMs are Biased dataset, our Sai agent with Scenes scored around 67% (it got around two-thirds correct). It's not that "double check" doesn't work, but rather if you want an LLM to double check, you ought to give it the tools to do so, and Scenes offers precisely these measurement tools!

So why can't we achieve almost 100% if we could theoretically measure everything? Well... sometimes the model is just so confident it chooses to skip the harness completely. The more well-known a concept is (i.e., country flags), the harder it is to overcome the inherent bias.

Not all games are as forgiving

Games run at different latencies. In turn-based games like Chess, you get all the time in the world to contemplate your next move (ignoring time controls), but for something like Doom, you need to be absorbing information and emitting your actions at 30 fps to not appear like a frozen potato. We've conquered structure with Scenes, let's figure out how to play first-person shooters.

If you work with the native LLM observation-action loop, you simply can't. A single response takes seconds to several minutes, far too slow to play fast-paced games. LLMs are great strategists, but they need something to help them execute their will, and that something is often known as a *System One* model.

The authors of VideoGameBench also recognized inference latency as a significant impediment, leading them to create *VideoGameBench Lite*, where the emulator is tweaked to halt whenever the agent is thinking, essentially reducing everything to a turn-based environment. You can replicate this trick for offline games too by taking a screenshot and hitting the pause button in quick succession, but where's the fun in that? Let's figure out how to actually solve the problem.

Decoupling the slow and fast loops

Multi-agent orchestration is the new paradigm: let Fable do the planning, dispatch boring work to Opus/Sonnet. Instead of talking to a small LLM, can we go even tinier? Perhaps a small Vision Language Action (VLA) model with only a hundred million to several billion parameters? What about the aforementioned VLMs? Or if we go back in time and use traditional architectures like convolutional neural networks (not everything has to be a transformer...)?

Ultimately, there just isn't a one-size-fits-all System One model; different circumstances demand different capabilities, and a tiny model can only hold so much "intelligence" and "context".

Our problem isn't a unique one. Decades ago, people invented the *blackboard architecture* where a slow planner orchestrated a bunch of fast-acting arms, communicating through a "shared state". Inspired by this architecture, I built a tiny prototype harness called *Chalkboard* that argues for process management and shared memory as the basis for communication.

Yes, a process is exactly what you're thinking, just a running program (in this case Python scripts, they're easy for LLMs to write after all). This intentionally loose definition avoids locking us into any one architecture, instead allowing us to enjoy the entire spectrum of algorithms and latencies.

So, who's responsible for what?

Chalkboard Overview

Arms never communicate with each other directly; state is conveyed through shared memory.

By design, Chalkboard *encourages* you to perform this role-based decomposition such that each *arm* (i.e., process) is responsible for a specialized subtask. For object classification, perhaps a tiny CNN is ideal, and for navigation, something like ViPlanner (think of it as a little compass telling you where to go next) would be more appropriate.

In general, I like to think there are three kinds of arms: *sensors* that take in information, *intermediaries* that process and analyze the data, and *effectors* that perform action on the environment.

Self Improvement

Figuring out what went wrong during a game is an essential part of the learning process. One solution is to watch the game yourself and offer human feedback to the LLM. This is tedious. An alternative is to use the *archive* system that Chalkboard offers. Compressed trajectories are stored across rounds and an LLM may easily retrieve key moments and statistics to answer questions like "Was I stuck in the same spot?" or "How many enemies were there when I died?".

More often than not, the initial pass is already pretty good! The role-based decomposition necessarily forces the LLM to pull off better abstractions that reduce mistakes. Introspection and self-improvement are what bootstrap the LLM to go from an average player to a great one.

VizDoom Gameplay

Free-for-all gameplay in VizDoom (async mode, i.e., the game runs at its natural pace without waiting for you to think at each frame); it took around ten self-improvement loops (roughly an hour of gameplay) to get to this state. Learning how to chase and dodge was an early priority; later iterations focused on objectives like healing and maintaining your ammo.

Can I run it locally?

Yes, the above gameplay was a result of tiny neural networks (such as ViPlanner) running on my MacBook. There are certainly other interesting models like Pixels to Play that I've reproduced and benchmarked at around 160 fps (raw inference) on a 5090. Their appeal is that they offer a *midpoint*, in that they can perform some amount of reasoning (with a couple hundred tokens in the context window), along with seeing and acting, but that reasoning is night and day compared to a real LLM, and it neither sees nor acts as well as the specialists. Despite that, I'm still optimistic about such models as finding the right specialists for a particular task can be exhausting, and small labs could potentially serve these models (on the edge) cheaply, performing a diverse amount of work at satisfactory standards.

Beyond games (and back to work)

Even if playing games was the seed of inspiration for these experimental harnesses, their true value lies beyond that. For one, on the Android platform, a "virtual" AX tree is used that's sparse and lacks information; perhaps Scenes could help alleviate some of the friction when automating Android apps.

Could we train a tiny model that helps to build the scene graph automatically?

LLMs have cemented themselves as some of the most intelligent machines today. But thoughts don't manifest unless they have a medium to act upon, and that medium is fragmented across all the different applications/platforms/services. Yes, growing adoption of MCP is one relief, but amidst the disorganization, harnesses are the way to bridge this gap. It's something everyone can engage in, from individuals to large organizations.

And it's not just about letting LLMs see and act on the environment. Considering how smart they've gotten, is it really fair to encumber them with minutiae like filling out a form or copying cells between spreadsheets? Some tasks are so mechanical they barely warrant an LLM, and there are entire untapped ecosystems of specialized architectures/algorithms out there that could act as the arms and legs of the LLM. Once again, harnesses fill that void, and such harnesses scale not only with the progress of frontier models, but with advancements in fields like computer vision.

If we can play games at 30 fps, can we do work just as quickly?

MiniWoB Footage

Concurrency is a first-class concept of Chalkboard, and you can have multiple arms performing tiny web tasks (MiniWoB) in parallel. The planner driving the interactions is Qwen3-VL (2B). How do we avoid data races? After all, only one Chrome window can get the focus at a time. Hint: because of single-writer enforcement, there are ways to guarantee mutual exclusion without introducing a mutex primitive.

Got questions for Jia Qing? Email limjiaqing2@gmail.com