How we trained our first RL AI: from a game idea to a playground.

Oct 25, 2026 · @Ach

Why, and where we started

We wanted to check, on a live example, that reinforcement learning (RL) can be launched and carried through to the end in the browser, as simply as ordinary training. Here, though, the network does not receive a dataset: it gathers experience by playing against itself.

RL in a nutshell: the model has no “right answers”. It plays, gets a reward for good outcomes and a penalty for bad ones, and adjusts its weights so that winning decisions are repeated more often. For a first experiment you need a game whose result is clear to the eye and fast to compute. Images and text are too heavy for this: it is unclear how to score “good”. A game on a grid gives a clear score, fast matches (thousands in seconds) and easy visual inspection.

That is how our version of the game Territory appeared: a board of cells with several agents, where each agent takes one cell per turn and tries to hold as many as possible. The agent is always the same: a tiny neural network (about 250–480 weights), while all the complexity is hidden in the rules and in the training.

Constraints

Training runs in the participant's browser, so everything has to be cheap to compute. A training round is 1,024 matches, a match is thousands of ticks, and on every tick the network is run for every agent over every candidate move (during training these rows are also stored for the backward pass). That is why the network does not see the whole board and does not choose “any cell”: it would have to be run over the entire border of the body (for a body of 100 cells that border is long), and by our estimates training would become noticeably slower.

We decided it like this: on every tick the engine picks up to 10 random cells on the border of the body (and up to 12 strike options, if there are charges), computes a feature vector for each and hands them to the network, so the choice is among at most 22 options. The price of this decision is noise: the best cell in terms of direction sometimes does not make it into the list. So that the network still holds its course, we always pass it a vector to the nearest available opponent (the direction to the enemy and the cosine of the move toward it). Over dozens of ticks the lists are refreshed, and the noise of a single move averages out.

The game: from a simple idea to a complex one

Step 0. The basic idea

A board of cells and several agents. On every tick an agent takes one cell next to its territory. If two agents want the same cell, it stays empty and both lose their move (a failed move costs one more cell). So that nobody can grow forever, every two ticks the territory loses its oldest cell. The winner is the last one remaining.

Step 1. Too easy

What we saw. In about 18 rounds the network beat a random bot in 100% of matches. Against a simple “greedy” bot (it takes cells next to the enemy) it stayed around 40% for a long time, judging by the training logs.

The problem. The game came down to “grow in an even circle”. There was nothing to watch and nothing to fear.

What we tried. Territory splitting: if a “bridge” cell disappears, the smaller part of the territory disappears with it. Plus “scars”: a lost cell cannot be taken for three ticks, and losing territory carries a penalty.

Step 2. Networks fear “necks”, but do not cut other people's

What we saw. The new rule worked halfway: the networks learned to build compact shapes and to avoid thin “necks”, but they did not cut other people's necks themselves.

The problem. Fear without hunting: the networks learn to defend, but not to attack.

What we tried. A bonus for cutting enemy territory.

Step 3. We want a real fight

The problem. An enemy cell cannot be taken, so combat is indirect.

What we tried. Attack: you can take an enemy cell if next to it there are more of your cells than cells of its owner.

What we saw. The networks started to attack.

Step 4. Investigation: why do they lose in the middle

Hypothesis. In the middle of the map the odds drop: there an agent is squeezed between two neighbours.

What we saw. We ran matches on a 200×100 board with 30 agents, all using the same network. Distance to the edge hardly matters (606 cells at the edge against 530–546 in the center). The number of neighbours within 20 cells matters a lot: with no neighbours 703 cells, with four 421, which is 40% fewer.

The problem. An agent squeezed between two loses cells on two fronts at once, and recovers at one move per tick.

What we tried. Spike strikes: energy accumulates into charges, and k charges make a spike up to 4k cells deep and up to k−1 wide on each side. A spike can cut through enemy territory. The idea: a small survivor can accumulate strength and turn the game around. But…

Step 5. Networks do not like to save up

What we saw. When there are many enemies nearby, the networks spend a charge as soon as it appears: in a crowd of 30 identical networks most of them have almost no charges at any moment, and the average strike is about 4 cells. When enemies are few and far away, charges pile up by the hundreds (the winner of one match had more than 390 by the end). A strike is offered only where a neighbour has at least three cells in a row, so there is no attacking into emptiness, and charges accumulate on their own until a target appears.

The problem. The networks save not by calculation but because there is nothing to hit: they do not hold back for one large strike instead of several small ones. The payoff from saving comes after hundreds of ticks, while REINFORCE sees the near reward.

What we tried. A charge schedule on a logarithm, a limit on body size and “trophies” for captured cells, and in the end energy from captured territory.

The result

In the end the agent was left with one desire: territory (in the latest version of the game this is “land”: the cells its body has passed through; it does not fade, and is lost only when another body takes the cell). New land is a bonus, losing it is a penalty, capturing someone else's is a big bonus, and heading toward enemies is a very small bonus. Everything else should follow from this.

The playground

If a model learns in a game, you need to be able to watch it. So we built a playground page where the game engine and the networks run right in the browser (WebAssembly, no server): a 200×100 board with 30 agents. The page is published at territory-playground.ai.weei.io. It is very absorbing to watch a newly trained network compete with the others. You know it is an algorithm, yet the mystery of its judgment when it picks a move is intriguing. Many hours were spent at a laptop screen watching training progress and, at last, the competition. And then a thought: what if we add bots from other neural networks? An interesting way to compare the strength of different AIs?

And now you can add a bot to the playground, as an ordinary JavaScript function. It receives a description of the position (its own state, the opponents, a list of candidates with the fields kind, landOwner, attackWins, cutSize, spend and so on) and returns the number of a move. For neural networks we wrote one and the same prompt (it is on the page itself, with a “Copy prompt” button): the rules of the game, the exact shape of view and a line saying “which strategy to implement”. Then the file is either pasted as text or uploaded as a file.

If a bot contains a model, it can prepare itself: an optional function async init(ctx) receives the attached files model.wasm and a checkpoint from the Weei widget, brings the model up through the standard ABI and loads the weights with a loadModel call. All the game logic lives in the playground, not in the model: a model trained on Weei does not have to know anything about territory. Our own networks can also be loaded straight from a checkpoint without any code: the format “four bytes of header length, JSON, weights” is parsed on the page.

Who won

In short: our networks do not survive in a crowd of bots, and the strongest of all is Astra6. In our runs on the playground (30 players on 200×100, the “land” rule, 20 matches with all agents; 14 matches ran to the end):

The sample is small, so the table should be read as an order, not as exact percentages, and the places after the first change from run to run. Only one thing is unshakable: Astra6 wins more than half of the finished matches in all of our tournaments. From watching the playground, Opus5.5 and Sol6.1 share second place by land, and everything else is far behind. Our trained networks and the Hunter we wrote cannot hold on there: Astra6 eats them.

And this should not be hidden: that is what the playground is for. It shows where a network learns the wrong thing: our networks beat Astra in a duel on a small board (81–86% wins) but lose in a crowd on a large one, because they were trained in a different game. The next step is training in a crowd (six players on 40×40, with Astra and Hunter in the pool), so that the networks learn to fight it where it is strong.

What we learned

1. Choose a game with a clear and fast result. Thousands of matches in seconds, a score of “who captured more” and easy visual inspection sped up everything else: debugging, arguments and explanations.
2. If you want one thing, reward one thing, and do not stuff in rules. Every added penalty and bonus is a new decision that the network will interpret in its own way.
3. Measure behavior, not only win rate. Our best debugging conclusions came from simple counters: the share of moves onto your own land, the cosine of the direction to the enemy, the size of strikes. The win percentage alone does not show this.
4. Self-play easily retrains a network for itself. Keep other people's opponents (bots) in the pool and in the exams, otherwise the network becomes strong only against opponents like itself.
5. Train in the same game you test in. A duel on a small board and a crowd on a large one are different games: our networks beat Astra in one and lose in the other.
6. REINFORCE does not see distant payoffs. The networks spent a charge as soon as a target appeared, and saved only when there was nobody to hit: the payoff from saving comes after hundreds of ticks. A plan needs an evaluation of the position or a larger near reward for large strikes.
7. Check the rules with tests and long matches. There were bugs that appeared only on the large board, seemingly just because of its size.
8. Separate training from play. A model trained on Weei should not know the rules of the game: the playground accepts any checkpoint and any JS adapter, and the game logic lives in the adapter.

And finally

If you are doing your first RL, take a small game, one simple algorithm as an examiner, and metrics that show behavior. That is enough to see that the network is learning and to understand where it is deceiving you. And you can train without extra fuss at ai.weei.io.