Most traffic lights run on a timer. Green for this road, green for that one, repeat, whether anyone is waiting or not. Where a busy road crosses a quiet one, that timer hands out green time to empty lanes while the queue on the busy side keeps growing. I wanted to see if a small reinforcement-learning agent could do better just by watching the queues.
The setup
Everything runs in SUMO, an open-source traffic simulator, controlled from Python through its TraCI interface. The road network is one intersection with four arms, one lane each way, through traffic only. No turns, on purpose: that keeps the problem about when to switch the lights and nothing else.
North–south traffic is twice as busy as east–west (400 vs 200 vehicles an hour). That imbalance is exactly where a fixed 50/50 timer falls apart, so it's the interesting case to test.
How the agent sees the world
Every 5 seconds the agent looks at the intersection and makes one decision: keep the current green, or switch.
- State: how many cars are stopped on each of the four approaches, bucketed into 0, 1–2, 3–5 and 6+, plus which direction currently has green. That's at most 512 possible states.
- Action: keep or switch. A switch only counts after at least 10 seconds of green, and it always goes through a 3-second yellow, so switching has a real cost.
- Reward: minus the total queue built up since the last decision. Less waiting, more reward.
The learning itself is plain tabular Q-learning: a lookup table of "how good is this action in this state", updated after every decision.
I used a learning rate of 0.1, a discount of 0.9, and ε-greedy exploration that starts fully random and decays to 5% over the first 200 episodes.
Why not a neural network?
The state space is small enough to fit in a table, and a table is something you can actually open and read. The whole trained agent is a file of about 2 KB, and every decision it makes can be traced back to one row. A deep Q-network would have been more fashionable and a lot harder to explain.
Training
Each episode is one simulated hour of traffic. Training ran for 300 episodes, so 300 hours of traffic, which took about two and a half minutes on my laptop.
The left chart is noisy for the first 200 episodes because the agent is still taking random actions on purpose; the rolling mean shows the real trend underneath. The right one is my favourite: out of 512 possible states, traffic only ever produced 39, which is why a simple table works so well here.
Does it actually beat a normal traffic light?
I compared it with two fixed-time signals: a plain 30/30 second split, and a 40/20 split that already knows north–south is twice as busy. All three ran on the same network and the same demand, over 10 random traffic patterns the agent never saw during training, with exploration switched off.
| Controller | Avg queue / step | Travel time (s) | Trips |
|---|---|---|---|
Q-learning | 5.31 ± 0.07 | 37.71 ± 0.13 | 1190.3 |
| Fixed 30/30 s | 14.64 ± 0.24 | 43.94 ± 0.23 | 1190.0 |
| Fixed 40/20 s | 12.66 ± 0.18 | 42.12 ± 0.18 | 1190.6 |
The agent cuts queuing by 63.7% against the plain timer and 58% against the tuned one, while moving the same number of cars. The second number is the one I care about more. The 40/20 timer already has the right average split, so the only thing left to win is reacting to the moment-to-moment ups and downs around that average. That's exactly what the agent does.
The demo
SUMO is a heavy native program, and I didn't want the demo to depend on it. So I recorded every second of real evaluation runs (car positions and signal states) into small compressed files, and the app just replays them with a bit of JavaScript. It loads instantly, and every frame is still a genuine simulation. There's also a policy explorer where you can set up any queue situation and see what the agent would do and why.
What it doesn't do (yet)
It's one intersection, through traffic only, with a fixed demand pattern. Real junctions have turns, several lanes and neighbours that affect each other. Turn-aware, multi-lane control, and coordinating several intersections, are the obvious next steps, and that's probably where a table stops being enough.