Digital Spark Learns to Seek Light Through Reinforcement Learning

A digital creature called a spark learns to pursue light through reinforcement learning. The spark has a simple brain comprising 4 neurons—two functioning as eyes and two controlling flipper movement. Three parameters define its brain’s wiring: the strength of connections between neurons on the same side of its body, connections between opposite sides, and a base drive value. The learning mechanism is iterative: before each attempt at the task, the wiring changes slightly at random. When performance improves, the wiring shifts in the direction of that random change and the spark receives a reward. When performance worsens, the wiring shifts the opposite way and the spark receives punishment.

The spark progresses from novice to expert over thousands of trials. Initially it can only spin in place. After 50 attempts, it takes its first steps and moves forward, though slowly. By 250 attempts, it has picked up speed but veers away from the light at the last moment. At 500 attempts, it charges boldly toward the light but hasn’t learned to slow down or stop. After 2,000 attempts, it becomes an expert that clings to the light and can turn sharply. Learning follows a nonlinear path with setbacks, wrong turns, and mistakes. Some learners require thousands of attempts to reach their goal.

Despite beginning with different random wiring, hundreds of novice sparks converge toward the same expert behavior. The spark can learn different behaviors based on reward structure: fleeing from or freezing around harmful light, or charging toward light when rewarded for speed. This parallels biological learning—baby songbirds learn songs by babbling, receiving dopamine increases when variations approach their memory of adult songs. While the spark’s 4-neuron system seems trivial compared to the human brain’s approximately 86 billion neurons and over a hundred trillion connections, the underlying principle remains identical: learning navigates through configuration space using feedback as a guide.