Celeste RL

Overview

Celeste RL is continual reinforcement learning research with Celeste as the testbed. When an agent that has learned one room is trained on the next, how much of the first does it lose, and how should that loss be measured?

The agent reads the game’s state and presses only the buttons in its controls menu, on unmodified game physics. Most of the project is Python: a Gymnasium environment, PPO training with Stable-Baselines3 and PyTorch, and experiments whose plans are committed before the data exists. A small C# plugin lets the game step as fast as the agent can act.

The main result: a policy that had learned Room 1 was taught Room 2, first by imitation and then with PPO. The imitation step cost 12 to 14 points of Room 1 success. Under the standard PPO recipe, Room 2 training then cut a further 37 to 56 points, in all four lineages. Two variants, one without PPO’s entropy bonus and one anchored to the original Room 1 behaviour, lost far less, and their median Room 2 results were not worse.

Scope: four lineages from one starting network, two rooms, one game. The comparisons with the standard recipe are descriptive, not controlled, and the technical report is explicit about what they can’t show.

Why Celeste

Neural network policies tend to lose old skills when they’re trained on new ones. The field calls it catastrophic forgetting. It’s easy to talk about and hard to measure cleanly, and Celeste’s first two rooms make a small, honest test:

• Distinct tasks. Each room has its own layout and a clear success criterion: leave the room.
• Shared controls and observations. The same network can play both rooms, so Room 2 training can interfere with Room 1.
• Deterministic. Room 1 skill can be scored on exactly the same 200 start states at every stage of training.
• Fast. The environment I built steps at about 2,600 frames a second, so each stage can be measured, not guessed.

The pipeline

Every measurement refers to one stage of this pipeline:

1. ImitationRoom 1 cloneOne behaviour-cloning network, fitted on five clears of Room 1 found by an automated search. No human play.
2. PPODonorA 500k-step PPO fine-tune of the clone on Room 1, from varied starts. Four donors, one per training seed, start the four lineages.
3. ImitationMixed copyA network fitted from the donor’s weights to imitate the donor’s own Room 1 behaviour and the Room 2 demonstrations at the same time.
4. PPORoom 2 training500k steps of PPO on Room 2 from the copy, under three recipes: standard, no entropy bonus (E0), and an anchor to the donor’s Room 1 button probabilities (A1).
5. ResultFinal policyScored on Room 1’s 200 frozen starts, and on Room 2.

All four donors descend from the same clone, so the independent sample size is four lineages, however many episodes sit behind each score. Repeated runs on one copy add precision, not independence.

Measuring forgetting

Room 1 skill is scored as route-macro success: 200 frozen start states on 11 routes, one episode per start, with each route weighted equally so a heavily sampled route can’t dominate.

A raw drop in points is hard to compare between lineages that started at different levels, so losses are also expressed as a share of the donor’s margin over a reference: the mean Room 1 score of policies that never saw Room 1. A loss of 0.25 means a quarter of that margin was lost, not 25 points.

Where Room 1 skill went

Room 1 success at each stage, by lineage. E0 and A1 ran twice on each copy.

Two stages lost Room 1 skill:

• The mixed copy, before any Room 2 PPO. Copies scored 12 to 14 points below their donors. That step also imitates Room 2, so it can’t be called a copying loss: it mixes imitation error with interference from the new room.
• Room 2 PPO, under the standard recipe. A further 37 to 56 points, in every lineage. Three of the four finals scored slightly below the weakest measured policy that had never seen Room 1.

The standard recipe still learned Room 2: its finals cleared the normal Room 2 start in 35 to 46 of 50 episodes. It just paid for it with Room 1.

Two recipes that forgot less

Run by run, Room 2 is more mixed: 4 of the 16 E0 and A1 runs scored below their own lineage’s standard run on the Room 2 start set, by 0.6 to 8.9 points. So the data don’t show a Room 2 cost, and they don’t show a benefit either.

• Why these are descriptive. The standard runs came from an earlier campaign that ran one at a time; the variants ran later, three at a time. There is one standard run per lineage. It’s a comparison worth following up, not a measured effect.
• Why they lose less is open. Removing the entropy bonus also changes which Room 2 states the policy visits. The anchor trains on recorded donor play, so rehearsal and retention are tangled together.
• Not a new algorithm. A1 closely resembles behaviour-cloning replay methods such as CLEAR (Rolnick et al. 2019), and FLaRe (Hu et al. 2025) has reported entropy bonuses damaging pretrained policies during fine-tuning. The contribution is the measurement, not the method.

When the measurement changes the answer

The other half of the report is about measurement. Three choices changed what the same data seemed to say:

• The baseline checkpoint. Counted from the copy, as declared, A1’s median loss is 0.06. Counted from the donor, it’s 0.30, because every arm inherits the copy step’s loss. Both are legitimate, so a retention number has to say which one it is.
• The start distribution. Two policies that scored 98.0% and 98.5% from Room 1’s normal start scored 59.7% and 84.6% across varied held-out starts. In a deterministic game, a fixed start can reward a memorised path.
• Subgroups and their baselines. Split by whether starts were near states the copy was trained on, A1 beat E0 by 9.3 points near and 0.8 far, which reads as “the anchor only protects rehearsed states”. But E0 barely lost anything far from them either (3.2 points), so there was almost nothing left to protect. The data can’t tell those stories apart.

What it doesn’t show

• Why forgetting happens here. The retained training records can’t reconstruct the PPO updates needed to attribute it.
• Anything beyond this pipeline. Four lineages share one clone and one demonstration set, with one copy fit per donor. No claim extends to other recipes, rooms or games.
• A confirmatory result. The same Room 1 start set carried several analyses, some chosen after earlier ones were seen. No fresh study was run.

What would settle it: independent lineages beyond one clone, more than one copy per donor, an exposure measure defined along trajectories, standard and variant runs in the same campaign, and update-level records of PPO training.

How the agent learned the rooms

Forgetting can only be studied once there’s something to forget. Getting there took three experiments, and the first one failed.

A Go-Explore-style search, which replays savestates and reads the room’s tile map, then found seven complete clears of Room 1 with no human play. Behavioural cloning on five of them, followed by PPO, cleared the room. But four runs from the same clone and settings scored 82.5%, 98.0%, 98.5% and 0%, so single runs meant little, and every later experiment used matched seeds.

Next, a predeclared comparison: fine-tune from the fixed start, or from varied starts? On 200 frozen held-out states from 11 routes, varied starts won by 22.6 points (52.2% to 74.8%, 95% interval +6.7 to +43.5, exact p = 0.031), in all six matched seeds.

The same design in Room 2 didn’t replicate: −5.8 points (interval −18.7 to +3.4, p = 0.50). Varied starts helped three seeds and hurt three, and training collapse turned out to hit fixed-start runs too (5 of 14). Both results are published because the plans said they would be. An earlier, post-selected version of the Room 1 comparison was withdrawn and replaced, and its numbers stay in the report as an audit trail.

The varied-start Room 1 policies from seeds 3 to 6 became the four donors for the forgetting study.

The environment underneath

None of this works without an environment that is fast, exact and stable. Celeste’s community automation tool, CelesteTAS, plays inputs from a file and reports state over HTTP, tied to the game’s 60 Hz clock: about 30 steps a second, or about 18 hours of collection for one seed of the first experiment. So I built a faster path, from the agent down to the game:

1. PythonPPO policyPyTorch and Stable-Baselines3, with a small CNN over the room grid.
2. PythonGymnasium environmentBuilds observations, computes reward and classifies how each episode ended.
3. PythonLockstep clientThe same reset() and step() as the slow reference bridge, over a local socket.
4. C#Lockstep pluginHooks CelesteTAS’s playback loop and replies with the new state after every frame.
5. GameCeleste with EverestThe normal update with the normal frame length. Physics are never modified.

Speed isn’t enough on its own. In RL, a subtly wrong environment doesn’t crash; it teaches the agent something false, and the results look plausible. So the slow file-based bridge stayed in the project as the reference, and the fast one had to match it.

• A seed that wasn’t a seed. Two runs with different seeds produced byte-identical episode logs. Stable-Baselines3’s load() re-seeds torch, NumPy and Python globally with the saved checkpoint’s seed, after the run has seeded itself, so every run that started from a checkpoint used the donor’s random stream. Fixed by re-seeding after loading. For a study built on matched seeds and lineages, this one mattered.
• Headless mode. It crashed in the game’s audio library after a savestate. Rather than patch around it, the project kept the normal renderer and made stepping fast instead.

What I learned