Mirror Learning

Mirror Learning asks whether there is an evolutionarily conservable system that could enable “watch and learn” style, human-like learning.

PLAI group member Yunpeng Liu and colleagues, under the supervision of Dr. Frank Wood, have written a paper introducing what they call “mirror learning,” a framework that transforms third-person demonstrations into synthetic first person observation and action data for training embodied agents. Humans and animals routinely learn by watching others, but modern imitation learning systems still depend heavily on first-person demonstrations collected through teleoperation or direct interaction. We show that video diffusion models can be adapted to “put the learner in the demonstrator’s shoes,” synthesizing first-person video from third-person observations. Pairing this with actions inferred by an inverse dynamics model results in mirror data that can be used to train effective policies on its own, and can further improve behavior cloning when added to standard first-person data.

Who do we learn from and can we put ourselves in their shoes?

We used CARLA to train a mirror video model which puts a learner in a demonstrators’ visual perceptual shoes. The video above shows two test-time tasks, overlaid for presentation conciseness. The top row shows what the learner sees. We use segment anything to identify the pose and outline of the demonstrator of interest. Here we show mirror video modeling results for two different demonstrators — indicated by orange and blue mask highlights. Because we’re in a simulator we can, for each demonstrator, know the ground-truth egocentric video. The two mirror videos samples from our model, one for each demonstrator, are remarkably good. This was a eureka moment for us; one of those scientific moments where it is clear that this “just works.” Video diffusion models, repurposed in the way we did, have rich enough spatial representations of the world to enable this perspective transformation while also inferring the pose of the demonstrator given only its segmentation mask. As we have previously demonstrated life-long learning of video diffusion models (blog) it suggests, left for future work, that this view-transforming capability can probably be learned entirely locally.
The next aha moment was seeing that this trained-in-simulation-only model zero-shotted to real-world data. May released a dataset in which, occassionally, one robotaxi of theirs saw another robotaxi, meaning, of course, that we could know perceptual ground truth for the demonstrator in those instances. The videos above show mirror video model data in which a “learner” May robotaxi follows and observes a “demonstrator” May robotaxi. Boom. Zero shot.

Remarkably the CARLA data used to trained the mirror video model did not have any pedestrians, yet, quite sensible pedestrian presence and motion inference remains.

Can we learn (to drive) by simply by watching others?

Without going into the boring details the simple answer is yes. You can also train an inverse dynamics model locally, i.e. a model that tells you which action was taken that gave rise to a change in observations, and then apply it to the mirror video data, giving you action labels. While there is inherent noise in both the inverse dynamics model (i.e. you don’t get the action exactly right) and in the mirror video model (you don’t get the pixels exactly right), it turns out that the data is good enough to behavior clone from anyway. We showed in the paper that an end to end self driving policy could be trained from it.

What is probably more interesting is the fact that we can compare real or “ground truth” data vs. mirror data, and when we do, we find that mirror data is about half as effective as first person data to train on. While that may sound like a negative, it is instead a huge win because both the mirror video model and the inverse dynamics model can both be trained locally – neither require anything more than the target embodiment itself.

We haven’t looked at cross-embodiment learning yet so in all experiments the learner and the demonstrators have the same sensor and actuator setup, however, it seems reasonable to expect that this could work even if they don’t. And in any event, the implications for teleoperation versus passive data gathering are profound. If you can train up a good enough inverse dynamics model and world model using your own interactions with the world, you have enough to do mirror learning. So, for instance, heavily instrumenting a single humanoid robot or human demonstrator with tactile, pressure, torque and other sensors might be enough to harvest data passively.

But wait, cars are solved. What about humanoids or other more complex embodiments and environments?

We took instrumented Minecraft videos from a complex multiplayer enviroment where we had ground truth for the demonstrator as well and guess what. It works there too. We were particularly floored by the third column example below where the mirror video model understands picking up items from the environment and modifying the toolbar.

Collaboration, Licensing, and Other Opportunities

Reach out to fwood@cs.ubc.ca for collaboration and other opportunities. The general mirror learning methodology is the subject of a provisional patent application whose rights are currently assigned to inverted.ai. There is a lot still to reduce to practice, including, multi-sensor and multi-sensor-modality variants, more complex action spaces and inverse dynamics models, etc. Various robotics data gathering companies have reached out and we are eager and willing to help out.

Upon acceptance at a major venue we will release source code.