SantaBot: Instruction-Following Manipulation at the LeRobot Hackathon
The Hackathon and the Idea
For the AMD × LeRobot 2025 hackathon, I built SantaBot — a festive tabletop robotic system where a child directs Santa’s robot to pick colored gifts from a central pile and place them into matching destination squares. The Christmas wrapper hides a serious question: can a small robot learn to follow instructions for manipulation, rather than just memorize a fixed color-matching rule?
Underneath the game are the same challenges that make real warehouse and logistics manipulation hard — clutter, varied object orientations, and lighting variability — which makes it a nice miniature of kitting, bin-packing, and tabletop sorting tasks.
Collecting the Data
Imitation learning is only as good as its demonstrations, so data collection got the most attention:
- 305 episodes, roughly 40,000+ frames total.
- 145 night-time episodes with deliberate overhead LED sweeps (warm / white / blue) to inject lighting variation.
- 160 daytime episodes to balance the set and reduce illumination bias.
- Scenes ranged from clean, ideal layouts to deliberately cluttered ones.
- Objects: LEGO-like blocks in five colors — red, orange, brown, light-blue, dark-blue.
The key design choice: the dataset intentionally includes non-identity color mappings — e.g. “brown → yellow square,” “light-blue → green square.” This forces a model to actually follow the instruction instead of trivially matching a block’s color to a same-color target. It’s a small trick that turns a color-matching demo into a genuine instruction-following benchmark.
Two Policies, Compared
I trained and compared two imitation-learning policies from the LeRobot ecosystem:
- ACT — a behavior-cloning baseline. It performed well, achieving reliable pickup and generally correct transport toward the target regions.
- smolVLA — an instruction-conditioned vision-language-action model, trained on natural-language prompts like “pick the red cube and put it in the red square.”
Results and Limitations
ACT was the reliable workhorse: consistent grasps and correct transport toward the right region. Its main weakness was placement precision — objects sometimes landed a cell away from the intended square. The VLA approach is the more exciting direction because it conditions on language, but it needed additional tuning to reach consistent performance in the hackathon timeframe.
The honest takeaway: behavior cloning gets you a dependable pick-and-transport policy quickly, while instruction-conditioned VLAs promise real generalization to novel commands but demand more data and tuning to pay off. That trade-off — reliable-but-rigid vs. flexible-but-hungry — is exactly the tension I work on day to day in extending VLMs into Vision-Language-Action models for robotic manipulation.