SantaBot: Instruction-Following Manipulation at the LeRobot Hackathon

The Hackathon and the Idea

For the AMD × LeRobot 2025 hackathon, I built SantaBot — a festive tabletop robotic system where a child directs Santa’s robot to pick colored gifts from a central pile and place them into matching destination squares. The Christmas wrapper hides a serious question: can a small robot learn to follow instructions for manipulation, rather than just memorize a fixed color-matching rule?

Underneath the game are the same challenges that make real warehouse and logistics manipulation hard — clutter, varied object orientations, and lighting variability — which makes it a nice miniature of kitting, bin-packing, and tabletop sorting tasks.

SantaBot in action: the arm grasps a colored gift block from the central pile and delivers it to its target square.

The team-22 SantaBot workstation — The team-22 workstation: two SO-ARM manipulators framing the printed board, an overhead ring light for controlled illumination, and the five colored gift blocks staged for a run.

Collecting the Data

Imitation learning is only as good as its demonstrations, so data collection got the most attention:

305 episodes, roughly 40,000+ frames total.
145 night-time episodes with deliberate overhead LED sweeps (warm / white / blue) to inject lighting variation.
160 daytime episodes to balance the set and reduce illumination bias.
Scenes ranged from clean, ideal layouts to deliberately cluttered ones.
Objects: 3D-printed LEGO-like blocks in five colors — red, orange, brown, light-blue, dark-blue — picked by a 3D-printed parallel gripper mounted on the SO-ARM. Additive manufacturing let us iterate block geometry and gripper jaws overnight to tune graspability.

The key design choice: the dataset intentionally includes non-identity color mappings — e.g. “brown → yellow square,” “light-blue → green square.” This forces a model to actually follow the instruction instead of trivially matching a block’s color to a same-color target. It’s a small trick that turns a color-matching demo into a genuine instruction-following benchmark.

The five colored gift blocks lined up beside the board

Left: the five gift blocks — red, orange, brown, light-blue, dark-blue — staged beside the numbered board. Right: a mid-run board state, with the deliberate clutter and orientation variety that keeps the task from being trivial.

Two Policies, Compared

I trained and compared two imitation-learning policies from the LeRobot ecosystem:

ACT — a behavior-cloning baseline. It performed well, achieving reliable pickup and generally correct transport toward the target regions.
smolVLA — an instruction-conditioned vision-language-action model, trained on natural-language prompts like “pick the red cube and put it in the red square.”

Results and Limitations

ACT was the reliable workhorse: consistent grasps and correct transport toward the right region. Its main weakness was placement precision — objects sometimes landed a cell away from the intended square. The VLA approach is the more exciting direction because it conditions on language, but it needed additional tuning to reach consistent performance in the hackathon timeframe.

Close-up of the gripper picking a block from the pile — A grasp from close range — the placement-precision challenge is visible here: getting the block reliably onto a single target cell is harder than picking it up.

The honest takeaway: behavior cloning gets you a dependable pick-and-transport policy quickly, while instruction-conditioned VLAs promise real generalization to novel commands but demand more data and tuning to pay off. That trade-off — reliable-but-rigid vs. flexible-but-hungry — is exactly the tension I work on day to day in extending VLMs into Vision-Language-Action models for robotic manipulation.

Code: github.com/sagarverma/AMD_Hackthon_2025_team22.