SantaBot

Instruction-Following Manipulation

Built for the AMD × LeRobot 2025 hackathon, SantaBot is a festive tabletop system where a child directs Santa’s robot to pick colored gifts from a central pile and place them into matching destination squares. The Christmas wrapper hides a real question: can a small robot follow instructions for manipulation rather than memorize a fixed color-matching rule? Underneath are the same challenges as warehouse manipulation — clutter, varied orientations, and lighting variability.

Data and Models

• 305 episodes (~40,000+ frames): 145 night-time episodes with overhead LED sweeps (warm / white / blue) and 160 daytime episodes to reduce illumination bias.
• Five-color LEGO-like blocks, with deliberate non-identity color mappings (e.g. brown → yellow square) to force genuine instruction-following.
• Two imitation-learning policies compared: ACT (behavior-cloning baseline) and smolVLA (instruction-conditioned VLA).

Results

ACT was the reliable workhorse — consistent grasps and correct transport — with placement precision as its main limitation (objects sometimes landing a cell off). The instruction-conditioned VLA is the more promising direction for generalization but needed additional tuning within the hackathon timeframe.

Code: github.com/sagarverma/AMD_Hackthon_2025_team22 · Read the blog post. </content>