SantaBot

Instruction-following tabletop pick-and-place robot built for the AMD × LeRobot 2025 hackathon.

Instruction-Following Manipulation

Built for the AMD × LeRobot 2025 hackathon, SantaBot is a festive tabletop system where a child directs Santa’s robot to pick colored gifts from a central pile and place them into matching destination squares. The Christmas wrapper hides a real question: can a small robot follow instructions for manipulation rather than memorize a fixed color-matching rule? Underneath are the same challenges as warehouse manipulation — clutter, varied orientations, and lighting variability.

SantaBot grasps a colored LEGO block from the central pile and places it on the target square.
SantaBot workstation at the hackathon
Board with five colored LEGO gift blocks
The team-22 rig: two SO-ARM manipulators, an overhead-lit board, and the five colored gift blocks the policy must sort by instruction.

Data and Models

  • 305 episodes (~40,000+ frames): 145 night-time episodes with overhead LED sweeps (warm / white / blue) and 160 daytime episodes to reduce illumination bias.
  • Five-color LEGO-like blocks, with deliberate non-identity color mappings (e.g. brown → yellow square) to force genuine instruction-following.
  • Two imitation-learning policies compared: ACT (behavior-cloning baseline) and smolVLA (instruction-conditioned VLA).

Results

ACT was the reliable workhorse — consistent grasps and correct transport — with placement precision as its main limitation (objects sometimes landing a cell off). The instruction-conditioned VLA is the more promising direction for generalization but needed additional tuning within the hackathon timeframe.

Gripper approaching blocks on the board
Overhead view of the board with placed blocks
Wrist-camera-style close-ups of a grasp and an overhead board state — the clutter and orientation variety the policy has to handle.

Code: github.com/sagarverma/AMD_Hackthon_2025_team22 · Read the blog post.