AI Now Understands Human Motion Without Encoders — And It's Better

What if the key to making AI understand human movement was simply... describing it in words?

Researchers from Aalto University and Georgia Tech have introduced SMD (Structured Motion Descriptions), a system that converts skeletal motion data into structured text that large language models can read directly — completely eliminating the need for learned motion encoders.

Traditional approaches require complex multi-stage pipelines: training a VAE encoder to compress motion into latent tokens, then training alignment modules to bridge the gap between motion and language. Change the underlying AI model? Start over.

SMD takes a radically different approach. Using biomechanical principles, it deterministically converts 22 joint positions into human-readable descriptions of angles, trajectories, and timing. Think: "Left hip flexion increases from 3° to 81° during 0.0–0.9 seconds."

The results are striking. On motion question-answering benchmarks, SMD achieves 66.7% on BABEL-QA and 90.1% on HuMMan-QA — beating previous state-of-the-art by 6.6 and 14.9 points respectively. Motion captioning scores jumped 31% on CIDEr.

Perhaps most impressive: the same text representation works across 8 different LLMs from 6 model families without modification. Training requires only lightweight LoRA fine-tuning on a single GPU — 7 to 20 hours versus days of multi-stage encoder training.

The system also delivers built-in interpretability. Attention analysis reveals exactly which body parts the model focuses on — hip and knee cycles for walking, shoulder and elbow for waving.

The insight is elegantly simple: meet the LLM in its native modality rather than forcing motion into opaque embeddings.