Fundamental Research Labs

Fundamental Research Labs isn't a typical software company. They're an a16z and Prosus-backed AI research lab whose team comes from MIT EECS, Stanford NLP Group, Google X, and Citadel. Their flagship product, Shortcut.ai, is an AI-powered analyst. Their Fairies platform powers autonomous AI agents, originally built for Minecraft simulations under their earlier name LyfeGame. Their research program, Project Sid, demonstrated thousands of AI agents developing specialized roles, democratic governance systems, and cultural propagation.

The work they do requires a constant cycle of training model checkpoints, deploying them for inference, testing quality, and evaluating performance against standardized benchmarks. That cycle needs to be fast, reliable, and repeatable. Manual deployment means wasted researcher time. Inconsistent evaluation means unreliable results. No environment isolation means one experiment can break another.

They needed infrastructure purpose-built for ML research workflows: a platform that understands model checkpoints, GPU requirements, inference APIs, and evaluation pipelines as first-class concepts. Not a generic dashboard bolted onto a deployment script. A system where a researcher can go from checkpoint to benchmark results without context-switching between different tools.

That's the half we built, working inside their engineering team.

Deploying an AI model for inference sounds straightforward until you account for the variables. A 7B parameter model needs different infrastructure than a 70B parameter model. Some checkpoints live on HuggingFace, others on a cloud volume. GPU selection depends on model size and throughput requirements. Memory utilization has to be pushed high enough to justify the GPU without crashing the server.

The platform resolves the provider question at deploy time, and on size alone. It estimates parameter count from the checkpoint's safetensors index metadata and torch_dtype instead of guessing from filenames, then picks the path: under roughly 10B parameters goes to Modal with vLLM, where cold starts are fast and idle deployments scale to zero; anything larger goes to Baseten with Truss for multi-GPU inference. GPU tier is configurable per deployment with sensible defaults. Deploys became observable too, with a Server-Sent Events stream reporting progress live and a deployment that stalls past 25 minutes marked failed rather than left hanging.

Every deployment comes up behind an OpenAI-compatible endpoint, so any tool built for GPT works against it. That surface was a colleague's; the layer that points it at the right model was ours. Resolving a model name to its live URL runs through a four-tier lookup ordered by cost, in front of an in-process edge cache with write-through invalidation on deploy and stale-on-error fallback. The most consequential line in that layer is a refusal: when Modal's output doesn't yield a real endpoint URL, the deploy fails loudly instead of constructing a plausible-looking one and handing researchers an address that quietly points nowhere.

Evaluation was the other half of the mandate. We containerized and operationalized EleutherAI's lm-evaluation-harness, the open-source standard rather than something we rebuilt, packaging it as a Modal service and putting a CLI and Airflow DAGs in front of it. Exactly one benchmark is wired: HellaSwag, as the reference path. We also built the task-registration and model-routing scaffolding for the Berkeley Function Call Leaderboard inside the harness fork, with dataset wiring still in progress when the engagement closed.

SWE-Gym evaluates models against real GitHub issues, each task isolated in its own Docker container. The lab's Modal runner for it was invoked by hand from a terminal. We wrote the Airflow DAG that orchestrates it: a concurrency parameter with guardrails at either end, a cancel branch that finds and stops running Modal apps, and runner output streamed line by line into Airflow logs. A benchmark run went from something an engineer babysat in a shell to something the platform could schedule, watch, and kill.

A professional CLI wraps the pipeline for terminal-first researchers. Run evaluations, list available tasks, test configurations, and manage settings from the command line. It supports local HuggingFace models, remote OpenAI-compatible endpoints, quantization options, automatic device detection, and configurable batch sizes.