Autonomous AI Research with Autoresearch on Vast.ai

Introduction

Autoresearch is Andrej Karpathy’s framework for autonomous AI-driven ML research. The idea is simple: point an AI agent (Claude Code) at a small but real LLM training setup and let it experiment autonomously overnight. The agent modifies the model code, trains for 5 minutes, checks if the result improved, keeps or discards, and repeats — running ~12 experiments per hour, ~100 overnight.

This guide walks you through setting up autoresearch on a Vast.ai GPU instance with Claude Code as the autonomous research agent.

Prerequisites

• A Vast.ai account with credits
• An Anthropic account for Claude Code
• SSH key pair (add your public key in your Vast account settings)
• Vast CLI installed on your local machine

Install the Vast CLI if you haven’t already:

pip install vastai
vastai set api-key YOUR_API_KEY

Rent a GPU Instance and Set Up

Autoresearch requires a single NVIDIA GPU with 80GB VRAM (H100 or A100 80GB). It needs CUDA 12.8+ and about 50GB of disk for the repo, data, and dependencies.

Use Template

Use the Autoresearcher template to launch a pre-configured instance with uv, Claude Code, and autoresearch already installed.

Learn more about templates

Templates are reusable configurations that bundle a Docker image, environment variables, and startup scripts into a one-click launch.

Search for available instances:

vastai search offers 'gpu_ram>=70 num_gpus=1 cuda_vers>=12.8 disk_space>=50 reliability>0.95' -o 'dph+'

Pick an instance ID from the results and rent it using the template:

vastai create instance INSTANCE_ID \
  --template_hash 934769670bfd9bc5e05d8696ef340c2b \
  --disk 50

Wait for the instance to be ready, then SSH in:

vastai show instances
ssh -p PORT root@HOST_IP

The template installs everything on first boot (~10 minutes). You can monitor progress with tail -f /var/log/provisioning.log.

The template automatically configures Claude Code permissions (Read, Edit, Write, Bash) in .claude/settings.json so it can run experiments without prompting — no manual setup needed.

Once provisioning completes, skip ahead to Launch Autonomous Research.

Manual Setup

Search for available instances:

vastai search offers 'gpu_ram>=70 num_gpus=1 cuda_vers>=12.8 disk_space>=50 reliability>0.95' -o 'dph+'

Pick an instance ID from the results and rent it:

vastai create instance INSTANCE_ID \
  --image vastai/pytorch \
  --disk 50 \
  --ssh \
  --direct

Wait for the instance to be ready:

vastai show instances

Once the status shows running, get your SSH connection details:

vastai ssh-url INSTANCE_ID

SSH in

ssh -p PORT root@HOST_IP

Vast instances start in a tmux session by default. This keeps your processes running if your SSH connection drops — essential for overnight research runs.

Install uv

uv is the package manager used by autoresearch:

curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env

Install Claude Code

Claude Code requires Node.js:

curl -fsSL https://deb.nodesource.com/setup_22.x | bash -
apt-get install -y nodejs
npm install -g @anthropic-ai/claude-code

Clone and install

cd /workspace
git clone https://github.com/karpathy/autoresearch.git
cd autoresearch
uv sync

Prepare the data

This downloads training data from HuggingFace and trains a BPE tokenizer. Takes about 2 minutes:

uv run prepare.py

Data is cached in ~/.cache/autoresearch/ — you only need to run this once.

Run a baseline experiment

Verify everything works by running a single 5-minute training experiment:

uv run train.py

After ~5 minutes you’ll see output like:

---
val_bpb:          0.995583
training_seconds: 300.3
total_seconds:    349.8
peak_vram_mb:     45060.2
mfu_percent:      39.57
total_tokens_M:   497.0
num_steps:        948
num_params_M:     50.3
depth:            8

The key metric is val_bpb (validation bits per byte) — lower is better. Note this baseline number; Claude will try to beat it.

Configure permissions

Claude Code normally asks for permission before running commands or editing files. For autonomous overnight research, you need to pre-approve the tools Claude will use. Create a settings file in the autoresearch directory:

mkdir -p /workspace/autoresearch/.claude
cat > /workspace/autoresearch/.claude/settings.json << 'EOF'
{
  "permissions": {
    "allow": [
      "Read",
      "Edit",
      "Write",
      "Bash"
    ]
  }
}
EOF

This tells Claude Code to run these commands without asking — essential for unattended operation.

Launch Autonomous Research

Start Claude Code

cd /workspace/autoresearch
claude

When Claude Code starts, log in to your Anthropic account:

/login

This will give you a URL to open in your browser. Follow the prompts to authenticate, then you’re ready to go.

Kick off the research loop:

Hi have a look at program.md and let's kick off a new experiment! let's do the setup first.

Claude will:

1. Read program.md for the research guidelines
2. Create a fresh git branch (e.g. autoresearch/mar10)
3. Run the baseline experiment
4. Begin the autonomous loop — modifying train.py, training for 5 minutes, evaluating, keeping improvements, discarding regressions
5. Log all results to results.tsv

Claude runs indefinitely until manually stopped. Each experiment takes ~5 minutes, so you can expect ~12 experiments/hour and ~100 experiments overnight. Each iteration also uses Claude API tokens.

What Claude can modify

Claude has full freedom to edit train.py — the model architecture, optimizer, hyperparameters, batch size, model size, training loop. The only constraints are:

• prepare.py is read-only — the evaluation harness and data loading are fixed
• No new packages — only dependencies in pyproject.toml
• 5-minute time budget — every experiment runs for exactly 5 minutes

Monitoring progress

In another tmux pane (Ctrl+b then %), you can watch the experiment log:

watch -n 30 cat /workspace/autoresearch/results.tsv

Or check the git log to see what Claude has tried:

cd /workspace/autoresearch
git log --oneline -20

Cleanup

When you’re done, download your results and destroy the instance:

# From your local machine — copy results
scp -P PORT root@HOST_IP:/workspace/autoresearch/results.tsv ./results.tsv

# Destroy the instance
vastai destroy instance INSTANCE_ID

Destroying an instance permanently deletes all data on it. Make sure to copy any results you want to keep before destroying.

Additional Resources

• Autoresearch GitHub repo
• Nanochat — the full training framework autoresearch is derived from
• Claude Code documentation
• Vast.ai CLI reference