Quickstart

Deploy your first vLLM endpoint in under a minute

Prerequisites

Before you begin, make sure you have:

Vast.ai Account

Sign up at cloud.vast.ai and add credits to your account

API Key

Generate an API key from your account settings

HuggingFace Token

Create a HuggingFace account and generate a read-access token for gated models

Configuration

Install the Vast SDK

Install the SDK that you’ll use to interact with your serverless endpoints:

pip install vastai

The SDK provides an async Python interface for making requests to your endpoints. You’ll use this after setting up your infrastructure.

API Key Setup

Set your Vast.ai API key as an environment variable:

export VAST_API_KEY="your-api-key-here"

The SDK will automatically use this environment variable for authentication. Alternatively, you can pass the API key directly when initializing the client:

client = Serverless(api_key="your-api-key-here")

HuggingFace Token Setup

Many popular models like Llama and Mistral require authentication to download. Configure your HuggingFace token once at the account level:

1. Navigate to your Account Settings
2. Expand the “Environment Variables” section
3. Add a new variable:
   • Key: HF_TOKEN
   • Value: Your HuggingFace read-access token
4. Click the ”+” button, then “Save Edits”

This token will be securely available to all your serverless workers. You only need to set it once for your account.

Without a valid HF_TOKEN, workers will fail to download gated models and remain in “Loading” state indefinitely.

Deploy Your First Endpoint

1. Open the Serverless Dashboard

   Navigate to the Serverless Dashboard and click “Get Started”.

   
2. Name Your Endpoint and Select a Workload

   Give your endpoint a descriptive name (e.g., vLLM-Qwen3-8B) and select a workload type from the dropdown:

   • LLM Inference: for large language model serving (vLLM, TGI)
   • Image Generation: for image generation workloads (Stable Diffusion, Flux)
   • Text Generation: for text generation tasks
   • Other: for custom workloads

   

   For this quickstart, select LLM Inference.

   

   Click “Continue” to create the endpoint.

   For more control over endpoint parameters and workergroup configuration, click “Advanced setup” instead. See Setting Up Endpoints for details.

3. Start Making Requests

   After creation, you’ll see a Congratulations page with next steps:

   1. Install the Vast.ai SDK: pip install vastai
   2. Get an API key to interact with your endpoint
   3. Copy the provided sample code

   

   The system will automatically provision GPU workers, select an appropriate template, and configure scaling defaults for your workload type.

4. Wait for Workers to Initialize

   Your serverless infrastructure is now being provisioned. This process takes time as workers need to:

   1. Start up the GPU instances
   2. Download the model (8GB for Qwen3-8B)
   3. Load the model into GPU memory
   4. Complete health checks

   Expect 3-5 minutes wait time for workers to become ready, especially on first deployment. Larger models may take longer.

   Monitor the worker status in the dashboard:

   • Stopped: Worker has the model loaded and is ready to activate on-demand (cold worker)
   • Loading: Worker is starting up and loading the model into GPU memory
   • Ready: Worker is active and ready to handle requests

   The SDK automatically holds and retries requests until workers are ready. However, for best performance, wait for at least one worker to show “Ready” or “Stopped” status before making your first call.

Make Your First API Call

Basic Usage

With the SDK installed, here’s how to make your first API call:

import asyncio
from vastai import Serverless
MAX_TOKENS = 100

async def main():
    # Initialize the client with your API key
    # The SDK will automatically use the VAST_API_KEY environment variable if set
    client = Serverless()  # Uses VAST_API_KEY environment variable

    # Get your endpoint
    endpoint = await client.get_endpoint(name="vLLM-Qwen3-8B")

    # Prepare your request payload
    payload = {
          "model": "Qwen/Qwen3-8B",
          "prompt": "Explain quantum computing in simple terms",
          "max_tokens": MAX_TOKENS,
          "temperature": 0.7
    }

    # Make the request
    result = await endpoint.request("/v1/completions", payload, cost=MAX_TOKENS)

    # The SDK returns a wrapper object with metadata
    # Access the OpenAI-compatible response via result["response"]
    print(result["response"]["choices"][0]["text"])

    # Clean up
    await client.close()

if __name__ == "__main__":
    asyncio.run(main())

The SDK handles all the routing, worker assignment, and authentication automatically. You just need to specify your endpoint name and make requests.

Troubleshooting

Workers stuck in 'Loading' state

• Check if the GPU has enough VRAM for your model
• Verify your model name is correct
• Check worker logs in the dashboard by clicking on the worker
• Ensure your HF_TOKEN is properly configured for gated models

'No workers available' error

• The SDK automatically retries requests until workers are ready
• If this persists, check endpoint status in the Serverless Dashboard
• Verify workers are not stuck in “Loading” state (see troubleshooting above)

Slow response times

• First request may take longer as workers activate from cold state
• Increase max_workers if all workers are full with requests
• Increase min_load if there aren’t enough workers immediately available when multiple requests are sent
• If there are large spikes of requests, increase cold_workers or decrease target utilization
• Consider worker region placement relative to your users

---

Need help? Join our Discord community or check the detailed documentation for advanced configurations.