Common errors, what they mean, and how to handle them.
INSUFFICIENT_BALANCE
Your Vault balance is less than the estimated cost for this run (timeout x GPU hourly rate).
Fix: Add funds to your Vault in the dashboard, or reduce the timeout to lower the reserve amount.
GPU_UNAVAILABLE
The requested GPU tier has no available workers right now. This is rare but can happen during peak demand.
Fix: Retry after a few seconds, or try a different GPU tier. H100s have the most limited supply.
TIMEOUT
Your code did not complete within the specified timeout. The process was killed.
Fix: Increase the timeout parameter, optimize your code, or use a faster GPU tier.
CODE_ERROR
Your Python code raised an unhandled exception. The traceback is available in the stderr field of the response.
Fix: Check stderr for the full traceback. Common causes: missing packages, CUDA OOM, import errors.
RATE_LIMITED
You have exceeded the maximum number of concurrent runs for your plan. Default limit is 5 concurrent runs.
Fix: Wait for existing runs to complete, or contact support to increase your concurrency limit.
When an error occurs, the MCP tool returns an error response that your agent can interpret. Most agents will automatically retry or ask you for guidance. Here are patterns for common scenarios:
OOM (Out of Memory)
stderr will contain “CUDA out of memory”. Tell your agent to retry with a larger GPU tier, reduce batch size, or use lower precision (fp16/int8).
Missing packages
stderr will show ModuleNotFoundError. Add the missing package to the packages parameter. The base image includes PyTorch, numpy, scipy, PIL, and transformers.
Network errors in code
Containers have outbound internet access for downloading models from HuggingFace, etc. If downloads timeout, increase the run timeout to account for download time.
GPU_UNAVAILABLE
Retry with exponential backoff: 5s, 10s, 20s. Or fall back to a different GPU tier.
RATE_LIMITED
Wait for at least one in-flight run to complete before retrying. Use list_gpu_runs to check active runs.
TIMEOUT
Do not blindly retry with the same timeout. Either increase the timeout or optimize the code. Retrying at the same timeout will likely fail again and waste credits.
CODE_ERROR
Do not retry without fixing the code. Read stderr, fix the bug, and resubmit.
INSUFFICIENT_BALANCE
Do not retry. This requires user action to add funds.
Need help? Reach out at [email protected]