// category · 10 tools · 3 open source
LLM Inference & Hosting Tools
Serving engines, local runtimes, gateways and cloud platforms for running models fast and cheaply in production or on your laptop.
Every agent needs somewhere to run its model. The tools in this category cover the four common answers: run the model on your own machine, serve it on your own GPUs, rent it per token from a hosted API, or put a gateway in front of several providers.
On your own machine. Ollama downloads open models and serve them through a local API, which is enough for development, private data and offline work. Our LangChain with Ollama guide shows how to build a local agent, including the context window settings that trip most people up.
On your own GPUs. vLLM is a widely used open source serving engine for production traffic, built around PagedAttention for many concurrent requests. It exposes an OpenAI compatible server, so frameworks and SDKs can switch to a self hosted model by changing a base URL.
Hosted APIs for open models. Fireworks AI, Together AI and Groq serve popular open models per token behind OpenAI compatible APIs. Baseten, Modal and Replicate focus on deploying your own or fine tuned models on GPUs billed by the second or minute. For example, the Fireworks pricing page lists serverless billing per token and on demand GPUs per second.
Gateways. LiteLLM is an open source proxy you run yourself to hold provider keys, set budgets and fail over between providers, and OpenRouter is a managed API over many providers. Our LLM gateway vs OpenRouter guide explains when to use each, and LiteLLM compromised covers the October 2026 supply chain incident and how to pin a gateway safely.
How to choose: start with a hosted API while you learn what the agent needs, measure tokens per task, then decide whether privacy, latency or cost justifies moving to dedicated GPUs or your own hardware. Agents make many model calls per task, so per token prices and rate limits matter more than they do for a chatbot. Prices change often; confirm them on each vendor's pricing page. Official reference: the vLLM documentation.
Ollamaollama.comThe simplest way to download and run open models locally, with a CLI, desktop app and local API.Inference & HostingOpen source★ 185.1k
LiteLLMlitellm.aiPython SDK and proxy gateway that calls many LLM providers through one OpenAI style interface with budgets and logging.Inference & HostingOpen source★ 62.2k
Basetenbaseten.coProduction inference platform for deploying open and custom models with dedicated, autoscaling GPUs.Inference & HostingPaid★ 1.2k
OpenRouteropenrouter.aiUnified API to hundreds of models from many providers with automatic fallbacks and price routing.Inference & HostingFreemiumHN 38
Modalmodal.comServerless platform for running Python and GPU workloads, from inference endpoints to sandboxes, defined in code.Inference & HostingFreemiumHN 21
vLLMdocs.vllm.aiHigh throughput LLM serving engine built around PagedAttention, with an OpenAI compatible server.Inference & HostingOpen source★ 95.7k
Replicatereplicate.comRun and deploy open models through a simple API, packaging custom models with Cog.Inference & HostingPaid★ 9.5k
Together AItogether.aiCloud for running, fine tuning and training open models, plus on demand GPU clusters.Inference & HostingPaidHN 4
Fireworks AIfireworks.aiFireworks AI serves open models through OpenAI and Anthropic compatible APIs, billed per token on serverless or per GPU second on dedicated deployments.Inference & HostingPaidHN 2
Groqgroq.comInference API running on custom LPU chips, known for very fast token generation.Inference & HostingFreemiumHN 1// guides
Guides to LLM inference & hosting tools
LLM Gateway vs OpenRouter: Which to Use?
What an LLM gateway is, and when to self host one like LiteLLM instead of using OpenRouter: routing, fallbacks, caching, keys, fees and security in 2026.
Is LangChain Free for Commercial Use?
Is LangChain free for commercial use? Yes. LangChain and LangGraph are MIT licensed. Here is what that allows, where costs start, and the one server that isn't MIT.
Vercel AI SDK Guide: Build an Agent With AI SDK 7
The Vercel AI SDK is a free, open source TypeScript toolkit for LLM apps and agents. AI SDK 7 changes, a tested ToolLoopAgent, streaming UI, MCP and costs.
LiteLLM Compromised: What to Do About 1.107.0
LiteLLM compromised again: release 1.107.0 on PyPI and Docker carried a credential stealer. How to check every host, what to rotate and how to pin safely.
GitHub Copilot Alternative for Visual Studio (2026)
The best GitHub Copilot alternative for Visual Studio in 2026: Antigravity, JetBrains Junie, Claude Code, your own model key, Tabnine and CodeRush, checked Oct 27.
Best Design Skills for Claude Code (2026)
Impeccable, Hallmark, Taste Skill and Anthropic's frontend design skill compared: what each one does, how to install it, and which to pick for your agent.
NVIDIA OpenShell: What It Is and How to Use It
NVIDIA OpenShell explained: how the open source agent sandbox enforces file, network and credential policy, how to install it, and how it compares to Docker.
Claude Managed Agents: Pricing and How It Works
Claude Managed Agents explained: what Anthropic hosts, the $0.08 session hour plus token pricing, self hosted sandboxes, budgets, limits and open source options.
How to Build an AI Agent from Scratch in Python
How to build an AI agent from scratch in Python: a tool loop, safe file tools, approvals, a step cap and memory in about 125 lines, on OpenAI or Ollama.
How to Build a RAG Pipeline with LangGraph
How to build a RAG pipeline with LangGraph: index your docs, wire retrieve and generate nodes, add a grading loop with a retry cap, then turn it into an agent.
How to Use LangChain with Ollama (2026 Guide)
How to use LangChain with Ollama: install langchain-ollama, pick a tool calling model, raise the context window and build a local agent with tools and RAG.
Best Pi Coding Agent Alternatives in 2026
Looking for a Pi coding agent alternative? Compare OpenCode, Claude Code, Codex CLI, Copilot CLI, Cline and Aider on plan mode, approvals and local models.
What Is Docker Agent (cagent)? A Practical Guide
Docker Agent, formerly cagent, runs AI agent teams from a YAML file. What it is, how to install and configure it, its MCP and sandbox options, and who it suits.
Best Open Source Coding Agents in 2026
The best open source coding agents in 2026, compared on license, interface, model choice and upkeep: OpenCode, Cline, Codex CLI, Gemini CLI, goose and Aider.
E2B vs Daytona vs Modal: Agent Sandboxes Compared
E2B vs Daytona vs Modal for running AI agent code safely: isolation, startup, persistence, time limits, GPUs and SDKs compared, with code for each sandbox.