Build a Local LLM Gateway with LiteLLM: Route Ollama and LM Studio Through One API (2026)

Your application should not need to know which local model runner happens to be active today. A project may start with Ollama and later need LM Studio for a different model or workflow. If every application hard-codes its own provider URL and model name, changing that setup becomes a migration every time.

A local gateway gives you one stable API in front of multiple model runtimes. This guide configures LiteLLM Proxy to expose Ollama and LM Studio behind named model aliases, authenticate incoming requests, and attempt a secondary route when the primary route fails. It is useful for developers building internal tools, prototypes, and local AI workflows.

Documentation checked on 22 October 2026. Provider syntax can change; compare the configuration with the official documentation for the version you install. The steps below are a configuration walkthrough, not a claim of hands-on testing on your machine.

What you are building

Your application
      |
      v
LiteLLM Proxy :4000
   |         |
   v         v
Ollama    LM Studio
:11434    :1234/v1

Your client sends a model alias such as local-ollama. LiteLLM routes the request to the corresponding provider and returns an OpenAI-style response. A fallback rule can try the configured LM Studio alias when a request to the Ollama alias fails in a way LiteLLM handles as a provider failure.

• Provides: one client endpoint, stable aliases, a central place for request authentication, and configurable retries and fallback rules.
• Does not guarantee: equal model quality, identical context limits, identical tool-calling behavior, zero downtime, or compatibility with every parameter across providers.
• Scope: this guide uses a simple config-file setup with a gateway master key. Database-backed virtual keys, budgets, and persistent spend tracking require additional infrastructure.

Who should use this setup?

Use it if multiple applications need the same local AI endpoint, you want to switch model runners without changing each client, or you need a deliberate fallback policy. If one small script talks to one model and will remain that way, direct access to Ollama or LM Studio may be simpler.

Prerequisites

• Python and pip, plus a virtual environment for the gateway.
• Ollama installed and running with at least one model.
• LM Studio installed, with a model available to its local server.
• Enough free RAM/VRAM for the models you select. A gateway does not reduce model memory requirements.

Step 1: Verify both model runners

Check Ollama

Run the following in a terminal and make sure Ollama is running:

ollama --version
ollama list

Then check its local API at http://localhost:11434/api/tags. It should return JSON describing the models available to that Ollama server. Copy the exact model tag from your own list; do not assume an example tag is installed. The endpoint is documented in the official Ollama API reference.

Check LM Studio

Start the local server from LM Studio's Developer area or its CLI. The documented OpenAI-compatible server examples use port 1234. Open http://localhost:1234/v1/models and note the exact model identifier returned by the endpoint. If server authentication is enabled, follow LM Studio's authentication documentation.

Reference: LM Studio OpenAI-compatible endpoints.

Step 2: Install LiteLLM in an isolated environment

Create a dedicated directory and virtual environment:

mkdir local-llm-gateway
cd local-llm-gateway
python -m venv .venv

Activate it with source .venv/bin/activate on macOS/Linux or .venv\Scripts\Activate.ps1 in Windows PowerShell. Then install the proxy extra:

python -m pip install --upgrade pip
python -m pip install "litellm[proxy]"

Check that the command is available with litellm --help. LiteLLM also provides an official container-based setup; see the Docker quick-start guide. If the gateway runs in a container while a model runner is on the host, remember that localhost inside the container refers to the container, not the host. Configure a host-reachable address for your operating system instead.

Step 3: Configure model aliases and fallback

Set these environment variables in the shell that will run LiteLLM: LITELLM_MASTER_KEY should contain a long, random secret; LM_STUDIO_API_BASE should be http://localhost:1234/v1; and LM_STUDIO_API_KEY should contain the LM Studio token if authentication is enabled. When LM Studio authentication is disabled, use the value recommended by its current provider integration or leave the key unset if your configuration supports that.

Create config.yaml and replace the example Ollama tag and LM Studio model identifier with values confirmed on your own machine:

model_list:
  - model_name: local-ollama
    litellm_params:
      model: ollama_chat/qwen2.5-coder:7b
      api_base: http://localhost:11434

  - model_name: local-lmstudio
    litellm_params:
      model: lm_studio/REPLACE_WITH_EXACT_MODEL_ID
      api_base: os.environ/LM_STUDIO_API_BASE
      api_key: os.environ/LM_STUDIO_API_KEY

litellm_settings:
  num_retries: 1
  request_timeout: 120
  fallbacks:
    - local-ollama: ["local-lmstudio"]

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

Understand the important fields:

• model_name is the public alias your application sends to LiteLLM. It need not match the provider's native model tag.
• litellm_params.model selects the provider adapter and underlying model. LiteLLM's Ollama documentation recommends the ollama_chat route for chat workloads; see the Ollama provider guide.
• The LM Studio route uses the lm_studio/ prefix and an API base supplied through an environment variable. See LiteLLM's LM Studio provider guide.
• The fallback is one-way: a request to local-ollama may fall back to local-lmstudio. A request to local-lmstudio does not automatically fall back to Ollama.
• A 120-second timeout is an example, not a universal setting. Cold starts and long prompts may need more time; shorter tasks may need less.

Check the official config reference before using this in production. Pin a tested LiteLLM release rather than depending on a floating package or container tag.

Step 4: Start the gateway

With the virtual environment active and the required variables set, start the proxy:

litellm --config config.yaml --port 4000

Keep the terminal open for the first test. Review startup output for YAML errors, missing environment variables, provider initialization failures, or a port conflict. A process that starts does not prove that either backend is reachable.

Step 5: Test both aliases independently

Use the gateway's OpenAI-compatible Chat Completions endpoint at http://localhost:4000/v1/chat/completions. Include your master key as a bearer credential and send a JSON body like this:

{
  "model": "local-ollama",
  "messages": [
    {"role": "user", "content": "Explain a Python dictionary in two sentences."}
  ],
  "stream": false
}

The expected result is an OpenAI-style JSON response containing an assistant message. The exact wording varies by model. Repeat with "model": "local-lmstudio". Each alias should succeed on its own before you rely on fallback. This separates a backend connectivity issue from a routing issue.

If you receive a provider error, test the same model directly against its runner, check the exact model identifier, then inspect LiteLLM's log for the upstream URL and failure reason.

Step 6: Connect a Python application

Install the OpenAI client in your application's environment:

python -m pip install openai

Then point it at LiteLLM instead of a provider-specific URL:

import os
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:4000/v1",
    api_key=os.environ["LITELLM_MASTER_KEY"],
)

response = client.chat.completions.create(
    model="local-ollama",
    messages=[
        {
            "role": "user",
            "content": "Suggest three edge cases for an email validator."
        }
    ],
)

print(response.choices[0].message.content)

After this works, the client can select another configured alias without changing the endpoint. Keep the master key out of source control. It is an administrator credential for this simple setup, not a per-user key for a shared service.

Step 7: Verify fallback behavior safely

A fallback is not quality-based routing. LiteLLM does not compare the two answers and select the better one; it attempts the configured secondary route after a qualifying failure and the configured retry behavior.

1. Confirm both aliases work independently.
2. On a development machine, create a temporary test configuration where the primary route points to an intentionally unreachable local port. Do not change a shared or production configuration to test this.
3. Start the test gateway on a different port and send a request to local-ollama.
4. Inspect logs and the response to see whether the request moved to the secondary deployment. If it fails, read the exact error before changing the fallback policy.
5. Restore the correct address and re-test both routes before using the gateway for real work.

Fallback may not help with malformed requests, unsupported parameters, a secondary model that lacks the required capability, or a failure affecting both runners. The backup model may also produce a different answer. See LiteLLM's model-fallback guide for current routing behavior.

Trade-offs

A gateway can be unnecessary complexity for one script. For multiple apps or teammates, a stable endpoint may be worth the extra service. Measure real latency and error rates with your own prompts and hardware; do not infer performance from the number of configured backends.

Security and privacy checklist

• Keep services private by default. Do not expose ports 4000, 11434, or 1234 directly to the internet. Remote access needs deliberate network design, firewall rules, authentication, and TLS.
• Protect the master key. Store it in an environment or secret manager, keep it out of Git, and rotate it if exposed.
• Verify the data path. Local model routes do not prove that every client, extension, update check, gateway log, or surrounding application stays offline. Confirm the complete path before sending sensitive data.
• Review logs and retention. Prompts and outputs can contain secrets or personal information. Restrict access and choose logging settings deliberately.
• Scope shared access. LiteLLM's virtual-key documentation explains database-backed keys. Do not distribute the gateway's master key to ordinary clients.
• Check fallback destinations. A fallback receives the prompt too. Do not add a cloud model as a backup for private data unless that transfer is explicitly allowed.
• Pin and update deliberately. The official LiteLLM changelog listed v1.106.1 dated 20 October 2026 when checked. Review release notes and breaking changes before upgrading a shared gateway.

Troubleshooting

What changes for a shared deployment?

This config-file setup is for private development, not a complete multi-user platform. If you need per-application keys, model permissions, budgets, and persistent spend records, follow the database-backed gateway quick start, which provisions PostgreSQL alongside LiteLLM. Current documentation states that virtual keys and spend tracking need a database; the no-database path does not enforce database-backed budgets.

For production, use a pinned release, TLS at a trusted ingress, restricted network paths to model servers, secret management, database backups, health checks, and monitoring. See the official production deployment guide. For remote backends, use private addresses and firewall allowlists instead of exposing raw inference ports.

Related GyanAangan guides

• Ollama CLI commands: models, storage paths, and Modelfiles — manage the models behind your route.
• llama.cpp Server and its OpenAI-compatible API — another local serving option.
• llama.cpp Router Mode — compare runtime-specific routing with a multi-provider gateway.
• How to secure a remote Ollama server — review network risks before allowing other machines to reach a model API.
• Continue + Ollama in VS Code — connect local models to a coding workflow.

Frequently asked questions

Does LiteLLM make Ollama and LM Studio run the same model?

No. LiteLLM routes requests to the model configured for each backend. Download or load models separately and test their context limits, quality, and supported features.

Can I change the model without changing my application code?

Usually, if the client already calls the gateway and you keep the same public alias. You may still need to adjust prompts or parameters because replacement models can behave differently.

Does fallback guarantee success?

No. It can help with certain provider failures when the backup is healthy and supports the request. Invalid inputs, unsupported features, and failures on both backends can still return errors.

Can I use virtual keys without PostgreSQL?

The current gateway documentation requires a database for virtual-key management and persistent spend tracking. Keep the master-key-only example private and use the database-backed setup for shared access.

Is this automatically offline?

No. The configured model routes are local, but that alone does not prove every component, client, extension, update check, or log stays offline. Verify the complete data path against your privacy requirements.

Final verification checklist

• Both model runners respond to their own API checks.
• Configured model identifiers match the values visible on your machine.
• LiteLLM starts with the intended config and a private master key.
• Each alias works independently through the gateway.
• Fallback behavior is tested on a development setup and understood as failover, not quality routing.
• The gateway and model ports are not exposed publicly, and secrets are not committed to source control.
• Any shared deployment has database-backed access control, backups, TLS, monitoring, and a pinned release plan.

The main benefit of a local LLM gateway is not that it makes a model magically better. It gives your applications a stable, testable boundary between application logic and a changing local AI stack. Start with two working aliases, verify each request path, and add routing complexity only when it solves a real operational problem.