Skip to main content
The inference gateway is a lightweight FastAPI proxy that sits between Cloudflare Tunnel and your local Ollama instance. It handles bearer auth, request forwarding, and timeout management.
Optional hosted APIs (OpenRouter, Venice) for evaluation use the same OpenAI-compatible client path but do not use this gateway—they call the vendor’s /v1 directly. That is documented under Hosted inference (testing) and does not change the local-first default.

​
Setup

Interactive walkthrough: tunnel configuration, gateway env vars, and .env setup. When the platform helper script is present (see below), the wizard can start the stack immediately after.

​
Install

The gateway requires the gateway extra:

​
Controls

Add --leave-ollama-running to off or restart if you only want to bounce the tunnel and gateway.
bumblebee gateway on | off | status | restart invoke a platform helper script from the repo: scripts/gateway.ps1 on Windows, or scripts/gateway.sh on macOS and Linux (bash). gateway setup works on any OS for tunnel/token/.env configuration even if you start processes manually.

​
Configuration

Environment variables (set in .env or system env):

​
Security

  • Bind to 127.0.0.1 only — the tunnel handles external access
  • Terminate the tunnel at the gateway, not a broad reverse proxy
  • Use a strong, random bearer token
  • The token lookup order: INFERENCE_GATEWAY_TOKEN → BUMBLEBEE_INFERENCE_GATEWAY_TOKEN → .env keys

​
Helper scripts and OS notes

The home stack needs Ollama, cloudflared, curl, and a Python interpreter where python -m bumblebee.inference_gateway resolves (e.g. your uv/venv environment). If the wrong interpreter is picked up, set BUMBLEBEE_PYTHON to the full path to python3. On macOS, install Ollama and cloudflared (e.g. Homebrew — cloudflared is often under /opt/homebrew/bin or /usr/local/bin). The CLI and scripts look there when cloudflared is not on PATH.

​
Windows

Batch shortcuts (optional):
Direct PowerShell:

​
macOS and Linux

The same operations use bash (the bumblebee gateway CLI calls this for you):
Run from the repository root so .env and imports resolve the same way as the CLI.