Oracle has published a reference implementation that uses the open-source LiteLLM gateway to route each AI request to a suitable model on OCI Generative AI. The announcement came on Oracle's AI and Data Science blog on November 12, 2026. The code is public in a repository named oci-litellm-routing.
The idea is simple. A quick question does not need the model that a hard planning task needs, so a classifier picks the least expensive group likely to answer well.
How the Gateway Works
LiteLLM has built-in OCI Generative AI support. It exposes twenty models from OpenAI, Google, Meta, xAI and Cohere through a single OpenAI-style endpoint. An app requests a named group of models rather than a specific vendor model, and it carries a LiteLLM virtual key in place of any OCI credential.
The models are sorted into eight routes by job: quick answers, balanced, strong, reasoning, code, images, long context and other languages. Each route has a backup group and carries its price from the public OCI list.
Three layers choose the model, in sequence, before anything runs. Plain rules cover facts like images or very long inputs, the trained router weighs what the request needs, and LiteLLM then applies tags, budgets and fallbacks.
The Trained Router
Oracle trained the classifier on how answers were graded, not on opinions about prompt difficulty. Two models wrote 591 test requests, each route answered every one three times, and a judge scored all 3,590 replies.
For every route it estimates the odds of a good reply, then chooses the cheapest route that clears a quality threshold, 75% unless you change it. The model is tiny at about 220 KB, trains in under a second, and a typical decision takes 79 ms on a single Ampere core.
Oracle tested 73 requests kept out of training. The trained router produced 84% good answers for $0.306, while sending everything to reasoning models scored 89% for $0.376 and sending everything to fast models scored 77% for $0.043. Oracle admits the router can miss on individual requests.
Codex, Sandboxes and Governance
- GPT-5.3-codex: OCI offers it through the Responses API, a protocol the stock LiteLLM OCI provider lacks, so a small custom provider fills that hole.
- Codex as a client: one block in its config file makes LiteLLM its model provider. In Oracle's checks, a simple question cost about $0.001 and a data race fix about $0.07.
- Governance: spending caps and rate limits on each key, a premium lane chosen by tag, retries, load balancing by latency and a spend report per key.
- Isolation: a sample agent inside an NVIDIA OpenShell sandbox can talk only to the gateway and stores no OCI credential.
What It Means for Developers
Teams running several models often overspend by sending everything to the strongest one. A gateway with per-request routing and per-key budgets gives platform teams one place to control cost and access. Each response also reports the route, model and cost in headers, which helps with showback.
Treat the numbers as one vendor test on a limited set of prompts. Retrain on your own requests and your own judge, and keep an evaluation that shows whether routing still pays. Related news for OCI AI developers is in our report on the OCI Atlas plugin for ChatGPT and Codex.
How to Get Started
Clone the oci-litellm-routing repository and write a small deployment file with your OCI profile and regions. You can run five scripts or ask Codex to build it from a bundled specification. The result is a Kubernetes cluster on OKE, one virtual machine and an OCI API Gateway inside your tenancy.
Oracle admits it has not yet run a first build of the final scripts in a blank tenancy. It also advises checking model IDs, prices and regional availability beforehand.
Conclusion
Oracle's reference design shows routing by outcome on OCI Generative AI, with open-source parts. Test it with your own traffic before trusting the savings.
Sources
See my articles more often when you search Google.
