RAG pipelines are supposed to be the killer app for LLMs. But the dirty secret is that most RAG implementations are wildly over-provisioned — you are running every retrieval-augmented query through the same expensive model, whether the user asked "what is our refund policy" (a simple lookup) or "analyze the trend in Q3 support tickets and suggest process changes" (genuine synthesis). The simple lookups are 80% of your traffic, and they are all burning GPT-4o tokens for no reason.
Antbase fixes this at the infrastructure level. When you plug it into LangChain as the LLM backend, every chain invocation — every retrieval call, every synthesis step, every agent tool-parse — gets independently classified and routed. Simple document lookups go to fast, free models. Complex multi-document synthesis goes to premium models. Your RetrievalQA chain works exactly the same way, but your bill drops dramatically because 80% of your queries are now hitting models that cost nothing.
This matters even more for LangChain agents, which make dozens of LLM calls per task. Each tool-parse call, each reasoning step, each output format check is a separate API call. Without intelligent routing, every single one of those calls hits your most expensive model. With Antbase, the simple parsing calls go to fast models and only the genuine reasoning steps get premium treatment. The result: agents that cost 70% less to run with no degradation in output quality.
LangChain is the most popular framework for building LLM applications. Since it uses the OpenAI client under the hood, you can swap in Antbase as the backend and get intelligent model routing for every chain invocation — RAG, agents, summarization, and more.
Installation
pip install langchain langchain-openai chromadbConfigure the LLM
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
base_url="https://antbase.ai/v1",
api_key="ant_your-api-key",
model="auto", # let Antbase route
)
# Test it
response = llm.invoke("What is RAG?")
print(response.content)Build a RAG Chain
Load documents, split them, embed them into a vector store, and query with context. Antbase handles the LLM calls — routing simple retrievals to fast models and complex synthesis to premium ones.
from langchain_community.document_loaders import TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings
from langchain.chains import RetrievalQA
# Load and split documents
loader = TextLoader("docs/knowledge-base.txt")
docs = loader.load()
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
chunks = splitter.split_documents(docs)
# Create vector store (use OpenAI embeddings or any provider)
embeddings = OpenAIEmbeddings(
base_url="https://antbase.ai/v1",
api_key="ant_your-api-key",
)
vectorstore = Chroma.from_documents(chunks, embeddings)
# Build the chain
qa = RetrievalQA.from_chain_type(
llm=llm,
retriever=vectorstore.as_retriever(search_kwargs={"k": 4}),
)
result = qa.invoke("How does the billing system work?")
print(result["result"])Streaming with Callbacks
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
llm_stream = ChatOpenAI(
base_url="https://antbase.ai/v1",
api_key="ant_your-api-key",
model="auto",
streaming=True,
callbacks=[StreamingStdOutCallbackHandler()],
)
# Streams tokens to stdout as they arrive
llm_stream.invoke("Explain vector databases in 3 paragraphs.")Tips
- •LangChain caches LLM calls by default. This works fine with Antbase — cached responses skip the routing entirely.
- •For agents that make many tool calls, Antbase routes each call independently. Simple tool-parse calls go to fast models; complex reasoning calls escalate.
- •Set temperature in the ChatOpenAI constructor as usual. Antbase passes it through to whichever model handles the request.
- •If you need a specific model for a chain step (e.g., GPT-4o for final synthesis), create a second ChatOpenAI instance with model='gpt-4o'.
