LlamaIndex Integration

Name: ANT
Author: Antbase

Use Antbase as the LLM backend for LlamaIndex RAG pipelines, document Q&A, and knowledge graph applications. Antbase handles model selection and failover while LlamaIndex manages your indexing and retrieval logic.

What this enables

RAG pipelines with intelligent model routing per query complexity
Document Q&A, summarization, and extraction over your own data
Multi-provider failover so your RAG pipeline never breaks
Use ant:fast for retrieval scoring and ant:best for synthesis

Quick Start

1. Install dependencies

pip install llama-index llama-index-llms-openai-like

2. Configure the LLM

from llama_index.llms.openai_like import OpenAILike

llm = OpenAILike(
    api_base="https://antbase.ai/v1",
    api_key="your-ant-api-key",
    model="ant:auto",
    is_chat_model=True,
)

3. Build a RAG pipeline

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.core import Settings

# Set Antbase as the default LLM
Settings.llm = llm

# Load and index documents
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)

# Query
query_engine = index.as_query_engine()
response = query_engine.query("What are the main findings in the report?")
print(response)

Configuration

You can also configure via environment variables:

# .env
OPENAI_API_BASE=https://antbase.ai/v1
OPENAI_API_KEY=your-ant-api-key

from llama_index.llms.openai import OpenAI

# When OPENAI_API_BASE is set, the standard OpenAI LLM works too
llm = OpenAI(model="ant:auto")

Usage Examples

Document summarization

from llama_index.core import SummaryIndex, SimpleDirectoryReader

Settings.llm = OpenAILike(
    api_base="https://antbase.ai/v1",
    api_key="your-ant-api-key",
    model="ant:best",
    is_chat_model=True,
)

documents = SimpleDirectoryReader("./reports").load_data()
index = SummaryIndex.from_documents(documents)

query_engine = index.as_query_engine()
summary = query_engine.query("Provide a comprehensive summary of all documents.")
print(summary)

Chat engine with memory

index = VectorStoreIndex.from_documents(documents)
chat_engine = index.as_chat_engine(chat_mode="condense_plus_context")

response = chat_engine.chat("What is the revenue for Q4?")
print(response)

# Follow-up (uses conversation history)
response = chat_engine.chat("How does that compare to Q3?")
print(response)

Streaming responses

query_engine = index.as_query_engine(streaming=True)
response = query_engine.query("Explain the methodology used.")

for token in response.response_gen:
    print(token, end="", flush=True)

Virtual Models

Model	Best for	Notes
ant:auto	General use	Classifies prompt and picks optimal model
ant:fast	Retrieval scoring, simple Q&A	Lowest latency for high-volume RAG queries
ant:best	Synthesis, complex analysis	Frontier models for detailed document understanding
ant:free	Development, testing	Zero cost; free-tier provider endpoints
ant:reasoning	Multi-step logic	Chain-of-thought for complex reasoning over documents
ant:coder	Code generation	Code-optimized models for technical docs

See Virtual Models for the full list.

Troubleshooting

Model not found errors

Make sure you are using OpenAILike from llama-index-llms-openai-like and that is_chat_model=True is set.

Slow first query

The first query builds the vector index. For large document sets, consider persisting your index with index.storage_context.persist().