Skip to main content

LLM Model Configuration

LLM (Large Language Model) settings control the AI model used for RAG chat responses in the Playground.

Available Providers​

ProviderModel catalogNotes
OpenAIModels available to the configured OpenAI projectIncludes compatible fine-tuned chat models
AnthropicModels available to the configured Anthropic accountIncludes current context and output-token limits when provided

Configuration Options​

  1. Provider: Select OpenAI or Anthropic
  2. API Key: Enter your provider's API key
  3. Model: Choose the specific LLM model
  4. Temperature: Adjust creativity (0.0 = focused, 2.0 = creative)
  5. Max Tokens: Limit response length

API Key Management​

Each provider requires its own API key:

The settings page shows a status banner indicating whether your API key is configured:

  • Green checkmark: API key is configured and ready
  • Yellow warning: API key required - enter your key to enable chat

API-key format and provider authentication are checked before saving. If the provider's model service is temporarily unavailable, the key is saved with a warning so you can retry model discovery later. Keys and provider responses are handled on the server and are not returned to the browser.

Model Discovery​

  • After you configure a key, the page loads compatible chat models available to that provider account.
  • Use Refresh after an account or model-access change. Results are cached briefly to avoid unnecessary provider calls.
  • If live discovery is unavailable, the page clearly labels and displays a small built-in fallback catalog.
  • A saved model that is no longer available remains visible with an Unavailable label. Select an available replacement before saving.
  • OpenAI image, audio, realtime, embedding, moderation, deep-research, and other specialist models are excluded because RAG chat uses the Chat Completions contract.
  • Controls adapt to model metadata. Temperature is disabled for model families that manage sampling internally, and response tokens are capped at the provider limit when one is reported.
  • OpenAI does not report token limits in its Models API. Known model families use maintained limits; unknown compatible models use a conservative response limit until their metadata is added.

Temperature Guide​

TemperatureBehaviorBest For
0.0 - 0.3Focused, deterministicFactual Q&A, technical docs
0.4 - 0.7BalancedGeneral use (default: 0.7)
0.8 - 1.2Creative, variedContent generation
1.3 - 2.0Highly creativeBrainstorming, exploration