Skip to content

CONTRIBUTING.md

Latest commit

 

History

History
356 lines (265 loc) · 14.5 KB

CONTRIBUTING.md

File metadata and controls

356 lines (265 loc) · 14.5 KB
Raw

Contributing to TrueFoundry Models Registry

Thank you for your interest in contributing to the TrueFoundry Models Registry! This document provides guidelines and instructions for contributing.

Table of Contents

What Belongs Here

This registry holds metadata for providers the TrueFoundry LLM Gateway already integrates with. The directories under providers/ are that list, and the Supported Providers table in the README renders it.

In scope:

  • Adding a model under an existing provider
  • Correcting pricing, limits, features, modalities, or sources
  • Marking a model deprecated or retired

Out of scope:

  • A new providers/<name>/ directory. The gateway can only route to providers it has an integration for, so registry YAML for an unsupported provider is inert. See Requesting a New Provider.
  • Gateway routing, authentication, or request/response transformation. None of that is configured from this repo.
  • Edits to the README's Supported Providers table. It's generated from providers/ by update-readme-counts.ts.

PRs that add an unsupported provider will be closed with a pointer to this section. It isn't a judgement on the provider, there's just nothing on the gateway side for the files to attach to yet.

Code of Conduct

By participating in this project, you agree to abide by our Code of Conduct. Please read it before contributing.

How to Contribute

Reporting Issues

  • Check if the issue already exists in the Issues section
  • If not, create a new issue with a clear title and description
  • Include relevant details like model name, provider, and what needs to be updated

Submitting Changes

  1. Clone the repository (git clone https://github.com/truefoundry/models.git)
  2. Create a new branch (git checkout -b feature/add-new-model)
  3. Make your changes
  4. Commit your changes (git commit -m 'Add new model: provider/model-name')
  5. Push to the branch (git push origin feature/add-new-model)
  6. Open a Pull Request

Adding a New Model

To add a new model, create a YAML file in the appropriate provider directory:

1. File Location

providers/<provider-name>/<model-name>.yaml

<provider-name> must be a directory that already exists. If it doesn't, see Requesting a New Provider rather than creating one.

2. Required Fields

model: <model-identifier>   # The model identifier used by the provider's API
mode: <mode>                 # The model's primary capability (see values below)

Valid mode values:

audio_transcription, audio_translation, chat, completion, decision, embedding, image, moderation, realtime, rerank, text_to_speech, unknown, video

canonical_model and external_ids are optional. When you know the foundation model, set canonical_model on the YAML. Add external_ids only for a verified Artificial Analysis match (requires canonical_model).

When you add or change external_ids, CI verifies the Artificial Analysis id/slug pair against the live catalog for files changed in the PR when repository secret AA_API_KEY is configured. Without the secret, CI skips the live check with a warning.

3. Optional Fields

# Optional stable foundation-model identity: <creator>/<foundation-model> (lowercase).
# Use the model creator, not the host (anthropic, not bedrock/vertex/azure/openrouter).
# Provider deployments of the same weights share one canonical name.
canonical_model: openai/gpt-5-4-mini

# Optional verified Artificial Analysis identifiers. Omit when unmatched.
# Never invent, null, or guess values. Requires canonical_model. All files in a
# canonical group must share the same external_ids (or all omit them).
external_ids:
    artificial_analysis_slug: gpt-5-4-mini
    artificial_analysis_id: 00000000-0000-0000-0000-000000000000

# Pricing — an array of cost entries, each with a region ("*" for global pricing)
costs:
    - region: "*"
      input_cost_per_token: <number>
      output_cost_per_token: <number>
      input_cost_per_token_batches: <number>
      output_cost_per_token_batches: <number>
      cache_read_input_token_cost: <number>
      # See the CUE schema for all supported cost fields (audio, image, video, etc.)

# Token and context window limits
limits:
    context_window: <number>
    max_input_tokens: <number>
    max_output_tokens: <number>
    max_tokens: <number>
    max_query_tokens: <number>              # For embedding/rerank models
    output_vector_size: <number>            # For embedding models
    tool_use_system_prompt_tokens: <number>

# Supported features/capabilities
features:
    - <feature>
# Valid features: assistant_prefill, cache_control, code_execution,
#   function_calling, parallel_function_calling, prompt_caching,
#   structured_output, system_messages, tool_choice, tools

# Input/output modalities
modalities:
    input: [text, image, audio, video, code, doc, pdf, embedding]
    output: [text, image, audio, video, code, doc, pdf, embedding]

# Whether the model supports extended thinking / reasoning
thinking: <boolean>

# Additional modes this model can be accessed through
supportedModes: [<mode>, ...]

# Lifecycle status
status: <active|preview|deprecated|retired>

# Deprecation date (YYYY-MM-DD format)
deprecationDate: <date-string>

# Whether the model is deprecated (prefer using status instead)
isDeprecated: <boolean>

# Documentation or pricing source URLs
sources:
    - <url>

# Message roles accepted by this model
messages:
    options: [system, user, assistant, developer]

# Parameter overrides relative to the provider default
params:
    - key: <param-key>
      defaultValue: <value>
      minValue: <number>
      maxValue: <number>
      type: <number|string|boolean|json|array-of-strings>

# Param keys to remove from the provider default
removeParams: [<param-key>, ...]

# Param keys that must always be provided by callers
requiredParams: [<param-key>, ...]

# Whether the model supports extended thinking / reasoning
thinking: true

# When true, inbound Chat Completions stay on /chat/completions unless a
# responses-only model or an existing trigger (header / reasoning / Cursor).
# Omit on new models so Chat Completions is always served via Responses.
preferChatCompletions: true

# Provider-specific escape-hatch configuration. Only set a key here when the
# behaviour cannot be expressed through the generic fields above, and only on
# models belonging to the provider that owns the key.
extra_configuration:
    # aws-bedrock-mantle only. The path segment AWS serves the model under,
    # injected before /v1/... Copy it from the bedrock-mantle row of the model
    # card's Programmatic Access table, reading it from the In-Region endpoint
    # URL: "/openai" or "/anthropic". Omit the key entirely when the model is
    # served on the plain https://bedrock-mantle.{region}.api.aws/v1 path.
    bedrock_mantle_adhoc_prefix: /openai

4. Examples

A chat model with some details:

model: gpt-5.4-mini-2026-03-17
mode: chat
costs:
    - region: "*"
      input_cost_per_token: 7.5e-7
      output_cost_per_token: 0.0000045
      cache_read_input_token_cost: 7.5e-8
      input_cost_per_token_batches: 3.75e-7
      output_cost_per_token_batches: 0.00000225
features:
    - function_calling
    - parallel_function_calling
    - tool_choice
    - prompt_caching
    - structured_output
    - system_messages
limits:
    context_window: 400000
    max_output_tokens: 128000
    max_tokens: 128000
modalities:
    input:
        - text
        - image
    output:
        - text
thinking: true
params:
    - key: max_tokens
      defaultValue: 128
      maxValue: 128000
      minValue: 1
    - key: reasoning_effort
      defaultValue: medium
sources:
    - https://developers.openai.com/api/docs/pricing
    - https://developers.openai.com/api/docs/deprecations

A minimal model definition (e.g., audio transcription or TTS):

model: nova-3-general
mode: audio_transcription

Requesting a New Provider

Support for a provider that isn't in the Supported Providers table starts in the gateway, not in this repo. Open an issue with:

  • The provider and a link to its API reference
  • Which endpoints you need (chat, embeddings, transcription, text-to-speech, images, rerank)
  • Whether the API is OpenAI-compatible, and on which routes
  • The use case and rough volume

Note that OpenAI compatibility on /chat/completions doesn't cover everything. Endpoints outside chat, embeddings, and responses need explicit work in the gateway even when the provider mirrors OpenAI's shape.

Once the integration lands, the provider directory gets created and model YAML is welcome.

Updating an Existing Model

When updating a model:

  1. Find the model file in providers/<provider>/ (note: file names may not exactly match the model field)
  2. Update the relevant fields
  3. Add sources with links to the official documentation/pricing pages
  4. Submit a PR with a clear description of what changed and why

Common updates include:

  • Pricing changes
  • New features
  • Token limit updates
  • Deprecation notices

Pull Request Process

  1. Title: Use a clear, descriptive title
  • Add: provider/model-name
  • Update: provider/model-name pricing
  • Fix: provider/model-name features
  1. Description: Include
  • What changes were made
  • Source/reference for the changes (official docs, pricing pages)
  • Any breaking changes
  1. Review:
  • All PRs require at least one review
  • Maintainers may request changes or clarifications
  1. Merge:
  • PRs are merged once approved
  • Squash merge is preferred for clean history

SDK live model tests

Same-repo PRs that change providers/**/*.yaml run SDK live checks (.github/workflows/sdk-live-test.yml):

  1. Build dist/ai-models.json from the PR head
  2. For each changed model YAML, run scenarios through @truefoundry/ai-gateway-sdk against the real provider
  3. Fail the check on any scenario failure (blocks merge)

Coverage (SDK only)

Mode What runs
chat baseline + declared features (structured_output, json_output, function_calling, parallel_function_calling, tool_choice, system_messages, assistant_prefill, mid_conversation_system_messages, prompt_caching / cache_control) + reasoning when thinking; each × stream and non-stream; params (defaults for max_tokens, max_completion_tokens, temperature, top_p, top_k, seed, verbosity, n, stop, min_tokens, reasoning_effort) × stream; if modalities.input includes image / pdf / doc / audio, non-stream smokes with fixtures under .github/scripts/sdk-live-test/fixtures/
completion baseline (+ params) × stream / non-stream
responses baseline + mappable features (structured_output, json_output, tools / tool_choice, reasoning) × stream / non-stream; params; vision / pdf / doc / audio smokes
embedding baseline embed; multimodal image / pdf when declared
rerank baseline rerank
text_to_speech baseline speech
audio_transcription / audio_translation baseline with a tiny WAV via FormData
image baseline imageGenerate
moderation baseline createModeration
decision baseline decision (OpenAI dialect for openai; System One for others)
realtime / video / ocr / unknown / unsupported fail (no SDK live harness yet)

Pass bar is HTTP 2xx only (no deep response-shape asserts). Each model’s primary mode plus any harnessed entries in supportedModes are planned (e.g. chat+responses). Not covered: realtime / video / ocr / unknown / unsupported, Azure OpenAI / Azure AI Foundry, code_execution, stream × vision/PDF, gateway adapter SDKs (boto3, etc.). Vertex uses SA JSON (VERTEX_KEY_FILE), not an API key. Bedrock and Bedrock Mantle use API keys with region us-east-1. Databricks also needs DATABRICKS_BASE_URL. Microsoft Foundry also needs MICROSOFT_FOUNDRY_OPENAI_ENDPOINT and MICROSOFT_FOUNDRY_PROJECT_ENDPOINT.

Manual re-run: push a new commit, or Actions → SDK live model tests → Run workflow (optional models= override and scenarios=baseline|all; default is baseline for smoke, all for explicit models).

Secrets smoke (one chat baseline per provider with a key): Actions → SDK live model tests → Run workflow (leave models empty).

Required GitHub secrets (provider credentials)

Each changed model’s provider must be registered in .github/scripts/sdk-live-test/src/providers.ts and have its credential secret set. Missing registry entry or secret fails the PR check:

OPENAI_API_KEY, ANTHROPIC_API_KEY, GROQ_API_KEY, XAI_API_KEY, CEREBRAS_API_KEY, TOGETHER_AI_API_KEY, DEEPINFRA_API_KEY, MISTRAL_AI_API_KEY, PERPLEXITY_AI_API_KEY, OPENROUTER_API_KEY, SAMBANOVA_API_KEY, AI21_API_KEY, COHERE_API_KEY, GOOGLE_GEMINI_API_KEY, VERTEX_KEY_FILE (GCP service-account JSON), AWS_BEDROCK_API_KEY, AWS_BEDROCK_MANTLE_API_KEY, DATABRICKS_API_KEY, DATABRICKS_BASE_URL, MICROSOFT_FOUNDRY_API_KEY, MICROSOFT_FOUNDRY_OPENAI_ENDPOINT, MICROSOFT_FOUNDRY_PROJECT_ENDPOINT

Local:

npm run build:json
# omits --catalogue → default repo-root dist/ai-models.json
npm run test:sdk-live -- --model openai/gpt-4o

This path does not use TFY_API_KEY / Tailscale / gateway-test-job-v2. Merge → S3 publish (trigger-sync.yml) is unchanged.

Style Guide

YAML Formatting

Run npm run lint:yaml:fix to auto-format your YAML files before submitting.

Commit Messages

  • Use present tense ("Add model" not "Added model")
  • Use imperative mood ("Update pricing" not "Updates pricing")
  • Keep the first line under 72 characters
  • Reference issues when applicable

Versioning for Dated Models

For models with version dates:

  • Use the date format from the provider (e.g., gpt-4-0613.yaml)

Questions?

If you have questions about contributing, feel free to:

  • Open an issue with the question label
  • Reach out to the maintainers

Thank you for contributing!