Skip to main content

AI Models

TestPlanIt integrates with leading AI providers to power features across the platform, including test case generation, intelligent test case selection, in-editor writing assistance, and AI-assisted imports.

Overview​

AI-powered features in TestPlanIt:

  • Test Case Generation — Generate test cases from requirements, issues, and documents
  • Magic Select — AI-assisted test case selection when building test runs
  • Writing Assistant — Improve, translate, and enhance content in any rich text field
  • Markdown Import — AI-assisted field mapping when importing markdown test cases
  • QuickScript — AI-assisted automation script generation from manual test cases

Supported AI Providers​

OpenAI​

  • Models: GPT-4, GPT-4 Turbo, GPT-3.5 Turbo
  • Authentication: API Key
  • Strengths: Excellent natural language understanding, reliable structured output

Google Gemini​

  • Models: Gemini Pro, Gemini Pro Vision
  • Authentication: API Key
  • Strengths: Strong reasoning capabilities, cost-effective

Anthropic Claude​

  • Models: Claude 3 Opus, Claude 3 Sonnet, Claude 3 Haiku
  • Authentication: API Key
  • Strengths: Excellent instruction following, safety-focused

Ollama (Self-Hosted)​

  • Models: Llama 2, Code Llama, Mistral, and other open-source models
  • Authentication: None (local deployment)
  • Strengths: Privacy, no API costs, customizable

Azure OpenAI​

  • Models: GPT-4, GPT-3.5 Turbo (deployed on Azure)
  • Authentication: API Key + Deployment Name
  • Strengths: Enterprise features, data residency, SLA guarantees

Custom LLM​

  • Models: Any OpenAI-compatible API endpoint
  • Authentication: Configurable (API Key)
  • Strengths: Maximum flexibility, support for custom models

System Configuration​

Administrator Setup​

  1. Navigate to Administration → LLM Integrations
  2. Click Add LLM Integration
  3. Configure your preferred AI provider:
Name: "Production OpenAI"
Provider: OPENAI
Model: gpt-4-turbo-preview
Status: ACTIVE

OpenAI Configuration​

API Key: sk-...your-openai-api-key
Model: gpt-4-turbo-preview
Max Tokens: 4096
Temperature: 0.7

Google Gemini Configuration​

API Key: your-gemini-api-key
Model: gemini-pro
Max Tokens: 8192
Temperature: 0.7

Anthropic Claude Configuration​

API Key: your-anthropic-api-key
Model: claude-3-sonnet-20240229
Max Tokens: 4096
Temperature: 0.7

Ollama Configuration​

Base URL: https://your-ollama-server.example.com:11434
Model: llama2:13b
Max Tokens: 4096
Temperature: 0.7

Azure OpenAI Configuration​

API Key: your-azure-openai-key
Endpoint: https://your-resource.openai.azure.com/
Deployment Name: gpt-4-deployment
API Version: 2024-02-15-preview
Max Tokens: 4096
Temperature: 0.7

Custom LLM Configuration​

Base URL: https://your-custom-endpoint.com/v1
API Key: your-custom-api-key
Model: your-model-name
Max Tokens: 4096
Temperature: 0.7

Note: Custom LLM endpoints must be compatible with the OpenAI API format.

Endpoint URL Requirements​

The Endpoint URL field is optional for every provider:

  • Leave it blank to use each provider's official URL (OpenAI, Anthropic, Gemini, Azure OpenAI defaults).
  • Set it to an OpenAI-compatible proxy (for example, LiteLLM) to route any provider through your own infrastructure. This works for OPENAI, ANTHROPIC, AZURE_OPENAI, GEMINI, OLLAMA, and CUSTOM_LLM.
  • Required for OLLAMA and CUSTOM_LLM, since they have no public default.

For security, all custom URLs are validated to prevent Server-Side Request Forgery (SSRF) attacks. The following are blocked:

  • localhost, 127.0.0.1, 0.0.0.0, IPv6 loopback (::1)
  • Private IP ranges: 10.x.x.x, 172.16–31.x.x, 192.168.x.x
  • Cloud metadata endpoints: 169.254.169.254, metadata.google.internal, *.internal

Recommended: Expose self-hosted services (Ollama, LiteLLM, etc.) through a reverse proxy with proper authentication and a publicly accessible URL. This preserves SSRF protection while allowing TestPlanIt to reach your internal services securely.

As a convenience alternative, you can set the ALLOWED_PRIVATE_HOSTS environment variable to a comma-separated list of trusted hostnames. This bypasses SSRF protection for the listed addresses and applies to LLM provider endpoints, code repository connections, and URL-based test case generation.

# Allow localhost and a private IP for self-hosted services
ALLOWED_PRIVATE_HOSTS="localhost,192.168.1.100,ollama.internal"
warning

Only add hosts that you trust. ALLOWED_PRIVATE_HOSTS disables SSRF protection for every listed address across all features. A reverse proxy is the safer option for production environments.

Billing Period​

Each LLM integration tracks usage against an optional Monthly Budget (USD). By default, the spend window aligns with the calendar month — costs reset to zero on the 1st.

If your provider invoices on a different cycle, set the Billing Period Start Day (1–31) on the Edit dialog to match. For example:

  • 1 — calendar month (default)
  • 15 — 15th-to-15th cycle (e.g., to match a provider invoice date)
  • 29, 30, 31 — automatically clamp to the last day of shorter months (e.g., February in non-leap years)

The setting affects three behaviors:

  • Spend aggregation — the cost shown on the admin LLM page totals usage from the period start through today
  • Budget alerts — 80% / 90% / 100% threshold notifications reset on each new period
  • Reset Spend — the manual reset action on the Edit dialog clears usage from the current period start onward (older periods remain in the audit trail)

Saving an integration with a changed billingPeriodStartDay clears any thresholds already fired so notifications can fire again under the new window.

Test Connection & Model Capability Probing​

Both the Test Connection button and the Update / Create action probe the configured model for parameter support before persisting. This avoids paying for a failed-and-retried request the first time an AI feature actually fires.

What gets probed. TestPlanIt sends a 1-token request with temperature: 1 to the configured model and watches for parameter-rejection errors. Today only the temperature parameter is probed — newer Anthropic adaptive-thinking models (e.g., Claude Opus 4.7) return a deprecation error rather than ignoring the field. The probe code is structured so additional parameters can be added later without changing the storage shape.

What gets stored. The result lands in LlmProviderConfig.settings.modelCapabilities keyed by model id:

{
  "modelCapabilities": {
    "claude-opus-4-7": {
      "unsupportedParams": ["temperature"],
      "probedAt": "2026-04-30T18:21:49.088Z"
    }
  }
}

unsupportedParams: [] means the model accepted everything — the probe still records this so the admin can see when the integration was last verified.

When the probe runs.

  • Clicking Test Connection runs a probe and captures the result in the form. A success toast confirms the connection.
  • Clicking Update / Create runs the probe automatically if no successful test has been performed in this session, or if any credential-affecting field (provider, API key, endpoint, deployment name, default model) has been edited since the last test. A failed probe at this stage aborts the save and surfaces the same error toast as Test Connection — the admin can fix and retry without losing form state.

Runtime fallback. Each adapter still carries an in-memory cache that catches and remembers parameter-rejection errors at chat-request time. If a model is updated by the provider after an integration is configured, the first chat request will hit the new error, retry without the rejected param, and remember it for the rest of the process — giving you correct behavior until the next Test Connection re-probes and persists the change.

Project Assignment​

After creating an LLM integration:

  1. Go to Project Settings → AI Models
  2. Select the integration from available options
  3. Optionally assign a Prompt Configuration to customize how AI prompts behave for this project
  4. Optionally override the LLM integration for individual AI features (see Per-Feature LLM Overrides below)
  5. Save settings

Per-Feature LLM Overrides​

Project admins can override which LLM integration is used for specific AI features, independently of the project-wide default. This is configured per project on the AI Models settings page and applies to the following features: Test Case Generation, Markdown Parsing, Smart Test Case Selection, Editor Writing Assistant, LLM Connection Test, QuickScript, and Auto-tagging.

How to Configure​

  1. Navigate to Project Settings → AI Models
  2. Find the Per-Feature LLM Overrides card, which shows a table with four columns:
    • Feature — The AI feature name
    • Override — A dropdown to select a specific LLM integration for this feature. An X button appears next to the dropdown when an override is active; click it to clear the override.
    • Effective LLM — The LLM integration that will actually be used when this feature is invoked
    • Source — A color-coded badge showing where the effective LLM comes from (see below)
  3. For any feature row, select an LLM integration from the Override dropdown to assign that integration for this feature in this project
  4. To explicitly disable a feature for this project, select No LLM (disabled) from the dropdown — the feature will be unavailable regardless of project default or prompt config settings
  5. To remove an override, click the X button next to the dropdown — the feature will fall back to the next level in the resolution chain

Source Badges​

BadgeColorMeaning
Project OverrideBlueA per-feature override is set on this project
Disabled (Override)OrangeThe feature is explicitly disabled for this project via override
Prompt ConfigGrayThe prompt configuration has a per-prompt LLM assignment
Project DefaultOutlineThe project's default LLM integration is being used
No LLM configuredRedNo integration is available at any level

Use Cases​

  • Use a fast, inexpensive model (e.g., GPT-3.5) for connection tests and a more capable model (e.g., GPT-4) for test case generation
  • Override a single feature to use a different AI provider without changing the configuration for other features
  • Allow project teams to customize their LLM usage independently of the system-wide prompt configuration

LLM Resolution Chain​

When an AI feature is invoked, TestPlanIt resolves which LLM integration to use by checking three levels in order of priority:

  1. Project Feature Override (highest priority) — If the project has a per-feature LLM override configured for this specific feature (via the Per-Feature LLM Overrides table), that integration is used.
  2. Prompt Configuration Assignment — If the resolved prompt configuration has a per-prompt LLM integration assigned to this feature, that integration is used (along with the model override, if one is set).
  3. Project Default Integration (lowest priority) — The project's default LLM integration is used as the final fallback.
Feature invoked
  |
  v
Project Feature Override set? --> YES (integration) --> Use override integration
  |                           \-> YES (disabled)    --> Feature unavailable
  NO
  v
Prompt has per-prompt LLM assigned? --> YES --> Use prompt's integration (+ model override if set)
  |
  NO
  v
Use project default integration

This resolution chain applies independently per AI feature. Different features in the same project can resolve to different LLM integrations depending on what is configured at each level.

Per-prompt LLM assignments are configured in the admin prompt editor. See Prompt Configurations — Per-Prompt LLM Assignment for details.

Security Considerations​

Data Privacy​

  • API Requests: Source material is sent to AI providers for processing
  • Retention: Most providers don't retain request data (verify with your provider)
  • Sensitive Data: Avoid including sensitive information in source material
  • Self-Hosted Options: Consider Ollama for maximum data privacy

Access Control​

  • Permission Model: Same as regular test case creation
  • Audit Logging: All AI generation activities are logged
  • Rate Limiting: Built-in rate limiting prevents abuse

Migration and Updates​

Upgrading AI Providers​

  1. Create new integration with updated settings
  2. Test generation quality with new provider
  3. Update project assignments
  4. Archive old integration when satisfied

Model Updates​

  • New models are automatically available when providers release them
  • Update model names in integration settings
  • Test generation quality with new models before switching

Monitoring and Analytics​

Usage Metrics​

Track important metrics in the admin dashboard:

  • Generation Volume: Number of test cases generated per period
  • Success Rate: Percentage of successful generations
  • User Adoption: Which teams are using AI generation
  • Cost Tracking: API usage and associated costs

Quality Metrics​

  • Review Rate: Percentage of generated cases that are reviewed before import
  • Acceptance Rate: Percentage of generated cases that are imported
  • Modification Rate: How often generated cases are edited post-import

Future Enhancements​

Planned improvements include:

  • Custom Model Fine-Tuning: Train models on your specific domain
  • Multi-Language Support: Generate test cases in different languages
  • Visual Test Generation: Generate test cases from UI mockups
  • Regression Analysis: Automatically update test cases when requirements change
  • Test Execution Integration: Connect generated cases to automation frameworks
  • Magic Select Improvements: Historical analysis of test run patterns for better suggestions