Skip to main content

AI Models

TestPlanIt integrates with leading AI providers to power features across the platform, including test case generation, intelligent test case selection, in-editor writing assistance, and AI-assisted imports.

Overview​

AI-powered features in TestPlanIt:

  • Test Case Generation — Generate test cases from requirements, issues, and documents
  • Magic Select — AI-assisted test case selection when building test runs
  • Writing Assistant — Improve, translate, and enhance content in any rich text field
  • Markdown Import — AI-assisted field mapping when importing markdown test cases
  • QuickScript — AI-assisted automation script generation from manual test cases
  • Automation Candidates — AI-ranked recommendation of which manual test cases to automate next, with a heuristic fallback when no LLM is configured

Supported AI Providers​

OpenAI​

  • Models: GPT-4, GPT-4 Turbo, GPT-3.5 Turbo
  • Authentication: API Key
  • Strengths: Excellent natural language understanding, reliable structured output

Google Gemini​

  • Models: Gemini Pro, Gemini Pro Vision
  • Authentication: API Key
  • Strengths: Strong reasoning capabilities, cost-effective

Anthropic Claude​

  • Models: Claude 3 Opus, Claude 3 Sonnet, Claude 3 Haiku
  • Authentication: API Key
  • Strengths: Excellent instruction following, safety-focused

DeepSeek​

  • Models: DeepSeek V4 Flash, DeepSeek V4 Pro, and the experimental DeepSeek V4 Flash Vision
  • Authentication: API Key
  • Strengths: Low cost, strong reasoning (thinking mode is on by default)

Ollama (Self-Hosted)​

  • Models: Llama 2, Code Llama, Mistral, and other open-source models
  • Authentication: None (local deployment)
  • Strengths: Privacy, no API costs, customizable

Azure OpenAI​

  • Models: GPT-4, GPT-3.5 Turbo (deployed on Azure)
  • Authentication: API Key + Deployment Name
  • Strengths: Enterprise features, data residency, SLA guarantees

Custom LLM​

  • Models: Any OpenAI-compatible API endpoint
  • Authentication: Configurable (API Key)
  • Strengths: Maximum flexibility, support for custom models

System Configuration​

Administrator Setup​

  1. Navigate to Administration → LLM Integrations
  2. Click Add LLM Integration
  3. Configure your preferred AI provider:
Name: "Production OpenAI"
Provider: OPENAI
Model: gpt-4-turbo-preview
Status: ACTIVE

OpenAI Configuration​

API Key: sk-...your-openai-api-key
Model: gpt-4-turbo-preview
Max Tokens: 4096
Temperature: 0.7

Google Gemini Configuration​

API Key: your-gemini-api-key
Model: gemini-pro
Max Tokens: 8192
Temperature: 0.7

Anthropic Claude Configuration​

API Key: your-anthropic-api-key
Model: claude-3-sonnet-20240229
Max Tokens: 4096
Temperature: 0.7

DeepSeek Configuration​

API Key: your-deepseek-api-key
Model: deepseek-v4-flash
Max Tokens: 65536
Temperature: 0.7

DeepSeek's V4 models reason before answering (thinking mode) by default. The reasoning is returned separately from the answer and is billed as output tokens, so keep Max Tokens Per Request generous enough to leave room for the answer. Only deepseek-v4-flash-vision-exp accepts image input; the other models receive images as text placeholders. Leave Endpoint URL blank to use https://api.deepseek.com.

Ollama Configuration​

Base URL: https://your-ollama-server.example.com:11434
Model: llama2:13b
Max Tokens: 4096
Temperature: 0.7

Azure OpenAI Configuration​

API Key: your-azure-openai-key
Endpoint: https://your-resource.openai.azure.com/
Deployment Name: gpt-4-deployment
API Version: 2024-02-15-preview
Max Tokens: 4096
Temperature: 0.7

Custom LLM Configuration​

Base URL: https://your-custom-endpoint.com/v1
API Key: your-custom-api-key
Model: your-model-name
Max Tokens: 4096
Temperature: 0.7

Note: Custom LLM endpoints must be compatible with the OpenAI API format. When the endpoint is OpenAI-compatible, the model dropdown is populated automatically from its /models listing and model pricing is applied when available; endpoints that don't serve /models fall back to manual model entry.

Token Limits​

The Add/Edit AI model dialog exposes two separate output-token fields rather than a single generic limit:

  • Max Tokens Per Request — the hard ceiling on the number of output tokens the model may generate in one response. Set it to the model's real maximum output (for example, 128000 for Claude Opus/Sonnet, 64000 for Claude Haiku, 65536 for Gemini, 16384 for GPT-4o). It does not limit how much context you send.
  • Default Max Tokens — the output length requests on this model ask for (bounded by Max Tokens Per Request). Test Case Generation (every source, including the Jira issue panel), Markdown Test Case Parsing, the Editor Writing Assistant, and Smart Test Case Selection always use this value. Export Code Generation, AI Tag Suggestions, AI Step Derivation, and the Automation Candidates Report use the Max Output Tokens from their Prompt Configuration instead. Models that think before answering (for example, Claude Opus 5) spend part of this budget on reasoning, so leave room for the reply.

Neither field controls how much source material is sent to the model. For QuickScript, the amount of connected-repository code included with each generation is shown read-only as the Repository Context Budget (QuickScript) — auto-sized from the selected model's input context window, so larger-context models get more and small local models get a safe amount that won't overflow. There is nothing to configure here.

Endpoint URL Requirements​

The Endpoint URL field is optional for every provider:

  • Leave it blank to use each provider's official URL (OpenAI, Anthropic, Gemini, DeepSeek, Azure OpenAI defaults).
  • Set it to an OpenAI-compatible proxy (for example, LiteLLM) to route any provider through your own infrastructure. This works for OPENAI, ANTHROPIC, AZURE_OPENAI, GEMINI, DEEPSEEK, OLLAMA, and CUSTOM_LLM.
  • Required for OLLAMA and CUSTOM_LLM, since they have no public default.

For security, all custom URLs are validated to prevent Server-Side Request Forgery (SSRF) attacks. The following are blocked:

  • localhost, 127.0.0.1, 0.0.0.0, IPv6 loopback (::1)
  • Private IP ranges: 10.x.x.x, 172.16–31.x.x, 192.168.x.x
  • Cloud metadata endpoints: 169.254.169.254, metadata.google.internal, metadata.google, 100.100.100.200, *.internal

Recommended: Expose self-hosted services (Ollama, LiteLLM, etc.) through a reverse proxy with proper authentication and a publicly accessible URL. This preserves SSRF protection while allowing TestPlanIt to reach your internal services securely.

As a convenience alternative, you can set the ALLOWED_PRIVATE_HOSTS environment variable to a comma-separated list of trusted hostnames. This bypasses SSRF protection for the listed addresses and applies to LLM provider endpoints, code repository connections, and URL-based test case generation.

# Allow localhost and a private IP for self-hosted services
ALLOWED_PRIVATE_HOSTS="localhost,192.168.1.100,ollama.internal"
warning

Only add hosts that you trust. ALLOWED_PRIVATE_HOSTS disables SSRF protection for every listed address across all features. A reverse proxy is the safer option for production environments.

Model Pricing​

Each integration stores two cost fields — Cost Per 1M Input Tokens ($) and Cost Per 1M Output Tokens ($) — used to compute the estimated cost of every AI call (see the LLM Usage report) and to track spend against the Monthly Budget.

When the provider reports per-model pricing, both fields are filled in automatically:

  • LiteLLM proxies — pricing is read from the proxy's /model/info endpoint. This works whichever provider type the proxy is configured under (OpenAI, Anthropic, or Custom LLM).
  • OpenRouter and Together AI — pricing is included in their model listings.

Auto-fill happens when you select a model from the fetched model list, and again after a successful Test Connection — the latter is also how you refresh the values for the already-selected model after a provider price change. A toast confirms the applied rates, and you can still edit both fields before saving; nothing is stored until you click Create/Update. Models whose pricing the provider doesn't report leave the fields untouched.

Direct OpenAI, Anthropic, Gemini, DeepSeek, and Azure OpenAI APIs do not report pricing, so enter the values from your provider's pricing page. Locally hosted models (Ollama) cost nothing — leave both fields at 0.

Billing Period​

Each LLM integration tracks usage against an optional Monthly Budget (USD). By default, the spend window aligns with the calendar month — costs reset to zero on the 1st.

If your provider invoices on a different cycle, set the Billing Period Start Day (1–31) on the Edit dialog to match. For example:

  • 1 — calendar month (default)
  • 15 — 15th-to-15th cycle (e.g., to match a provider invoice date)
  • 29, 30, 31 — automatically clamp to the last day of shorter months (e.g., February in non-leap years)

The setting affects three behaviors:

  • Spend aggregation — the cost shown on the admin LLM page totals usage from the period start through today
  • Budget alerts — 80% / 90% / 100% threshold notifications reset on each new period
  • Reset Spend — the manual reset action on the Edit dialog clears usage from the current period start onward (older periods remain in the audit trail)

For a historical breakdown of spend by feature, model, user, or project — beyond the current billing period — use the LLM Usage report under Administration → Reports.

Saving an integration with a changed billingPeriodStartDay clears any thresholds already fired so notifications can fire again under the new window.

Test Connection & Model Capability Probing​

Both the Test Connection button and the Update / Create action probe the configured model for parameter support before persisting. This avoids paying for a failed-and-retried request the first time an AI feature actually fires.

What gets probed. TestPlanIt sends a 1-token request with a non-default temperature to the configured model and watches for parameter-rejection errors. Today only the temperature parameter is probed — newer Anthropic adaptive-thinking models (Claude Opus 4.7 and later) return a deprecation error rather than ignoring the field, and a LiteLLM proxy in front of such a model rejects any temperature other than 1. The probe runs for Anthropic, OpenAI, Azure OpenAI, DeepSeek, and Custom LLM integrations, so it covers a proxy configured under any of those provider types. The probe code is structured so additional parameters can be added later without changing the storage shape.

What gets stored. The result lands in LlmProviderConfig.settings.modelCapabilities keyed by model id:

{
  "modelCapabilities": {
    "claude-opus-4-7": {
      "unsupportedParams": ["temperature"],
      "probedAt": "2026-04-30T18:21:49.088Z"
    }
  }
}

unsupportedParams: [] means the model accepted everything — the probe still records this so the admin can see when the integration was last verified.

Vision support override. Whether a model can receive images (used by test-case generation's image context) is detected from the model name. When the detection is wrong — a custom deployment name, a fine-tune, or a proxy that strips images — set supportsVision for that model in the same modelCapabilities map:

{
  "modelCapabilities": {
    "my-custom-model": {
      "supportsVision": true
    }
  }
}

An explicit supportsVision always wins over the name-based detection, in both directions. Custom-provider integrations can also declare vision support for the whole integration via the visionSupport setting; without it, image content is flattened to text placeholders before the request is sent.

When the probe runs.

  • Clicking Test Connection runs a probe and captures the result in the form. A success toast confirms the connection.
  • Clicking Update / Create runs the probe automatically if no successful test has been performed in this session, or if any credential-affecting field (provider, API key, endpoint, deployment name, default model) has been edited since the last test. A failed probe at this stage aborts the save and surfaces the same error toast as Test Connection — the admin can fix and retry without losing form state.

Runtime fallback. The same adapters also carry an in-memory cache that catches and remembers parameter-rejection errors at chat-request time. If a model is updated by the provider after an integration is configured, the first chat request will hit the new error, retry without the rejected param, and remember it for the rest of the process — giving you correct behavior until the next Test Connection re-probes and persists the change.

Project Assignment​

After creating an LLM integration:

  1. Go to Project Settings → AI Models
  2. Select the integration from available options
  3. Optionally assign a Prompt Configuration to customize how AI prompts behave for this project
  4. Optionally override the LLM integration for individual AI features (see Per-Feature LLM Overrides below)
  5. Save settings

Per-Feature LLM Overrides​

Project admins can override which LLM integration is used for specific AI features, independently of the project-wide default. This is configured per project on the AI Models settings page and applies to the following features: Test Case Generation, Markdown Parsing, Smart Test Case Selection, Editor Writing Assistant, LLM Connection Test, QuickScript, Auto-tagging, and Automation Candidates.

How to Configure​

  1. Navigate to Project Settings → AI Models
  2. Find the Per-Feature LLM Overrides card, which shows a table with four columns:
    • Feature — The AI feature name
    • Override — A dropdown to select a specific LLM integration for this feature. An X button appears next to the dropdown when an override is active; click it to clear the override.
    • Effective LLM — The LLM integration that will actually be used when this feature is invoked
    • Source — A color-coded badge showing where the effective LLM comes from (see below)
  3. For any feature row, select an LLM integration from the Override dropdown to assign that integration for this feature in this project
  4. To explicitly disable a feature for this project, select No LLM (disabled) from the dropdown — the feature will be unavailable regardless of project default or prompt config settings
  5. To remove an override, click the X button next to the dropdown — the feature will fall back to the next level in the resolution chain

Source Badges​

BadgeColorMeaning
Project OverrideBlueA per-feature override is set on this project
Disabled (Override)OrangeThe feature is explicitly disabled for this project via override
Prompt ConfigGrayThe prompt configuration has a per-prompt LLM assignment
Project DefaultOutlineThe project's default LLM integration is being used
No LLM configuredRedNo integration is available at any level

Use Cases​

  • Use a fast, inexpensive model (e.g., GPT-3.5) for connection tests and a more capable model (e.g., GPT-4) for test case generation
  • Override a single feature to use a different AI provider without changing the configuration for other features
  • Allow project teams to customize their LLM usage independently of the system-wide prompt configuration

LLM Resolution Chain​

When an AI feature is invoked, TestPlanIt resolves which LLM integration to use by checking three levels in order of priority:

  1. Project Feature Override (highest priority) — If the project has a per-feature LLM override configured for this specific feature (via the Per-Feature LLM Overrides table), that integration is used.
  2. Prompt Configuration Assignment — If the resolved prompt configuration has a per-prompt LLM integration assigned to this feature, that integration is used (along with the model override, if one is set).
  3. Project Default Integration (lowest priority) — The project's default LLM integration is used as the final fallback.
Feature invoked
  |
  v
Project Feature Override set? --> YES (integration) --> Use override integration
  |                           \-> YES (disabled)    --> Feature unavailable
  NO
  v
Prompt has per-prompt LLM assigned? --> YES --> Use prompt's integration (+ model override if set)
  |
  NO
  v
Use project default integration

This resolution chain applies independently per AI feature. Different features in the same project can resolve to different LLM integrations depending on what is configured at each level.

Per-prompt LLM assignments are configured in the admin prompt editor. See Prompt Configurations — Per-Prompt LLM Assignment for details.

Security Considerations​

Data Privacy​

  • API Requests: Source material is sent to AI providers for processing
  • Retention: Most providers don't retain request data (verify with your provider)
  • Sensitive Data: Avoid including sensitive information in source material
  • Self-Hosted Options: Consider Ollama for maximum data privacy

Access Control​

  • Permission Model: Same as regular test case creation
  • Audit Logging: All AI generation activities are logged
  • Rate Limiting: Built-in rate limiting prevents abuse

Migration and Updates​

Upgrading AI Providers​

  1. Create new integration with updated settings
  2. Test generation quality with new provider
  3. Update project assignments
  4. Archive old integration when satisfied

Model Updates​

  • New models are automatically available when providers release them
  • Update model names in integration settings
  • Test generation quality with new models before switching

Monitoring and Analytics​

Usage Metrics​

Track important metrics in the admin dashboard:

  • Generation Volume: Number of test cases generated per period
  • Success Rate: Percentage of successful generations
  • User Adoption: Which teams are using AI generation
  • Cost Tracking: API usage and associated costs

Quality Metrics​

  • Review Rate: Percentage of generated cases that are reviewed before import
  • Acceptance Rate: Percentage of generated cases that are imported
  • Modification Rate: How often generated cases are edited post-import

Future Enhancements​

Planned improvements include:

  • Custom Model Fine-Tuning: Train models on your specific domain
  • Multi-Language Support: Generate test cases in different languages
  • Visual Test Generation: Generate test cases from UI mockups
  • Regression Analysis: Automatically update test cases when requirements change
  • Test Execution Integration: Connect generated cases to automation frameworks
  • Magic Select Improvements: Historical analysis of test run patterns for better suggestions