LLM API Migration Checklist: Move an AI Application Between Providers Safely

Overview

An LLM provider migration changes one or more layers of your application: authentication, request formatting, model selection, context handling, response parsing, observability, rate limits, and commercial configuration. Even when two providers expose similar chat or completion interfaces, their defaults and supported capabilities may differ.

Start by defining what “successful migration” means for your application. A useful definition usually includes:

• Required prompts produce acceptable outputs on a representative evaluation set.
• Structured responses remain valid and parseable.
• Tool or function calls preserve the expected names, arguments, and execution flow.
• Latency, error handling, throughput, and usage controls remain within your operational limits.
• Security, privacy, logging, and retention settings match your approved design.
• The previous provider can be restored without a rushed code reversal.

Keep provider-specific code behind a small adapter or gateway where possible. Your application should express its own stable concepts—such as messages, tools, response schemas, retries, and usage metadata—while the adapter translates them into a provider’s API format. This reduces the cost of future provider switches and makes compatibility gaps visible during testing.

Before changing production traffic, review the production-ready AI workflow guide and establish monitoring, ownership, and rollback responsibilities.

Checklist by scenario

1. Preparing for any provider switch

• Record the current provider, model identifiers, SDK versions, API version settings, regions, and deployment names.
• Inventory every model call, including chat, embeddings, moderation, reranking, vision, batch, and background jobs.
• Export representative inputs and approved outputs. Include short prompts, long contexts, edge cases, multilingual content, refusals, empty results, and malformed user input.
• List system instructions, prompt templates, few-shot examples, retrieval instructions, output schemas, and post-processing rules.
• Document timeout values, retry behavior, concurrency limits, fallback routes, caching, streaming, and token or usage budgets.
• Identify where credentials, model names, endpoints, and provider settings are stored. Move them into configuration rather than hard-coded application logic.
• Define acceptance thresholds for correctness, format validity, latency, failure rate, and cost controls without assuming the new provider will behave identically.

2. Comparing provider compatibility

Create a comparison record before writing migration code. Mark each item as supported, supported with changes, unavailable, or not yet tested.

3. Adapting the application

1. Centralize configuration. Add a provider configuration object containing endpoint, credential reference, model, timeout, retry policy, and feature flags. Keep secrets outside source control.
2. Build an adapter. Normalize messages, generation settings, tool definitions, response text, structured data, usage metadata, and errors into an internal interface.
3. Translate prompts carefully. Do not rewrite prompts merely to make them look provider-specific. First test the existing prompt, then make the smallest deliberate changes needed for roles, delimiters, context ordering, or output instructions.
4. Handle unsupported features explicitly. If a provider lacks a capability, choose a documented fallback, disable the feature, or stop the migration for that workflow. Silent degradation is difficult to detect.
5. Update parsing and validation. Treat model output as untrusted input. Validate JSON, enum values, required fields, tool arguments, citations, and length constraints before downstream use.
6. Preserve observability. Continue recording request identifiers, model configuration, latency, retry counts, validation failures, and safe usage metadata. Avoid logging sensitive prompts or outputs unless your data-handling design permits it.

4. Testing before cutover

Run the same evaluation set against the old and new implementations. Compare task success, factual or business-rule accuracy, format compliance, tool-call correctness, refusal behavior, latency, errors, and resource consumption. Review outputs manually for high-impact workflows; aggregate scores alone can hide regressions in a small but important class of requests.

Version the prompts, adapter code, model configuration, test data, and evaluation results together. The prompt testing and regression guide can help organize repeatable checks. For retrieval-augmented applications, test retrieval and generation separately so a change in embeddings, context formatting, or answer synthesis is not mistaken for a single model issue.

5. Cutting over and rolling back

• Deploy the new integration dark or behind a feature flag before sending user traffic.
• Run health checks that exercise authentication, a minimal request, structured output, and any critical tool path.
• Start with a controlled traffic slice or internal users where the application allows it.
• Watch errors, validation failures, latency, rate-limit responses, user feedback, and downstream business metrics.
• Increase traffic only after the agreed checks remain healthy for a defined observation period.
• Keep the previous provider configuration available until the new path is accepted.
• Document the rollback command or configuration change, decision owner, trigger conditions, and data reconciliation steps.

What to double-check

Model names and aliases: Treat model identifiers as configuration, not durable business logic. Record the exact identifier used in each environment and verify access before deployment.

Prompt semantics: Differences in role handling, instruction priority, tokenization, default sampling, stop behavior, and context truncation can change results without any visible code error. Make important settings explicit and test long-context cases.

Structured responses: A response described as JSON may still contain invalid syntax, missing fields, extra properties, or a schema mismatch. Use a parser and schema validator, then define a safe recovery path.

Tool execution: Validate tool names and arguments before execution. Confirm whether parallel calls, repeated calls, streaming tool events, and tool errors are represented the same way. Never let a provider-specific response shape bypass authorization checks.

Retries and idempotency: Retrying a failed request may duplicate an external action if tool execution or side effects are involved. Separate generation from execution where possible, and use idempotency controls for operations that change data.

Limits and usage accounting: Recheck input limits, output limits, request size, concurrency, quotas, and usage fields. Do not assume that a setting with the same name has the same meaning across providers.

Security and privacy: Revisit data classification, credential scope, access controls, prompt logging, document handling, and retention. Use the AI data privacy checklist alongside your organization’s own requirements.

Common mistakes

• Changing provider and prompt at the same time: This makes regressions hard to attribute. Establish a baseline first and isolate changes where practical.
• Testing only the happy path: Include long inputs, empty fields, adversarial instructions, malformed tool arguments, provider errors, timeouts, and partial streams.
• Assuming API compatibility means behavior compatibility: Similar request schemas do not guarantee similar outputs, reasoning patterns, refusals, or formatting.
• Ignoring downstream contracts: Check databases, queues, search indexes, UI components, and analytics consumers that depend on response shape or metadata.
• Removing the old provider too early: Keep a tested rollback path until production evidence supports decommissioning.
• Comparing only average quality: Inspect critical failure cases and segment results by workflow, language, input length, and tool use.
• Leaving model configuration scattered: Centralized configuration makes audits, experiments, and future migrations safer.

When to revisit

Revisit this checklist before every provider switch, model replacement, major SDK upgrade, or architecture change. It is also useful during seasonal planning cycles, when usage patterns or capacity assumptions change, and whenever a provider changes model availability, authentication requirements, context limits, structured-output behavior, tool calling, rate limits, or pricing. Treat provider documentation and your own integration tests as change inputs rather than relying on an old comparison table.

After each migration, record what changed, which tests passed, which limitations remain, and how to roll back. Schedule a small migration rehearsal for critical applications so the team can verify credentials, deployment configuration, evaluation data, alerts, and rollback access before an urgent change is required.

Action plan: create the configuration inventory, build the provider comparison table, freeze a baseline evaluation set, implement or verify the adapter, run regression tests, stage the new route behind a feature flag, and write the rollback procedure before directing production traffic. For ongoing prompt maintenance, see the prompt templates for AI developer workflows and the guide to AI output evaluation metrics.