How do you troubleshoot a production API issue in Azure?

Troubleshooting a Production API Issue in Azure

The goal is to stabilize first, diagnose second, fix third. Work from the outside in: is it the platform, the network, your code, or a dependency?

1. Triage (first 5 minutes)

Establish the facts before touching anything:

• What is the symptom? Errors (5xx/4xx), slowness, timeouts, or total outage?
• Scope: all users or some? All endpoints or one? One region or instance, or everywhere?
• When did it start? Line it up against deployments, config changes, certificate expiries, and traffic spikes. Most incidents trace back to a recent change.
• Is it Azure? Check Azure Service Health and Resource Health for your subscription and the specific resource. Platform incidents and maintenance show up here.

If a recent deployment is the likely cause, roll back first and investigate later. With deployment slots, swap back. That's the fastest mitigation you have.

2. Check the golden signals in Application Insights

Open Application Insights and go through these in order:

• Failures blade: which operations and exception types are spiking, with sample failed requests.
• Performance blade: which operations got slower, and whether the time is in your code or in a dependency.
• Application Map: which node is red (slow or failing). This shows quickly whether the API or something it calls is at fault.
• Live Metrics: real-time view while you test or mitigate.
• End-to-end transaction details: follow a single failed request through all services using its operation ID.

Useful KQL:

// Failure trend by result code
requests
| where timestamp > ago(2h)
| summarize total = count(), failed = countif(success == false) by bin(timestamp, 5m)
| extend failureRate = todouble(failed) / total
| render timechart

// Is it a dependency?
dependencies
| where timestamp > ago(2h) and success == false
| summarize count() by target, type, resultCode
| order by count_ desc

// Latency percentiles by operation
requests
| where timestamp > ago(2h)
| summarize p50 = percentile(duration, 50), p95 = percentile(duration, 95), count() by name
| order by p95 desc

// Top exceptions
exceptions
| where timestamp > ago(2h)
| summarize count() by type, outerMessage
| order by count_ desc

Compare the bad period against the same time yesterday or last week to separate real regressions from normal patterns.

3. Narrow down by symptom

Symptom	Likely causes	Where to look
500s after a deploy	Bug, missing config, bad migration	Exceptions, deployment history, diff of app settings
500s / timeouts, no deploy	Failing dependency (SQL, Redis, downstream API), expired secret or cert	Dependencies table, Application Map
401/403	Expired token or secret, changed app role or RBAC, clock/audience mismatch, rotated key	Entra sign-in logs, role assignments, Key Vault access logs
429	Throttling by a downstream service (Cosmos DB RU limits, Key Vault, Storage) or by APIM	Response headers, service metrics
502/503/504	App crashing or restarting, instances unhealthy, gateway timeout (App Gateway/Front Door ~230s limit on App Service)	Health check status, restarts, Diagnose and solve problems
Slow responses	Slow SQL queries, cold starts, thread pool starvation, CPU/memory pressure, chatty calls	Performance blade, dependency durations, App Service metrics
Intermittent failures	One bad instance, SNAT port exhaustion, transient faults without retries, DNS issues	Per-instance breakdown (cloud_RoleInstance), connection metrics
Works in one place, not another	NSG/firewall/private endpoint/DNS change	Network diagnostics (below)

4. Platform-level diagnostics

App Service / Functions:

• Diagnose and solve problems blade: guided detectors for HTTP 5xx, high CPU, memory, slow requests, SNAT exhaustion, and app crashes. Start here when the cause isn't obvious.
• Metrics: CPU, memory working set, requests, response time, HTTP 5xx, and instance count. Look for saturation or a scale-out that didn't happen.
• Log stream for live output: az webapp log tail --name my-app --resource-group my-rg
• Kudu (SCM) console for process explorer, environment variables, and crash dumps.
• Health check: verify the path is configured and which instances are being removed from rotation.

Containers / AKS / Container Apps:

• kubectl get pods, kubectl describe pod, kubectl logs --previous for restart loops (CrashLoopBackOff, OOMKilled).
• Check readiness and liveness probes, resource limits, and node health.
• For Container Apps, use the Log stream and ContainerAppConsoleLogs / ContainerAppSystemLogs tables.

APIM / Front Door / Application Gateway (if in front):

• Check whether failures originate at the gateway or the backend. Gateway logs show backend response codes, backend time vs. total time, and WAF blocks.
• A WAF rule or rate limit blocking legitimate traffic often looks like an app problem.

5. Check dependencies

Most production API incidents originate in something the API calls:

• Azure SQL: Query Performance Insight, blocking and deadlocks, DTU/vCore saturation, connection limits, failover events. Check whether a new query plan or missing index is the cause.
• Cosmos DB: 429s and RU consumption, hot partitions.
• Service Bus / Event Hubs: growing queue depth, dead-letter counts, lock-expired errors, throttling.
• Key Vault: throttling, or access denied after a role or network change.
• Redis: memory pressure, evictions, connection limits.
• Storage: throttling and latency metrics.
• Third-party APIs: check their status pages and your outgoing call durations.

6. Network and connectivity

If the app can't reach something or can't be reached:

• Network Watcher: Connection troubleshoot, IP flow verify, and NSG flow logs show whether a rule is dropping traffic.
• DNS: private endpoints need correct private DNS zone links. From Kudu, run nameresolver <host> and tcpping <host:port> (on Windows App Service).
• Firewall and access restrictions: recent changes to storage, SQL, or Key Vault firewalls often cause sudden 403s or timeouts.
• SNAT port exhaustion: many outbound connections to the same endpoint can exhaust ports. Reuse HttpClient through IHttpClientFactory, use connection pooling, or add a NAT Gateway.
• TLS and certificates: expired or rotated certificates cause sudden handshake failures.

7. Compare against change history

Ask "what changed?" and verify it from records, not memory:

• Activity Log: shows control-plane changes (config updates, scaling, role assignments, deployments, key rotations). Filter by time window around the incident start.
• Deployment history / CI-CD pipeline runs and any app setting changes.
• Entra audit logs: app registration, secret, or permission changes.
• Resource Graph or Change Analysis (in Diagnose and solve problems) to see configuration diffs.

8. Go deeper when needed

• Memory or CPU issues: capture a memory dump or profiler trace through Diagnose and solve problems (Collect Memory Dump, Profiler) or dotnet-dump/dotnet-trace in containers.
• Thread pool starvation (common with sync-over-async): look for rising response times with low CPU, and check ThreadPool metrics in Application Insights or dotnet-counters.
• Reproduce in a lower environment using the same request payload and the operation ID details from the failure.
• Increase logging temporarily for the affected component, then revert it. Sampling can hide rare failures, so check whether it's dropping the data you need.

9. Mitigate while you investigate

• Roll back or swap slots.
• Scale up/out if it's resource saturation (buys time, doesn't fix the cause).
• Restart the app or a single bad instance as a last resort. Note the time and capture evidence first, because a restart destroys state.
• Disable the faulty feature with a feature flag, or route traffic away (Front Door / Traffic Manager) from an unhealthy region.
• Add retries with backoff and circuit breakers (Polly, or Microsoft.Extensions.Http.Resilience) if a dependency is flaky.
• Communicate: post status updates early, even if the cause is unknown.

10. Afterwards

• Write a blameless post-incident review: timeline, root cause, detection gap, and action items.
• Add the missing alert that would have caught it sooner (failure rate, p95 latency, dependency failures, queue depth, certificate and secret expiry).
• Add health checks, synthetic availability tests (Standard tests), and dashboards for the signals you had to hunt for.
• Make the fix safer to ship: deployment slots, canary releases, automated rollback, and load testing for the failure mode.

Quick checklist:

1. Check Service Health and recent changes (deploys, config, secrets, certs).
2. Open Application Insights: Failures, Performance, Application Map.
3. Decide: my code, a dependency, or the platform/network?
4. Pull logs and traces for a single failing operation ID.
5. Mitigate (rollback, scale, failover) before the deep dive if users are affected.
6. Confirm recovery from the metrics, then run the post-incident review.