~2.3M claims records delayed 42 minutes. Downstream reports published late. Customer dashboard 1 hour stale. SLA impact: 0.8% of daily reliability window.

DPU allocation exhausted due to undocumented parallel activity. Resource contention between incremental load and ad-hoc reporting queries.

1. Scaled IR from 2 to 4 DPUs (15 min) 2. Paused non-critical reporting queries (3 min) 3. Increased ADF activity timeout from 120 to 180 min 4. Restarted pipeline with prioritization (18 min) 5. Validated data consistency post-recovery (6 min)

Implement usage tracking dashboard for ADF IR resources. Document capacity planning assumptions. Add runbook validation step for competing workloads before pipeline execution. Increase standard IR size from 2 to 3 DPUs.

~145K records in limbo state briefly. Downstream processes waited 18 minutes. Recovery validated against raw data source.

Long-running standardization transaction holding locks on claims and policy tables. Concurrent update process waiting for same tables. Lock wait timeout exceeded.

1. Identified blocking session (2 min) 2. Analyzed lock hierarchy with sp_who2 and sp_locks (5 min) 3. Safely killed long transaction after documenting state (3 min) 4. Reran standardization with updated lock hints (8 min)

Add index on claims.policy_id to reduce scan time. Implement connection pooling timeout settings. Create alert for lock waits > 10 sec. Document lock hierarchy in runbook.

~1.2M validation records pending 32 minutes. ML pipeline features delayed. Analytics queries timeout.

Auto-scale policy hit node quota limits. Cluster could not add workers. Pending tasks queued indefinitely.

1. Detected quota issue in Databricks API logs (4 min) 2. Contacted cloud ops for quota increase approval (8 min) 3. Manually scaled to max allowed workers as interim (5 min) 4. Once quota increased, resumed auto-scale policy (15 min)

Implement threshold-based pre-scaling before peak hours. Set up Databricks quota alerts. Document cloud ops escalation process. Test multi-region failover strategy.

Vendor data arrival delayed 67 minutes. Daily batch completion pushed 1.5 hours later. Report SLA impact minimal.

Network latency to vendor SFTP server spiked. Default timeout of 120 sec insufficient for 1.2 GB file transfer.

1. Observed transfer timeouts in logs (8 min) 2. Verified vendor connectivity and file size (12 min) 3. Updated timeout in automation config from 120 to 300 sec (5 min) 4. Manually resumed transfer with new timeout (42 min)

Implement parallel chunk transfer for large files. Add progress monitoring with streaming logs. Coordinate with vendor on off-peak transfer windows. Monitor MTU settings for network optimization.

Healthcheck missed 4 cycles. Alerts delayed ~23 minutes. No customer impact (monitoring only).

Long-running healthcheck accumulated connection objects without cleanup. Memory usage crept to 2GB, script OOM killed.

1. Identified memory growth pattern (6 min) 2. Analyzed script for unclosed connections (9 min) 3. Applied context manager fix and tested locally (5 min) 4. Deployed fixed version and restarted task (3 min)

Implement memory profiling in CI/CD. Add health metrics to script (memory, open connections). Set up memory threshold alerts. Use Python context managers as default pattern.