ADF Pipeline Cascading Failure - March 2024

Claims Data Mart, Reporting Dashboard, Daily SLA Reports

Root Cause: DPU allocation exhausted due to undocumented parallel activity. Resource contention between incremental load and ad-hoc reporting queries.

Detection: 3 min | Resolution: 42 min

Lessons: Implement usage tracking dashboard for ADF IR resources. Document capacity planning assumptions. Add runbook validation step for competing workloads before pipeline execution. Increase standard IR size from 2 to 3 DPUs.

Database Deadlock Under Load - February 2024

SSIS Standardization Job, Data Lake Writes, Read Replicas

Root Cause: Long-running standardization transaction holding locks on claims and policy tables. Concurrent update process waiting for same tables. Lock wait timeout exceeded.

Detection: 2 min | Resolution: 18 min

Lessons: Add index on claims.policy_id to reduce scan time. Implement connection pooling timeout settings. Create alert for lock waits > 10 sec. Document lock hierarchy in runbook.

Databricks Cluster Auto-Scale Failure - January 2024

Data Validation Workflow, ML Feature Engineering, Ad-hoc Analytics

Root Cause: Auto-scale policy hit node quota limits. Cluster could not add workers. Pending tasks queued indefinitely.

Detection: 5 min | Resolution: 32 min

Lessons: Implement threshold-based pre-scaling before peak hours. Set up Databricks quota alerts. Document cloud ops escalation process. Test multi-region failover strategy.

Vendor Transfer SFTP Timeout - December 2023

Vendor File Ingestion, Data Lake Daily Batch, Reporting Refresh

Root Cause: Network latency to vendor SFTP server spiked. Default timeout of 120 sec insufficient for 1.2 GB file transfer.

Detection: 10 min | Resolution: 67 min

Lessons: Implement parallel chunk transfer for large files. Add progress monitoring with streaming logs. Coordinate with vendor on off-peak transfer windows. Monitor MTU settings for network optimization.

Python Healthcheck Script Memory Leak - November 2023

Healthcheck Alerts, Infrastructure Monitoring, Alert Dashboard

Root Cause: Long-running healthcheck accumulated connection objects without cleanup. Memory usage crept to 2GB, script OOM killed.

Detection: 8 min | Resolution: 23 min

Lessons: Implement memory profiling in CI/CD. Add health metrics to script (memory, open connections). Set up memory threshold alerts. Use Python context managers as default pattern.