Search · Streaming
240m → 6m~40× fasterNo backfill stalls
OpenSearch ingestion pipeline rebuild
- Problem
- The existing DMS + Kinesis pipeline was slow, and large indexing runs blocked new data from being indexed — a hard ceiling on throughput.
- Approach
- Re-architected the path to Debezium → MSK (Kafka) → OpenSearch Ingestion, decoupling backfills from live writes and parallelizing the heavy lifting.
- Outcome
- Indexing 1M records went from 240 minutes to 6 minutes (~40× faster), and large runs stopped starving new indexing.