System status and uptime

Impact

Inbound WhatsApp messages queued instead of arriving. Six design partner workspaces saw a delay of up to 34 minutes on that one network. Sending still worked from the handset, and search, the API, the apps and the other networks kept running. No message was lost.

Cause

WhatsApp shipped a protocol version bump to a slice of paired sessions. Our session worker rejected the unknown frame type and exited, and the supervisor restarted it into the same rejection about every four seconds.

Resolution

We shipped a handler that accepts and ignores unknown frame types, then drained the queue. The 34 minute backlog delivered in 6 minutes and 20 seconds. A change on WhatsApp's side sits outside the SLA, so the credit we put on the six affected invoices was discretionary rather than owed.

What changed after

Session workers now run a contract test against the WhatsApp beta channel every hour, and queue depth above 500 pages an engineer instead of waiting for the uptime check.

Impact

Searches returned correct results, slowly. p95 latency reached 2.4 seconds across two of nine index nodes, so roughly a fifth of queries felt slow. Message delivery, sending and the AI features were normal.

Cause

A design partner imported 1.2 million messages from three platforms at once. The import wrote about 9 million index segments, and the merge policy took query threads on the two nodes holding those shards.

Resolution

We moved the import onto a dedicated indexing pool and throttled merges back to a quarter of the node's threads. Latency was under 300ms within four minutes of the change.

What changed after

Bulk import traffic now lands on separate nodes from live queries, and every tenant has a merge thread budget that cannot take the query pool.

Impact

Thread summaries, translation and voice transcription arrived up to 20 minutes late. The inbox, search, sending and the webhooks ran normally. Every queued job eventually completed.

Cause

One GPU node in Frankfurt failed its health check partway through a driver rollout. The scheduler kept placing work on it because a failed check only removed the node after three consecutive failures, and the node passed every third check.

Resolution

We drained the node and rolled the driver back to the previous version. The queue cleared in 11 minutes once the node stopped accepting work.

What changed after

Driver rollouts now go one node at a time behind a canary, and two consecutive failed health checks drain a node hard instead of three.