Incidents

Post-incident reports for service disruptions.

2026-06-22 — Realtime Gameplay Disruption (Redis Memory)#

Duration: ~7 minutes (01:14–01:21 UTC) Severity: Medium Impact: Brief disruption to in-progress games; no data lost

What happened

A single game generated an unusually large amount of data in a short space of time, which pushed Redis, the in-memory store that powers live gameplay, past its memory limit. Redis was then killed and automatically restarted several times over about 7 minutes.

Each restart briefly dropped the real-time connection that carries game actions between players, so anyone in an active game during that window saw stutter, dropped actions, or reconnects. Browsing, decks, and match history were unaffected, and no data was lost — game state recovered on its own.

Root cause

Game state stores the full details of every card in play. Under certain conditions a single game's saved state can grow far larger than expected — here, large enough to exceed Redis's memory limit.

Two things let one game's data disrupt Redis for everyone:

• Full card data is copied into each game's state, so an unusually large board can multiply the size dramatically.
• The Redis persistence we enabled after the 2026-02-27 incident (below) temporarily increases memory use while saving, and Redis shares a server with limited spare memory. Together these turned a single oversized save into repeated restarts rather than one rejected action.

This was not a capacity problem. Under normal peak load Redis runs at a small fraction of its limit; one game behaved pathologically.

Remediation

• Adding a size limit on game saves, so an oversized game is rejected and logged instead of affecting the server
• Reworking game state to reference cards rather than copy full card data, so an unusually large board no longer bloats saved games (the durable fix)
• Adjusting Redis persistence and memory settings to remove the amplification
• New monitoring to catch oversized saves and memory spikes before they affect players

2026-02-27 — Game State Loss (Redis Restart)#

Duration: ~06:37 UTC until manual detection Severity: High Impact: All in-progress games lost

What happened

Redis restarted during an unattended-upgrades cycle at 06:37 UTC. Since game state lived only in Redis with no durable backup, every active game was wiped. The cleanup cron then marked all affected tables as expired.

Decks, accounts, and match history were unaffected (stored in Postgres).

Root cause

• Redis had no persistence (no RDB/AOF) — a restart means total data loss
• No backup layer existed for game state
• needrestart was configured to auto-restart services including Redis

Remediation

• Disabled unattended-upgrades to prevent uncontrolled service restarts
• Enabled Redis RDB persistence (save 60 1) — worst-case data loss reduced from "entire game" to "last 60 seconds of actions"