19th November @ 1:00 UTC

• We start receiving more than 5000 events/minute, while our max rate is usually around 1000 events/minute.

19th November @ 3:00 UTC

• The high load of incoming events continued our Redis database got full as it has been sized for only 3000 events/minute.

• Events processing got stuck, and some processes started to crash.

19th November @ 6:00 UTC

• The engineering team is notified and investigates the issue and remediation solution.

• The Redis Database gets replicated for further investigation.

• We increased the Redis database size to be able to absorb up to 6000 events/minute.

• The engine starts reprocessing events.

19th November @ 6:10 UTC

• The abusing user has been identified and flagged. Its Mergify installation has been suspended. Its account was generating 100 commit/s on a repository triggering associated CIs. The abusing repository also has been suspended/deleted on the GitHub side.

• The engine has automatically dropped all its events and does not receive events from it anymore.

Everything is back to normal.

We are continuing to work on a fix for this issue.

We're implementing long term fixes.

We have fixed the underlying issue and restored the service. We are now monitoring the platform and planning long term action to have this incident not happen again.