We start receiving more than 5000 events/minute, while our max rate is usually around 1000 events/minute.
The high load of incoming events continued our Redis database got full as it has been sized for only 3000 events/minute.
Events processing got stuck, and some processes started to crash.
The engineering team is notified and investigates the issue and remediation solution.
The Redis Database gets replicated for further investigation.
We increased the Redis database size to be able to absorb up to 6000 events/minute.
The engine starts reprocessing events.
The abusing user has been identified and flagged. Its Mergify installation has been suspended. Its account was generating 100 commit/s on a repository triggering associated CIs. The abusing repository also has been suspended/deleted on the GitHub side.
The engine has automatically dropped all its events and does not receive events from it anymore.