The performance issues impacting our webhooks system have been resolved, and our queues remain at healthy levels. Webhooks are again functioning normally.
Posted Sep 24, 2015 - 16:56 UTC
Monitoring
Our webhooks workers have fully worked through the backlog that built up since we originally started seeing DB performance degradation. Webhooks should be working normally now, but we're still monitoring to make sure everything's working smoothly.
Posted Sep 24, 2015 - 16:14 UTC
Identified
We've identified a database issue underlying our degraded webhooks performance. We've taken some steps to alleviate the problem and the backlog is now down to roughly half its size at the time of the last update. We will continue to monitor the queue until everything's back to normal.
Posted Sep 24, 2015 - 15:06 UTC
Investigating
Our workers responsible for delivering webhooks are experiencing degraded performance at the moment. No events have been lost, but we currently have a large backlog of undelivered hooks that the workers are working through slowly. We're investigating the issue and will provide an update when we have more information.