Webhooks experiencing delays
Incident
Report for Atlassian Bitbucket
Resolved
Webhooks have been fully operational for the past hour. We are now working to restore request history functionality and will indicate publicly when we've done so. In the meantime, this incident is resolved as the issue that was directly negatively impacting customers (delayed webhook delivery) has been addressed.
Monitoring
In order to address the immediate impact of this issue we have disabled webhook request logging. Webhooks are now being delivered once again with minimal delay, but customers will notice that the request histories displayed in the UI are outdated—there will be no record of new requests.
The team is working on a solution to restore this functionality without impacting performance. We will provide another update when there is progress on that front.
The team is working on a solution to restore this functionality without impacting performance. We will provide another update when there is progress on that front.
Identified
In order to address the immediate impact of this issue we have disabled webhook request logging. Webhooks are now being delivered once again with minimal delay, but customers will notice that the request histories displayed in the UI are outdated—there will be no record of new requests.
The team is working on a solution to restore this functionality without impacting performance. We will provide another update when there is progress on that front.
The team is working on a solution to restore this functionality without impacting performance. We will provide another update when there is progress on that front.
Update
We have increased the capacity of our webhooks database to mitigate the delays we are seeing. Webhooks continue to be delivered but are currently delayed by up to 40 minutes. The team has also implemented a patch that should significantly improve performance and are in the process of getting it released. We will provide another update once the patch has been deployed.
Investigating
We are seeing significant delays in our webhooks service (up to 20 minutes) as a result of degraded database performance used by Webhooks. We are investigating mitigation strategies now and will provide an update when we have more information.
This incident affected: Webhooks.