Incidents added from:
Incident Summary
We experienced degraded performance when creating sandboxes. During this period, users attempting to create sandboxes experienced longer wait times and could have encountered timeouts during these operations.
What Happened
An unusually high volume of sandbox creation requests overwhelmed our NFS cache infrastructure. As a result, the cache was unable to serve requests efficiently, causing reads to bypass the cache layer and hit our underlying cloud storage backend directly. This led to elevated storage read traffic and intermittent failures when creating new sandboxes.
Next Steps
We have implemented additional monitoring and alerting to detect slow NFS reads in the future. We are also working on improvements to prevent similar cache scenarios during high-volume workload spikes.
We apologize for any disruption this caused to your operations. If you have any questions or concerns, please contact our support team.