Traffic redirection for 4 domains, 28th September 2022

At bunny.net, we have a strong belief and commitment to transparency and openness, so when things go wrong, we will always share with you, our customers. We want to take incidents like this as an opportunity to learn, get better and grow, or even help other companies better protect their systems. Today, as part of this commitment, we're sharing the details of a security incident that took place on September 28th and affected four domains on our network.

What happened?

A third-party security researcher created several customer accounts in our system and discovered a (then) unknown vulnerability in our platform. The researcher acted without notification, without our knowledge, and without our consent.

The vulnerability was based on exploiting the preview release of the new Bunny DNS platform. This allowed the security researcher to use a chain of configuration settings in their accounts to bypass otherwise established security checks in the bunny.net platform due to a validation bug when creating automatically accelerated domains. This allowed the security researcher to take over a domain from a different account and apply a HTTP 301 redirect to a portion of the traffic. As a result, this caused the targeted domain to redirect traffic to a hostname set by the security researcher.

At this point, instead of approaching this discovery ethically and disclosing this vulnerability to us responsibly, the security researcher decided to instead target four production customer domains and disrupted those customers' services. The security researcher then emailed two affected customers directly during the disruption with details and the solution to the vulnerability.

Incident Timeline

• 16:40 UTC The first two domains of the same service were affected with limited customer impact
• 17:10 UTC The first two domains were released by the researcher
• 18:27 UTC The third domain was affected with limited customer impact
• 18:50 UTC The fourth and primary target domain by the researcher was affected and de-provisioned from our CDN nodes
• 19:01 UTC  We received a customer support ticket about an unconfigured zone that affected the fourth domain. The customer corrected the domain within a minute by themselves by re-deploying their own zone configuration
• 20:01 UTC We received an update from the customer regarding the fourth domain of a potential security issue with the domain now causing unwanted 301 redirect responses
• 20:02 UTC Our security incident response process was triggered, and we applied a temporary fix to stop the domains from redirecting traffic
• 20:05 UTC We identified the root cause and discovered the vulnerability
• 20:09 UTC The security researcher attempted to re-activate the redirect on the fourth affected domain which was briefly stopped, but the vulnerability remained unfixed
• 20:14 UTC The security researcher re-activated the redirect on the fourth affected domain once again
• 20:20 UTC The affected customer applied a fix for the fourth domain as suggested by the security researcher via email
• 20:22 UTC We hot patched the vulnerability from being exploited and mitigated the impact to the affected domain
• 20:35 UTC Our security incident response team deployed a platform-wide security update to permanently mitigate the vulnerability
• 20:35-22:00 UTC We observed the security researcher unsuccessfully attempting to use the (now mitigated) vulnerability on other domains

During and after the incident, our team performed an extensive security overview of the platform with no further vulnerabilities detected. Following up on 28th and 29th of September 2022, we finished our investigation confirming the three other domains were also affected by this incident on top of the primary one. By September 30th, we implemented a number of additional security measures to reduce any possibilities of similar indicents in the future. We reached out to the affected customers shortly after that.

Hopping Forward

Security incidents are part of making the internet hop faster. This incident was the first time in 8 years we have ever had to use our security incident response process. Despite the urgent situation, it was a proud moment to see our team operating so efficiently, and great to see that our planning for security incidents worked amazingly well.

Unfortunately, while the security researcher approached our customer claiming they operate ethically by being associated with a well-known coordinated vulnerability disclosure platform, it is clear that the security researcher should have disclosed the vulnerability in our platform to us, using the hostnames and data from the accounts they created to find the vulnerability, rather than disrupting live customer traffic.

While we don't agree with the security researchers approach, we do believe that the researcher had good intentions but did not consider the impact of their methods.

We will be launching our own bug bounty on a coordinated vulnerability disclosure platform in the near future. We hope this will give security researchers a better platform to perform testing responsibly and take a better approach to keeping the internet a safer place.

Going forward, we are determined to treat this as an opportunity to take an even stronger stance on security. As a public gateway to the internet for over 1 million websites, we take security very seriously, and we are continuing a platform-wide security overview to make sure similar incidents cannot happen again in the future.  On top of the existing improvements, we are dedicating the following weeks to continued security strengthening of the bunny.net platform. We are committed to using this event to make the platform more secure than ever.

We also are increasing our code testing and we are hiring a dedicated Security focused Test & QA team to further augment our existing DevSecOps model. If you would like to be part of our team and think your skillset fits, please send your CV over to careers@bunny.net.