Imagine this
You are an owner of a non-profit website hosted on a Starter tier plan. This day you woke up early morning. Following your daily schedule, you first log in your e-mail account for new messages. While checking your inbox, you see a suspicious e-mail from your hosting company. It is an invoice message. You have used a free Starter tier for over 4 years now and never received any e-mail like that. You immediately open it and cannot believe what you now see…
At first I thought this is a joke or some scam email but after checking my dashboard it seems like I am truly owing them 104K dollars
After checking the hosting provider “Billing” panel, you quickly realize it is not a scam nor mistake. You are now owing the hosting company a ridiculous six-figure due to be paid, without even knowing why.
The scariest part? This is all real. This may happen to you or any person hosting a website online. A bit of neglection mixed with a malicious actor is a recipe for catastrophe.
Further investigation
Right after receiving a disturbing invoice, the worried customer had notified his Hosting Company – Netlify – about the incident. He wrote an e-mail and response from Customer Support Team came back quickly. The Team had confirmed what the customer was suspecting: his website was targeted by a large Distributed Denial of Service (DDoS) attack. In span of just 4 days, the website had consumed over 164 Terabytes of bandwidth.
After looking into this further, it seems a lot of this bandwidth usage came from some user agents that are quite ancient using Google Cloud addresses […] So either you have a fanbase with a passion for older technology, or this wa likely a DDoS attack. To me, this seems to be the latter.
The extensive e-mail by the Support Team has revealed the main cause of excess bandwidth usage – an seemingly innocent audio file. It was a Cantaneese song with a size of 3.44 MB. The issue was that the file was hosted directly on the victim’s website, which means that every single download of this file would cause a bandwidth usage rise of 3.44 MB. Normally, such large files should be hosted elswhere – for example directly on music platforms (such as YouTube or SoundCloud).
A hacker abused the customer’s mistake, and initiated a huge DDoS attack onto his website. It focused specifically on the song file, which resulted in over 48 million of its’ downloads in just 4 days. Consequently, this malicious action consumed 164 Terabytes of bandwidth, which in return generated an enormous six figure bill.
Rejected discount
In the same e-mail, Customer Support has also mentioned about a special discount they are willing to give to the customer to reduce the invoice due amount. In case of DDoS attacks, the company normally reduces the bill cost to 20%, but because of how huge the invoice would still be, the Customer Support decided to reduce it further down to 5%.
We normally discount these kinds of attacks to about 20% of the cost, which would make your new bill $20,900. I’ve currently reduced it to about 5%, which is at $5,225.
Below is the actual e-mail from Customer Support:
What might seem as a surprise is that the Customer did not accept the discount. He felt deceived by the hosting company and did not want to pay anything. In order to look for some help – under nickname of liubanghoudai24 – he posted his story on Reddit, expressing his disappointment:
This feels more like a scam to me. Why do serverless platforms […] not have ddos protection, or at least a spend limit? They should have alerted me if the spending skyrocketed […] It feels like they deliberately not support these features so that they can cash grab in situations like this.
The post spread like a wildfire, receiving many comments. The most popular one suggested liubanghoudai24 to post the story further – to HackerNews forum – which he eventually did. Shortly after, the story went viral on both platforms, which caught attention of other Netlify customers. Many of them were shocked and decided to move out their projects from this hosting platform. Some of the most upvoted comments available on Reddit were from readers, who admittedly said that:
I’m not using Netlify ever again after reading this.
The situation was very alarming – more and more people shared similiar perspective both on Reddit and HackerNews. It caught the attention of the Hosting Company itself. Just 3 hours after initial Reddit post, the time has come for Netlify to take action.
Hosting company response
The situation for the hosting company was getting out of control. Customer posts on HackerNews and Reddit were ramping up in popularity very quickly, reflecting badly on company’s image. Matt Biilmann, Netlify CEO, had no better choice, than waive the entire bill:
How do we know that the HackerNews user “bobfunk” is an actual CEO of hosting company? Based on his prior activity on HackerNews, he presented himself as the CEO several times before (and this is also true for his newer entries). Furthermore, liubanghoudai24 edited his post on Reddit, confirming that the company waived the invoice, which is compliant with former response from “bobfunk”.
The bill cancellation, while seemingly an empathetic move, was more likely a PR approach to save the tarnished company image. Who knows, what would have happened if liubanghoudai24 posts have not gone viral? Would $5,225 invoice be ultimately cancelled?
Despite final outcome, the damage has already been done. The terrified customer has moved his website away from Netlify, so did many other clients, who left the hosting company for once and for all.
A non-entirely free Starter tier
An important take from this story is to be very careful with seemingly free hosting plans. For liubanghoudai24, he had never paid anything during his 4 years of hosting, but it took just one unfortunate DDoS attack to get a $104,500 bill overnight. You might think it should never happen on a free plan – and you are right. The catch is, Netlify Starter Plan, despite being framed as “Free to get started”, under some cicumstances was not completely free. Netlify was partially transparent with it.
The Starter Plan, where liubanghoudai24 used to host his site, was described as “Free To Get Started” with an important addition of “Start free, then pay as you go”. Here is the screenshot of Netlify Pricing Page from around late 2023 :
This could have been confusing to some users, who might not realize Starter Plan limitations and risks just by looking at the Pricing Page shown above.
By checking beyond about Starter Plan details, there was an actual note about the bandwidth limit of 100 GB. Once exceeded, an extra 100 GB bandwidth package is automatically added, which costs a fairly high amount of $55. Exceeding that 100 GB package meant adding yet another one worth $55, and this can repeat over and over again. There is no stopping point here. Netlify would never shut down the site, but rather keep charging a customer further with extra bandwidth packages. Even the Netlify has confirmed that in two separate threads: this and that
It is very likely that many of Netlify Starter Plan customers have chosen this plan without checking the more concerning details. It is clearly visible especially on Reddit thread, where some of the shocked users were viewing the Starter Plan as a “Free Plan”. While partially true, the actual free plan should shut down the hosted website as soon as limited resources are used up – which is not really the case for Starter Plan. As a result, a seemingly innocent “free tier” can become the worst nightmare of unaware clients, especially when a DDoS attack comes into play.
New Free Tier made the right way
Lessons learned, and on 12th November 2024 Netlify has announced a brand new Free Plan to come. Here is what we can read from Netlify’s blog post:
Whether you’re launching your next big idea or experimenting with the latest framework, every project should start without financial risk. That’s why we’re excited to introduce Netlify’s Free plan – an always-free solution for deploying your web projects.
On the surface, the Free Plan looked very similiar to the Starter Plan, but there is a significant difference between them. When user exceeds monthly limits in the Free Plan, Netlify will never charge the customer. Instead, the hosted website becomes instantly shut down for the remaining days in the calendar month. When new month starts, the usage metrics are resetted and website access is restored.
When Free Plan was initially added, it did not directly replace the Starter Tier. As you can see in the image from late 2024, both plans were included in their Pricing Page:
This change made it more transparent for potential customers of what are the implications when opting for a specific plan. It finally cleared up potential confusion what is a true free plan and what is not. Learning from own mistakes is a good habit, but this time it will not be enough to bring back lost customers and many others, who will likely never come back.
