How much Traffic is too much Traffic for CloudFlare?
Evidence suggests it's 100TB per month.
Before I go into the details I want to state two things first:
- CloudFlare generously provided most of the bandwidth for our site for a year, without any hiccups.
- We (unknowingly) violated their TOS. However, I was assured that was not the reason we were kicked.
So the reason I'm writing this is not because we were kicked (after all, CloudFlare was in the right to do so), but because of how shitty it went down.
For the last 6 years some friends and I have been operating this site that collects images posted in chat rooms on Quakenet IRC. The content gathered is somewhere between reddit and 4chan. It's purely a hobby project - I tried to monetize it through ads some time ago, but failed. Advertising-Networks don't want us as a customer, but I'm fine with that.
Until about a year ago, the site was running fine on a single dedicated server. However, bandwidth consumption was becoming a problem. At times the server's (unmetered) 100mbit/s connection was completely saturated. We've been growing quite a bit.
So we looked for a solution and stumbled upon CloudFlare. Their free plan was exactly what we needed. We didn't want any of the "speed" optimisations, but just their distributed loading to serve our images. It was extremely easy to set up and just worked. In disbelief I commented on HN that I don't understand how they could offer their service for free and that we'd happily pay a few hundred USD (screenshot, orig thread).
In that same comment thread I also expressed my concern that we're somehow using CloudFlare in a way that it was not intended for, but I was assured by CloudFlare's jgrahamc that it's fine. I was later told that jgrahamc's comment was made in error and that our usage was indeed against CloudFlare's TOS (Section 10: Limitation on non-HTML caching).
(Running the whole site through CloudFlare wouldn't make that much sense to us, because it's just one single HTML file, one CSS file and one JS file - about 70kb in total - everything else comes from a JSON API. We never load another HTML file. However, we would happily have enabled it for these files as well, if CloudFlare wanted us to. I suggested this on numerous occasions.)
Everything was running fine until Jan 22. 2013 when suddenly all images on our site didn't load. Opening an image URL directly showed that instead of the image, an HTML site was returned. This site is part of CloudFlare's DOS protection. It's an interstitial that intents to sort out bots.
However, we initially turned off CloudFlare's security settings, because a) if someone wanted to attack us, they'd send automated search requests to our backend and not try to load lots of images and b) CloudFlare just serves images; the interstitials they present would only be presented as broken images.
Logging in into CloudFlare I discovered that the security setting miraculously was changed from the lowest setting ("Essentially off") to the highest ("I'm under attack"). I never received any notification or explanation for the change. I switched it back and everything was running fine again, so I believed it was a fluke.
On Feb 1. CloudFlare completely disabled their service for us and sent all visitors directly to our main Server. The site was now unusably slow so I replaced it with a "Sorry, we'll be back soon" site.
When I logged in into CloudFlare, I was greeted by a notice:
CloudFlare has been temporarily disabled due to a system issue. To ensure there is no performance degradation to your website, we are temporarily routing all traffic directly to your server. Once peak performance is back, we will automatically re-enable CloudFlare.
Again, we were not notified about this by email or any other way.
I waited till the next day, then contacted CloudFlare, asking when we can expect the service to be back. They told us that it's not a "system issue", but that our site was under a "layer 7 attack":
Our operations team routed your site off CloudFlare because was seeing a large layer 7 attack that was negatively impacting other CloudFlare customers. They will review and re-enable the site automatically and no further contact is required.
CloudFlare never enabled our site again.
Curiously, at this time our CloudFlare stats boldly proclaimed "103TB traffic saved by CloudFlare in the last 30 days". Remember the site was disabled on Feb 1. It was the first month our traffic grew beyond 100TB, so I suspect that "layer 7 attack" is a sleazy term for "too much traffic".
On that weekend I talked to their support a bit more and was told that, based on our traffic, we need to be at least on the business plan ($200/mo) - which I would have agreed to - but was assured that someone else would talk to me again on Monday.
One week passed with no answer (I asked a few times), then, yesterday I was contacted again. They apologized for the delay and then told us:
At 100TB/mo., pure file delivery, you'd need to be an Enterprise customer. Let me know if this works within your budget.
The Enterprise plan comes at $3000/mo.
So CloudFlare disabled our site because of a "layer 7 attack", then let us in limbo for two weeks, where we couldn't commit to another solution, only to tell us we need to pay $3000/mo in the end.
We now ordered two servers from Leaseweb in NL, each with 1Gbit uplink and 100TB traffic included to run varnish caches and serve the images. These cost us $200/mo in total.
Tags: Random Thoughts
26 Comments:
That's pretty crappy of them.
As an aside, you might want to look into losslessly optimising your images using JpegOptim, jpegrescan, jpegtran, pngcrush, etc. You can try these tools out as part of ImageOptim on Mac OS X.
The result would be reduction of your image file sizes by around 20%, and even better in the case of the smaller images. Obviously, that would ease your bandwidth usage.
I think this is probably what they meant by "Layer 7" attack. It's a technical terms, not a marketing one.
This was flagged to my attention and I've reviewed all the interactions between you and our team. You were using the free version of CloudFlare's service. On February 2, 2013, your site came under a substantial Layer 7 DDoS attack. While we provide basic DDoS mitigation for all customers (even those on the Free CloudFlare plan), for the mitigation of large attacks you need at least the Business tier of CloudFlare's service. In an effort to keep the site online, our ops team enabled I'm Under Attack Mode, which is available for Free customers.
The attack continued and began to affect the performance of other CloudFlare customers, at which point we routed traffic to your site away from our network. While we encouraged you to take advantage of the Enterprise tier of service given your needs and traffic levels, your site would have been brought back onto CloudFlare's network if you upgraded to the Business tier of service ($200/mo) which included Advanced DDoS mitigation.
I am sorry for the confusion. To be clear, we do not bill based on traffic. However, resources are not infinite and when an attack against a Free customer begins to affect the performance of other customers we will take measures to protect the overall integrity of the CloudFlare service.
Best wishes,
Matthew Prince
CEO, CloudFlare
@eastdakota (Twitter)
Matthew: thanks for responding. I offered multiple times to upgrade to the Business plan, but was ignored. Also, if you change a site's setting, you should probably notify the account holder about it.
I didn't notice any "attack" when CloudFlare began to route all traffic directly to us. It looked like normal web traffic - much of it, but no more than usual.
lol cloudflare rep (oops, CEO) keeps calling it an attack.
I've reviewed the logs. There was definitely both a Layer 4 and Layer 7 attack. I don't know how long the attack lasted after we routed traffic directly since, obviously, we were no longer receiving logs. It may be that traffic is not being captured in your typical server logs. We pull both sflow data (off our routers and switches) as well as web server logs in order to detect attacks.
While we recommended the Enterprise level plan, if you had upgraded to Business you would have been brought back onto our network. I apologize if that was not clearly communicated.
@Matthew Prince the customer service problems OP detailed supersede the technical issues you are reiterating. Unless OP grossly misrepresents the interaction, you failed to communicate with him, while essentially bringing his site down. And you never really rectified it.
I started out reading this post thinking "Oh yeah, I should move my stuff over to cloudflare ... and anyway I don't do anything like that amount of traffic" to thinking "no way, this would be nightmarish" again, NOT because of the pricing tiers and TOS, but because of the lack of communication and inaction.
en.wikipedia.org/wiki/OSI_model#Layer_7:_application_layer
Sorry, this was the link I thought I had included earlier. ^
LeaseWeb is awesome! We've been with them for the past 2 years and they're rock-solid.
This sounds like a layer8 problem at CloudFare more than anything else.
Free plan or not, you don't just cut people off without even giving a notice. What tells me you don't do the same when I exceed some unknown limit on my "Business" or "Enterprise" plan?
Both enabling "I'm Under Attack Mode" or routing the traffic direct are both supposed to generate an automated message to the customer letting them know what happened. We've reviewed the logs and don't see a message having been sent. I'm investigating why that didn't happen since I agree it is not acceptable.
This is an example of a customer service fail not only in the initial correspondence, but also in the CEO's response. Even though you were a non-paying customer, his response should have been, "I'm so sorry for the confusing and inconsistent communication you experienced, and especially the downtime. We're working to find out why you received incorrect and conflicting information and why we didn't communicate the situation more effectively. If there's any way we can make this right and you'd consider returning as a customer, please call me at (xxx) xxx-xxxx."
Instead, the CEO's response sounded defensive and almost accusatory ("your site would have been brought back onto CloudFlare's network if you upgraded to the Business tier of service ($200/mo)"). This despite the fact the user reported he told customer service he'd be willing to pay at that level.
Customer is always right. Unless you don't want them to be your customer anymore.
@splurge: I think this is one of the best answers a company can give in such a situation, what else do you expect?
100TB is plenty IMHO and yes, there seem to be some problems - which the CEO already gave in.
A customer is a person who pays for a service. Someone who doesn't pay for a service (yet) is a *lead*. Not all leads are good business. 100TB of traffic does not sound like a good lead to me, not even at the $200 level.
For the record, I am not a shill for the company, just a happy CloudFlare user, nowhere near the usage levels of the OP, and I appreciate that CloudFlare thinks of other customers on the network. I'm actually baffled that I get for free what I do. Before a friend recommended CloudFlare, my site was attacked frequently, and taken down by my hosting company several times. I still see those attacks in the CloudFlare logs, but my site hasn't been down in a looooong time.
Looking at your site, I see an IMGUR clone which was running for free off of CloudFlare's servers. I really don't understand the nonsensical comments above. WTF is wrong with people these days thinking that everything is supposed to be free? Are you all 16 and on an weekly allowance?
Matt also has a very valid point that some sort of optimization of the stored (cached) files would have been a smart option for yourselves (less local storage) as well as CF (less to cache, less bandwidth). I can recommend www.jpegmini.com/server (Oh wait, it's not free, now what...)
I doubt Mathew is looking at this as a 'Oh no, lets keep a customer', instead more like 'Oh crap something went wrong, let's find out what'. And I believe his responses are in line with that. - I also believe this is how a CEO should look at it; a rep can give you that cookie-cutter style response that @splurge posted.
I personally use Cloudflare, It's been simple, trouble-free, and covering 80% web-requests and 35% bandwidth (Not to mention almost 20% faster site load). If you're reading this topic and thinking like @sarth "no way, this would be nightmarish" - put cloudflare test, and see if you're not thinking "This is free!?".
Maybe its just me, but something like AWS Cloudfront doesn't care if its images or whole site, that's what I would end up using.
"This is free!?" is the only possible reaction from any sane person to the range of services that Cloudflare provide with their free plan.
Try to get it all free anywhere else. Doubt it.
Complain about getting huge amounts of free stuff because the company's response is a bit geeky? Great idea, really annoy Cloudflare so they drop the free plans for the minor MiBs per month accounts like me.
The review for LeaseWeb hasn't been great. Anyone else use their service? Any comment about theri service?
Thanks,
You know, leaseweb's 100TB servers, also very clearly state, that the 100TB servers cannot be used as Content Delivery Networks, caches or streaming servers. It seems you have a problem at reading each service rules! Bandwidth is not cheap, and you will probably have problem with Leaseweb too at some point
If you want something for free, check CoralCDN ... just make sure it will get your needs (wiki.coralcdn.org/wiki.php?n=Main.FAQ#quota).
@Dominic: Why don't you just route your image traffic through cloudfront? It would cost you 8$ for the bandwidth plus $0,0075 per 10K requests
@Stéphane you probably misread GB for TB? According to the cloudfront website ( aws.amazon.com/cloudfront/ ), the price per GB in Europe is $0.12. At 100TB, or 100,000GB per month that's $12,000, not $8 :)
Blah blah attack blah ...
Blah blah free blah blah...
I don't think this is systemic, but I also don't think this is trivial. CloudFlare's "engineers" messed up. That's not the CEO's fault, even though he's out here damage-controlling. That's the fault of some dummy (or dummies) in the NOC at CloudFlare. Or perhaps some programmer who didn't test his "customer alerting auto-messaging" script enough. CloudFlare is still pretty awesome on the whole, especially considering you don't pay a penny for what you get. Look at it this way: You could be running a Rails project on Heroku........
I agree cloudflare is awesome. But if you look at the whole picture this seems like "lets get loose of that unprofitable customer". Support tells him he has to pay 3000USD for the enterprise plan and then after a blog post the CEO says all that lack of communication (and probably lies and procrastination) was a mistake and the 200USD plan will be ok. Who are you kidding? You disagreed to the 200USD plan because this customer just is not profitable and never will be (traffic will probably go even up more). If cloudflare had nuts they just admitted they dont want dominic as a customer and told him that instead of playing those unprofessional games.
@ quarkie ah I was going to say the same thing.
Cloudflare is looks like an insurance company.
You pay them when every thing is normal to protect you when something goes wrong.
But cloudflare get your money and once some DDOS attack happen they cut your website down and send all traffic directly to your server...
They only protect their brand customers, such as Spamhaus... If a DDOS attack use 50-60TB of their bandwidth, then you are done, Only enterprise customers can survive, Their enterprise plan start at $3000 and goes up $2500 for each additional 100TB
Post a Comment: