Skip to content
 
If you can read this, either the style sheet didn't load or you have an older browser that doesn't support style sheets. Try clearing your browser cache and refreshing the page.

(Wired)   How GitHub survived the biggest DDOS attack ever recorded   ( wired.com) divider line
    More: Interesting, Denial-of-service attack, DDoS attacks, DDoS attack, Attack, Internet service provider, increasingly popular DDoS, major DDoS attack, DDoS mitigation service  
•       •       •

2381 clicks; posted to Geek » on 02 Mar 2018 at 10:25 AM (33 weeks ago)   |   Favorite    |   share:  Share on Twitter share via Email Share on Facebook   more»



54 Comments     (+0 »)
 
View Voting Results: Smartest and Funniest


Oldest | « | 1 | 2 | » | Newest | Show all

 
2018-03-02 08:00:13 AM  
Not nice.
 
2018-03-02 08:23:47 AM  
That's a lot of traffic. And reassuring to see githib stood up to it.
 
2018-03-02 10:30:51 AM  

Gubbo: That's a lot of traffic. And reassuring to see githib stood up to it.


They didn't.  Akamai did.  The only reason they went down at all is because their traffic isn't routed through Akamai 100% of the time.  Which is something they should reconsider.
 
2018-03-02 10:31:23 AM  
I find it bizarre and nearly criminally negligent that people would leave memcache servers exposed like that.  Put them in your DMZ for crying out loud.
 
2018-03-02 10:45:39 AM  

gingerjet: Gubbo: That's a lot of traffic. And reassuring to see githib stood up to it.

They didn't.  Akamai did.  The only reason they went down at all is because their traffic isn't routed through Akamai 100% of the time.  Which is something they should reconsider.


As someone who's currently trying to do just that, it's not as simple as it sounds.
 
2018-03-02 10:52:35 AM  
Probably just a test.
 
2018-03-02 11:05:44 AM  
Once upon a time, I was in a meeting where a VP asked the Akamai rep whether Akamai could take our traffic. His answer was that they could handle Netflix, Amazon, Youtube, Facebook, all the porn and all the cat photos without breaking a sweat.

I remember that. And I have never once been given cause to doubt that they could in fact handle all of that.
 
2018-03-02 11:10:04 AM  
Even the cat pictures?  Wow!
 
2018-03-02 11:12:57 AM  

WeedBong420: Probably just a test.


FTFA: Akamai's Shaul says he suspects that attackers targeted GitHub simply because it is a high-profile service that would be impressive to take down. The attackers also may have been hoping to extract a ransom.

Or maybe an advertisement for Akamai.
 
2018-03-02 11:17:14 AM  

montreal_medic: Once upon a time, I was in a meeting where a VP asked the Akamai rep whether Akamai could take our traffic. His answer was that they could handle Netflix, Amazon, Youtube, Facebook, all the porn and all the cat photos without breaking a sweat.

I remember that. And I have never once been given cause to doubt that they could in fact handle all of that.


I've seen that porn video where they handle all of it.
 
2018-03-02 11:17:42 AM  

FrancoFile: I find it bizarre and nearly criminally negligent that people would leave memcache servers exposed like that.  Put them in your DMZ for crying out loud.


Isn't the DMZ specifically unprotected?
 
2018-03-02 11:39:52 AM  

OldJames: FrancoFile: I find it bizarre and nearly criminally negligent that people would leave memcache servers exposed like that.  Put them in your DMZ for crying out loud.

Isn't the DMZ specifically unprotected?


No.

That would just be "outside the firewall".

The DMZ is a zone that's set up to provide external access to selected internal assets.  Like the public-facing web server, or an API interface, or a VPN server.  It's very straightforward to provide a moderate level of access control (you can do straightforward port filtering and redirecting with an inexpensive boundary router, for example) without choking off throughput.
 
2018-03-02 11:53:22 AM  
I wonder if the same attacks are hitting bitbucket right now.
 
2018-03-02 12:01:38 PM  

montreal_medic: Once upon a time, I was in a meeting where a VP asked the Akamai rep whether Akamai could take our traffic. His answer was that they could handle Netflix, Amazon, Youtube, Facebook, all the porn and all the cat photos without breaking a sweat.

I remember that. And I have never once been given cause to doubt that they could in fact handle all of that.


Is it a crazy expensive service? Maybe Github didn't think it was worth the expense to use all the time?
 
2018-03-02 12:03:10 PM  
I am going to guess they survived by being awesome. Because Github is awesome.
 
2018-03-02 12:05:41 PM  

Gubbo: montreal_medic: Once upon a time, I was in a meeting where a VP asked the Akamai rep whether Akamai could take our traffic. His answer was that they could handle Netflix, Amazon, Youtube, Facebook, all the porn and all the cat photos without breaking a sweat.

I remember that. And I have never once been given cause to doubt that they could in fact handle all of that.

Is it a crazy expensive service? Maybe Github didn't think it was worth the expense to use all the time?


Yes. But when you absolutely positively need to serve every single mother farker on the net, accept no substitute
 
2018-03-02 12:08:03 PM  

montreal_medic: Gubbo: montreal_medic: Once upon a time, I was in a meeting where a VP asked the Akamai rep whether Akamai could take our traffic. His answer was that they could handle Netflix, Amazon, Youtube, Facebook, all the porn and all the cat photos without breaking a sweat.

I remember that. And I have never once been given cause to doubt that they could in fact handle all of that.

Is it a crazy expensive service? Maybe Github didn't think it was worth the expense to use all the time?

Yes. But when you absolutely positively need to serve every single mother farker on the net, accept no substitute


Github was only down for a few minutes. Ok, I'm sure they were bent over and gone in dry when it came to the price to save them, but if you're only going to be down for a few minutes, maybe it's worth the risk and saving the cash.
 
2018-03-02 12:13:48 PM  
Of all the things to attack - why Github?
 
2018-03-02 12:15:47 PM  

Fark_Guy_Rob: Of all the things to attack - why Github?


Why anything. But it's relied on by almost every tech company and essentially every open source project.

Bragging rights?
 
2018-03-02 12:34:21 PM  

Gubbo: montreal_medic: Gubbo: montreal_medic: Once upon a time, I was in a meeting where a VP asked the Akamai rep whether Akamai could take our traffic. His answer was that they could handle Netflix, Amazon, Youtube, Facebook, all the porn and all the cat photos without breaking a sweat.

I remember that. And I have never once been given cause to doubt that they could in fact handle all of that.

Is it a crazy expensive service? Maybe Github didn't think it was worth the expense to use all the time?

Yes. But when you absolutely positively need to serve every single mother farker on the net, accept no substitute

Github was only down for a few minutes. Ok, I'm sure they were bent over and gone in dry when it came to the price to save them, but if you're only going to be down for a few minutes, maybe it's worth the risk and saving the cash.



Absolutely. Akamai may cost more, but if you really need a CDN with some serious security and a pile of other features, you probably need Akamai

Every place I've been has had someone come up with some amazing combo of cheaper services and a bunch of custom code marrying a bunch of open source solutions, but when the excrement impacted the rotary ventilation device, the bits still hooked up to Akamai were the last ones still standing - every time
 
2018-03-02 12:42:12 PM  

montreal_medic: Absolutely. Akamai may cost more, but if you really need a CDN with some serious security and a pile of other features, you probably need Akamai

Every place I've been has had someone come up with some amazing combo of cheaper services and a bunch of custom code marrying a bunch of open source solutions, but when the excrement impacted the rotary ventilation device, the bits still hooked up to Akamai were the last ones still standing - every time


Am I hearing you right that you're accusing Akamai of launching these DDoS attacks themselves to enforce their protection racket?
 
2018-03-02 12:46:16 PM  

24601: montreal_medic: Absolutely. Akamai may cost more, but if you really need a CDN with some serious security and a pile of other features, you probably need Akamai

Every place I've been has had someone come up with some amazing combo of cheaper services and a bunch of custom code marrying a bunch of open source solutions, but when the excrement impacted the rotary ventilation device, the bits still hooked up to Akamai were the last ones still standing - every time

Am I hearing you right that you're accusing Akamai of launching these DDoS attacks themselves to enforce their protection racket?


I think you might to read that again?
 
2018-03-02 01:06:47 PM  

24601: montreal_medic: Absolutely. Akamai may cost more, but if you really need a CDN with some serious security and a pile of other features, you probably need Akamai

Every place I've been has had someone come up with some amazing combo of cheaper services and a bunch of custom code marrying a bunch of open source solutions, but when the excrement impacted the rotary ventilation device, the bits still hooked up to Akamai were the last ones still standing - every time

Am I hearing you right that you're accusing Akamai of launching these DDoS attacks themselves to enforce their protection racket?


Nope. I'm saying - and meaning 100% - that Akamai is often imitated, that lots of places look for ways to accomplish the same thing cheaper, and that I have never ever seen anyone who succeeded in equalling them. Expensive, yes - but worth every penny when uptime has value
 
2018-03-02 01:21:48 PM  
No snark, just curious:  What is the most enduring DDOS on record, and has any company (above some single proprietor business, of course) ever failed or taken permanently offline due to one?
 
2018-03-02 01:26:50 PM  

Flab: gingerjet: Gubbo: That's a lot of traffic. And reassuring to see githib stood up to it.

They didn't.  Akamai did.  The only reason they went down at all is because their traffic isn't routed through Akamai 100% of the time.  Which is something they should reconsider.

As someone who's currently trying to do just that, it's not as simple as it sounds.


As someone who's been doing it for six years.  Yes it is.
 
2018-03-02 01:38:42 PM  

gingerjet: Flab: gingerjet: Gubbo: That's a lot of traffic. And reassuring to see githib stood up to it.

They didn't.  Akamai did.  The only reason they went down at all is because their traffic isn't routed through Akamai 100% of the time.  Which is something they should reconsider.

As someone who's currently trying to do just that, it's not as simple as it sounds.

As someone who's been doing it for six years.  Yes it is.


I'm not talking about just having www.gingerjet.com be a CNAME for www.gingerjet.com.edgekey.net

I'm talking about blocking up traffic at your firewall when it does not come from Akamai, including managing the ever changing list of source IPs that Akamai uses.
 
2018-03-02 01:43:03 PM  

Flab: gingerjet: Gubbo: That's a lot of traffic. And reassuring to see githib stood up to it.

They didn't.  Akamai did.  The only reason they went down at all is because their traffic isn't routed through Akamai 100% of the time.  Which is something they should reconsider.

As someone who's currently trying to do just that, it's not as simple as it sounds.


also expensive
 
2018-03-02 02:04:20 PM  
To bad there's no way to reroute all that traffic back at the attackers.

/ya ya, no real grasp on how the inter webs really work
//would still be halarious! To me
 
2018-03-02 02:18:24 PM  

wolfemane: To bad there's no way to reroute all that traffic back at the attackers.

/ya ya, no real grasp on how the inter webs really work
//would still be halarious! To me


You can't easily do that.

What this was was the equivalent of calling every pizza place in the US and giving 1600 Pennsylvania Ave, Washington, DC as your address.
 
2018-03-02 02:46:41 PM  

The Googles Do Nothing: montreal_medic:
I've seen that porn video where they handle all of it.


I laughed out loud, in the office.  Thank you.  That will be the only joy I get today.
 
2018-03-02 02:56:29 PM  

Gubbo: 24601: montreal_medic: Absolutely. Akamai may cost more, but if you really need a CDN with some serious security and a pile of other features, you probably need Akamai

Every place I've been has had someone come up with some amazing combo of cheaper services and a bunch of custom code marrying a bunch of open source solutions, but when the excrement impacted the rotary ventilation device, the bits still hooked up to Akamai were the last ones still standing - every time

Am I hearing you right that you're accusing Akamai of launching these DDoS attacks themselves to enforce their protection racket?

I think you might to read that again?


I think you might [need] to read that again yourself.
 
2018-03-02 03:03:41 PM  

SansNeural: I think you might [need] to read that again yourself.


My Kingdom (or $5) for an edit button.

/and for mobile to work properly again
 
2018-03-02 04:12:31 PM  

wolfemane: To bad there's no way to reroute all that traffic back at the attackers.


I knew this would be memcache before I even opened TFA. I have some time before my next meeting so let me sketch some quick diagrams.

img.fark.netView Full Size


The attacker is faking their IP address so that when the memcache server responds, it responds to the target site (and the response is much larger than the request, so it's easy to get high magnification numbers with this type of attack).

The network the memcache server is on can be configured to detect and reject packets that have been spoofed.

BUUUUUUT, memcache servers aren't supposed to be exposed to the internet in the first place. So the very presence of a publically reachable memcache server is an indication of an IT department that has no clue what they're doing.
 
2018-03-02 04:16:56 PM  

lordargent: wolfemane: To bad there's no way to reroute all that traffic back at the attackers.

I knew this would be memcache before I even opened TFA. I have some time before my next meeting so let me sketch some quick diagrams.

[img.fark.net image 794x308]

The attacker is faking their IP address so that when the memcache server responds, it responds to the target site (and the response is much larger than the request, so it's easy to get high magnification numbers with this type of attack).

The network the memcache server is on can be configured to detect and reject packets that have been spoofed.

BUUUUUUT, memcache servers aren't supposed to be exposed to the internet in the first place. So the very presence of a publically reachable memcache server is an indication of an IT department that has no clue what they're doing.


A question to show my ignorance. The hell is memcache?

/don't say some kind of memory cache :)
 
2018-03-02 04:51:49 PM  

Gubbo: lordargent: wolfemane: To bad there's no way to reroute all that traffic back at the attackers.

I knew this would be memcache before I even opened TFA. I have some time before my next meeting so let me sketch some quick diagrams.

[img.fark.net image 794x308]

The attacker is faking their IP address so that when the memcache server responds, it responds to the target site (and the response is much larger than the request, so it's easy to get high magnification numbers with this type of attack).

The network the memcache server is on can be configured to detect and reject packets that have been spoofed.

BUUUUUUT, memcache servers aren't supposed to be exposed to the internet in the first place. So the very presence of a publically reachable memcache server is an indication of an IT department that has no clue what they're doing.

A question to show my ignorance. The hell is memcache?

/don't say some kind of memory cache :)


Pretty sure it's where all memes are stored until needed.
 
2018-03-02 05:04:23 PM  

Gubbo: A question to show my ignorance. The hell is memcache?


memcache daemon (memcached).
 
2018-03-02 05:08:58 PM  

lordargent: BUUUUUUT, memcache servers aren't supposed to be exposed to the internet in the first place. So the very presence of a publically reachable memcache server is an indication of an IT department that has no clue what they're doing.


From the Farking Article:

These database caching systems work to speed networks and websites, but they aren't meant to be exposed on the public internet; anyone can query them, and they'll likewise respond to anyone. About 100,000 memcached servers, mostly owned by businesses and other institutions, currently sit exposed online with no authentication protection, meaning an attacker can access them, and send them a special command packet that the server will respond to with a much larger reply.

Good luck getting one hundred thousand dumbass IT managers to fix their shiat.
 
2018-03-02 05:12:34 PM  

SansNeural: Gubbo: A question to show my ignorance. The hell is memcache?

memcache daemon (memcached).


I'll be honest, this isn't helpful (I have some tech knowledge, but not this)

What is Memcached?
Free & open source, high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.
Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, or page rendering.
 
2018-03-02 05:14:10 PM  

Gubbo: A question to show my ignorance. The hell is memcache?
/don't say some kind of memory cache :)


In this case 'some kind of memory cache' is the best description of it.

"Memcached is a general-purpose distributed memory caching system. It is often used to speed up dynamic database-driven websites by caching data and objects in RAM to reduce the number of times an external data source must be read." - Wikipedia
 
2018-03-02 05:16:39 PM  
 
2018-03-02 05:35:04 PM  

Gubbo: SansNeural: Gubbo: A question to show my ignorance. The hell is memcache?

memcache daemon (memcached).

I'll be honest, this isn't helpful (I have some tech knowledge, but not this)

What is Memcached?
Free & open source, high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.
Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, or page rendering.


OK.  So when the world-wide-web started being a Big Thing and bandwidth was much more expensive than it is now, software like squid (squid-cache) were invented to help thin the incoming load by caching web server requests.  Let's say your community college back in the day only had a T1 (1.53 Mbps) connection and 1/2 of the students hit Yahoo or MSN several times every day.  Squid will cache commonly accessed web stuff, main pages and so on, and reduce overall Internet connection bandwidth by handing them directly to the user on the local network from its local cache.

These days with Big Data having become a Big Thing... in this case, bandwidth is cheaper so we put our daily-use databases on the Internet.  Our big data might be at work and we need it on the road, or it's in the Cloud and we need it anytime.  memcached is like squid, but kinda turned around.  It mostly relieves server CPU and storage demands from repetitive queries and data accesses.

Not sure if you're familiar with databases, but imagine everyone using the company database always pulls from the "basic info" table, no matter what other tables they may or may not be using.  If a database+web server system is going to blindly execute every access request, then maybe 80% of all accesses to the database, all the way down to the harddrive, will be to the "basic info" table while the remaining 20% is the stuff the users are actually working with.  What if you could automatically cache the "basic info" table (or huge chunks of it)?  Now it's closer to 100% of the low-down database accesses are for the stuff the users actually need.
 
2018-03-02 05:41:49 PM  
/not sure my understanding of actual memcached usage is accurate, but the concept explained should be mostly right
 
2018-03-02 05:51:18 PM  

SansNeural: Gubbo: SansNeural: Gubbo: A question to show my ignorance. The hell is memcache?

memcache daemon (memcached).

I'll be honest, this isn't helpful (I have some tech knowledge, but not this)

What is Memcached?
Free & open source, high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.
Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, or page rendering.

OK.  So when the world-wide-web started being a Big Thing and bandwidth was much more expensive than it is now, software like squid (squid-cache) were invented to help thin the incoming load by caching web server requests.  Let's say your community college back in the day only had a T1 (1.53 Mbps) connection and 1/2 of the students hit Yahoo or MSN several times every day.  Squid will cache commonly accessed web stuff, main pages and so on, and reduce overall Internet connection bandwidth by handing them directly to the user on the local network from its local cache.

These days with Big Data having become a Big Thing... in this case, bandwidth is cheaper so we put our daily-use databases on the Internet.  Our big data might be at work and we need it on the road, or it's in the Cloud and we need it anytime.  memcached is like squid, but kinda turned around.  It mostly relieves server CPU and storage demands from repetitive queries and data accesses.

Not sure if you're familiar with databases, but imagine everyone using the company database always pulls from the "basic info" table, no matter what other tables they may or may not be using.  If a database+web server system is going to blindly execute every access request, then maybe 80% of all accesses to the database, all the way down to the harddrive, will be to the "basic info" table while the remaining 20% is the stuff the users are actually working with.  What if you could automatically cache the "basic info" table (or huge chunks of it)?  Now it's closer to 100% of the low-down database accesses are for the stuff the users actually need.


I think I have the gist of it
 
2018-03-02 05:58:16 PM  
"So I would have been certain that we could handle 1.3 Tbps, but at the same time we never had a terabit and a half come in all at once. It's one thing to have the confidence. It's another thing to see it actually play out how you'd hope."

Akamai crushed its enemies, saw them driven before it, and heard the lamentations of their servers. I mean, damn.
 
2018-03-02 05:59:00 PM  

FrancoFile: OldJames: FrancoFile: I find it bizarre and nearly criminally negligent that people would leave memcache servers exposed like that.  Put them in your DMZ for crying out loud.

Isn't the DMZ specifically unprotected?

No.

That would just be "outside the firewall".

The DMZ is a zone that's set up to provide external access to selected internal assets.  Like the public-facing web server, or an API interface, or a VPN server.  It's very straightforward to provide a moderate level of access control (you can do straightforward port filtering and redirecting with an inexpensive boundary router, for example) without choking off throughput.


"Perimeter network." No one uses "DMZ" any more, as it's a culturally charged term in Asian markets.
 
2018-03-02 06:09:56 PM  

dready zim: I am going to guess they survived by being awesome. Because Github is awesome.


Hey that's what I was going to say!!!
 
2018-03-02 06:14:51 PM  

FrancoFile: I find it bizarre and nearly criminally negligent that people would leave memcache servers exposed like that.  Put them in your DMZ for crying out loud.


Put them in a fully protected private network.  They don't need any connectivity with the public internet, the just need connectivity with the application servers that need connectivity with the public internet.
 
2018-03-02 06:15:53 PM  

Geotpf: lordargent: BUUUUUUT, memcache servers aren't supposed to be exposed to the internet in the first place. So the very presence of a publically reachable memcache server is an indication of an IT department that has no clue what they're doing.

From the Farking Article:

These database caching systems work to speed networks and websites, but they aren't meant to be exposed on the public internet; anyone can query them, and they'll likewise respond to anyone. About 100,000 memcached servers, mostly owned by businesses and other institutions, currently sit exposed online with no authentication protection, meaning an attacker can access them, and send them a special command packet that the server will respond to with a much larger reply.

Good luck getting one hundred thousand dumbass IT managers to fix their shiat.


Allow the attacked company to sue for damages from other companies whose open memcahe servers are leveraged for an attack?

Ok, not a very feasible idea in an international internet, but still... Maybe a bit of imposed liability for incompitance is not always a bad thing.
 
2018-03-02 06:19:12 PM  
Of course, this article *would* be posted on the same day that Atlassian had a major outage that made our Bitbucket repos inaccessible for a good chunk of the day...

/it had cheaper private repos at our team size a few years ago
 
2018-03-02 06:33:39 PM  

poot_rootbeer: Of course, this article *would* be posted on the same day that Atlassian had a major outage that made our Bitbucket repos inaccessible for a good chunk of the day...

/it had cheaper private repos at our team size a few years ago


Yeah, well, you know... Github offers enterprise service that comes with their same awesome reliability ;)
 
Displayed 50 of 54 comments


Oldest | « | 1 | 2 | » | Newest | Show all


View Voting Results: Smartest and Funniest

This thread is archived, and closed to new comments.

Continue Farking





On Twitter



Top Commented
Javascript is required to view headlines in widget.
  1. Links are submitted by members of the Fark community.

  2. When community members submit a link, they also write a custom headline for the story.

  3. Other Farkers comment on the links. This is the number of comments. Click here to read them.

  4. Click here to submit a link.

Report