If you can read this, either the style sheet didn't load or you have an older browser that doesn't support style sheets. Try clearing your browser cache and refreshing the page.

(Mother Jones)   Metadata schmetadata. The NSA is actually recording every call made every day   (motherjones.com) divider line 382
    More: Followup, NSA, NSA Revelations, international call  
•       •       •

4808 clicks; posted to Politics » on 02 Jul 2013 at 7:10 PM (1 year ago)   |  Favorite    |   share:  Share on Twitter share via Email Share on Facebook   more»



382 Comments   (+0 »)
   
View Voting Results: Smartest and Funniest

Archived thread

First | « | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | » | Last | Show all
 
2013-07-02 07:46:19 PM

LewDux: unlikely: This is simply technically impossible.

I'm not saying they wouldn't if they could, but the technology does not exist.


oi44.tinypic.com
 
2013-07-02 07:46:30 PM

MrEricSir: RexTalionis: Seriously, I don't see how any reasonable person can read that slide and think "OMG, they're recording all of my emails and chat messages."

Well, you could read the rest of the slide you posted and notice items B and E on the list are IMs and email, respectively.


And you can tell me for sure that a slide about "case notation" definitively shows that they are storing everything and not metadata? Because there's still nothing on that slide that indicates it.
 
2013-07-02 07:47:08 PM

Shotgun Justice: LasersHurt: Shotgun Justice: They probably can't store actual phone calls because the data is too great. However, they could process them with speech to text programs - kinda like a teletype system that would store the phone call in plain text format. Assuming they have good deciphering algorithms they have the gist of the call on file without having to actually save audio.

This is ridiculous, and presumes that there isn't enough storage (this is very wrong) but there is more than enough computing power to accurately transcribe every call (this is also wrong).

Try recording raw audio data and see what type of file size you get.  To compress it down would require much more processing power.  Not to mention organizing the data and drawing out relevant keywords.


You're out of your element here. For one, you can record the calls in their ALREADY COMPRESSED FORMAT if you're intercepting calls.

I work for one of the largest VoIP companies out there, I'm slightly familiar here.
 
2013-07-02 07:47:21 PM

mjjt: What is the comparison? - the best known compression is ? times better than mp3?


I think phone conversations use a lot less bandwidth than recorded music, for starters.  Anyone recall how many kHz of bandwidth they used per phone line in the days when landlines were still analog?
 
2013-07-02 07:47:41 PM
I lost the # of that hot chick I met in Boulder 6 years ago, who do I call to get that information ?
 
2013-07-02 07:49:43 PM
Anyone got any evidence that any of it is being accessed without a warrant?

Still no? Ok. Forgive me if I hold my outrage.
 
2013-07-02 07:51:33 PM

Shotgun Justice: They probably can't store actual phone calls because the data is too great. However, they could process them with speech to text programs - kinda like a teletype system that would store the phone call in plain text format. Assuming they have good deciphering algorithms they have the gist of the call on file without having to actually save audio.


You're thinking like a person, and it's wasting processing power... instead of translating waveforms to text, then comparing text to the list of acceptable texts, it would be easier to just look for similar waveforms. That said, we're still talking about an inordinate amount of processing power, and substantively good programming and creative thinking that I doubt the NSA possesses.
 
2013-07-02 07:53:19 PM

firefly212: Shotgun Justice: They probably can't store actual phone calls because the data is too great. However, they could process them with speech to text programs - kinda like a teletype system that would store the phone call in plain text format. Assuming they have good deciphering algorithms they have the gist of the call on file without having to actually save audio.

You're thinking like a person, and it's wasting processing power... instead of translating waveforms to text, then comparing text to the list of acceptable texts, it would be easier to just look for similar waveforms. That said, we're still talking about an inordinate amount of processing power, and substantively good programming and creative thinking that I doubt the NSA possesses.


The NSA is contracting the dirty linguistic work to American Universities of various kinds. At least, the research and development thereof.
 
2013-07-02 07:54:38 PM

mjjt: Can any of you come up with a guess as to how much data storage a billion phone calls would require?

All I can think of is my mp3 files - a 3 min song is about 4MB

assume an average call also 3 min?

As pointed out above, I realize their compression tech will be slightly better than mp3

What is the comparison? - the best known compression is ? times better than mp3?


GSM calls are 16kbps, I believe CDMA is similar.  Much, much more compressed than music since there is a limited frequency range you care about.
 
2013-07-02 07:56:52 PM

Evil High Priest: intercept call.
translate to text file.
keep forever.


"I'm going to put the bum on the plan that will land in the basin street airport at behind gerty."
 
2013-07-02 07:57:14 PM

Nobodyn0se: Anyone got any evidence that any of it is being accessed without a warrant?

Still no? Ok. Forgive me if I hold my outrage.


Usually the burden of proof lies with the party attempting to justify their questionable activities, not the victims of that activity.


But for the sake or argument, lets say that a FISA court has secretly issued a warrant for every bit of information the NSA is gathering.


What would be the probable cause to support that warrant?
 
2013-07-02 07:59:27 PM

1derful: Nobodyn0se: Anyone got any evidence that any of it is being accessed without a warrant?

Still no? Ok. Forgive me if I hold my outrage.

Usually the burden of proof lies with the party attempting to justify their questionable activities, not the victims of that activity.


But for the sake or argument, lets say that a FISA court has secretly issued a warrant for every bit of information the NSA is gathering.


What would be the probable cause to support that warrant?


It would depend on which warrant you were talking about. Every time they want to actually use that data, they have to get a warrant for the particular information they want to access. That has happened multiple times for multiple different reasons.
 
2013-07-02 08:03:50 PM

firefly212: Thus far, the keywords released have indicated that they could have their data mining operation alert someone if there was a major event happening, but none of it is indicative of an ability to foresee events prior to occurrence.


Also, if such an event did happen, wouldn't all the communications between people using the keywords cause so much clutter for days/weeks months as to render the keywords unusable?

I'm sure that after Boston for example, use of the words 'bomb', chechnya, terrorism etc in disucssions between people would've skyrocketed...filtering out meaningful comms would be almost impossible.
 
2013-07-02 08:05:28 PM

cannotsuggestaname: There is a reason they have 5 ZB in SLC and another 2.5 in San Antonio. They didn't just put that crazy amount of storage capability out there because ZOMG SO COOL!

They did it because of intercepts.



They have that much storage space for their awesome Netflix subscription, obviously.

Look, they collect all the metadata because it's the indexing system for the phone calls.  It's the card catalog, which allows them to find and pull the content of the recorded phone calls.

Everything is being captured.  All of it, all the time.
 
2013-07-02 08:05:51 PM

LasersHurt: Shotgun Justice: LasersHurt: Shotgun Justice: They probably can't store actual phone calls because the data is too great. However, they could process them with speech to text programs - kinda like a teletype system that would store the phone call in plain text format. Assuming they have good deciphering algorithms they have the gist of the call on file without having to actually save audio.

This is ridiculous, and presumes that there isn't enough storage (this is very wrong) but there is more than enough computing power to accurately transcribe every call (this is also wrong).

Try recording raw audio data and see what type of file size you get.  To compress it down would require much more processing power.  Not to mention organizing the data and drawing out relevant keywords.

You're out of your element here. For one, you can record the calls in their ALREADY COMPRESSED FORMAT if you're intercepting calls.

I work for one of the largest VoIP companies out there, I'm slightly familiar here.


I'm sorry I didn't know I was speaking to the mighty mind of the help desk worker at Vonage.  My apologies.  However, VoIP transmits differently than traditional land lines.  VoIP is compressed and is processed by both the transmitter and receiving end to prevent jitter, disguise packet loss, etc....You don't have that type of need with a land line.  Your phone doesn't process it, and the receivers phone doesn't have to compensate.
 
2013-07-02 08:05:59 PM

Nobodyn0se: Anyone got any evidence that any of it is being accessed without a warrant?

Still no? Ok. Forgive me if I hold my outrage.


Apparently you missed the part about how there's a fake court system setup with the express purpose of issuing fake warrants for the NSA.

Using the language of due process doesn't mean due process is in action any more than adding "Democratic" to their country's name made North Korea into a democracy.
 
2013-07-02 08:06:55 PM

Elmo Jones: Bullsh*t!
There is no way that lazy government workers could come up with technology, before the holy Private Sector!
That's what they'd need to store the exabytes for all the phone calls, texts, e-mails, and associated metadata.


Gov workers don't design, they buy. From secret contracts with places like skunkworks....

As stated in several places here, its not the storage, its getting it there. You can have the biggest pool in the world but imagine the time it takes to fill it unless you have unlimited garden hoses, no one else uses water, all the fire trucks in town come, it rains, everyone spits.... But I digress
 
2013-07-02 08:09:31 PM

MrEricSir: Apparently you missed the part about how there's a fake court system setup with the express purpose of issuing fake warrants for the NSA.


Apparently you missed the part where that wasn't true. If there was a rubber stamp process for those warrants, there would be tens of thousands of warrants issued every year. As there's a TINY fraction of that number being issued, we can assume that the FISA court is doing its job and holding law enforcement accountable.
 
2013-07-02 08:10:30 PM
Why do people assume they are collecting calls so that they can datamine them?

By having a complete record if a "person of interest" crops up then his calls from the past are in the DB and you only need a targeted search to try to find them and listen to them.
 
2013-07-02 08:11:05 PM
PF: Fark
 
2013-07-02 08:12:06 PM

Shotgun Justice: I'm sorry I didn't know I was speaking to the mighty mind of the help desk worker at Vonage.


Don't get all butthurt because you don't know what you're talking about and got called out on it. Also I didn't say VoIP PROVIDER. I don't work for Vonage.
 
2013-07-02 08:14:57 PM

LasersHurt: It's not just storage space, dude. It's ALSO the combined amount of power to intercept, decode, record, and then transmit that all to a datacenter. It's a HUGE undertaking.


Then it sure is a good thing the telecom providers would never give them direct access to make it all easy.
 
2013-07-02 08:16:57 PM
You'd think we would have heard about all the cases that got thrown out because of illegal wiretaps. Or of people going to jail because of illegal wiretaps. Or of people being convicted because of legal wiretaps. Something. I'm hearing crickets.
 
2013-07-02 08:17:37 PM

thurstonxhowell: Yup. Collecting every phone call ever made would render the data useless. You'd be searching for a needle in a universe.


But you have the search backwards.

We just found out person X is suspicious, thanks to a report from someone at their church or mosque or a relative or whatever. Or because they were arrested as part of a plot, whatever.

Now we can go back and listen to all of their calls.  All of the calls their first and second level contacts made.

That's the purpose: it's a wiretapping time machine.  There are quotes from authorities that strongly indicate this happened with the Boston bombers.
 
2013-07-02 08:17:50 PM

HairBolus: Why do people assume they are collecting calls so that they can datamine them?

By having a complete record if a "person of interest" crops up then his calls from the past are in the DB and you only need a targeted search to try to find them and listen to them.


This is how they caught the Boston Bombers.

They get a warrant to pull information on a specific individual.

If they collect this information about everybody then your Fourth Amendment Rights aren't being violated.
 
2013-07-02 08:18:27 PM

Nobodyn0se: MrEricSir: Apparently you missed the part about how there's a fake court system setup with the express purpose of issuing fake warrants for the NSA.

Apparently you missed the part where that wasn't true. If there was a rubber stamp process for those warrants, there would be tens of thousands of warrants issued every year. As there's a TINY fraction of that number being issued, we can assume that the FISA court is doing its job and holding law enforcement accountable.


Right, because why wouldn't you trust the government that's been hiding a massive spying program to be completely honest with you about the fake court system they operate in secret? Oh wait...
 
2013-07-02 08:19:19 PM

unlikely: This is simply technically impossible.

I'm not saying they wouldn't if they could, but the technology does not exist.


What parts don't exist?  And be specific.
 
2013-07-02 08:21:08 PM

MrEricSir: Right, because why wouldn't you trust the government that's been hiding a massive spying program to be completely honest with you about the fake court system they operate in secret? Oh wait...


1. They haven't really done a bangup job of hiding this program, considering it's been all over the news for years now.

2. That "fake court system" was set up before most Farkers were even born. It's not anything new.

3. I will trust the evidence I have until I find evidence that contradicts it. So if you have some evidence that ANYONE is viewing this data without a warrant, or that the warrant process is less than ethical, I'd love to see it. Until then, I'll trust the evidence I have that warrants are required and not easy to get.
 
2013-07-02 08:22:03 PM

Shotgun Justice: LasersHurt: Shotgun Justice: LasersHurt: Shotgun Justice: They probably can't store actual phone calls because the data is too great. However, they could process them with speech to text programs - kinda like a teletype system that would store the phone call in plain text format. Assuming they have good deciphering algorithms they have the gist of the call on file without having to actually save audio.

This is ridiculous, and presumes that there isn't enough storage (this is very wrong) but there is more than enough computing power to accurately transcribe every call (this is also wrong).

Try recording raw audio data and see what type of file size you get.  To compress it down would require much more processing power.  Not to mention organizing the data and drawing out relevant keywords.

You're out of your element here. For one, you can record the calls in their ALREADY COMPRESSED FORMAT if you're intercepting calls.

I work for one of the largest VoIP companies out there, I'm slightly familiar here.

I'm sorry I didn't know I was speaking to the mighty mind of the help desk worker at Vonage.  My apologies.  However, VoIP transmits differently than traditional land lines.  VoIP is compressed and is processed by both the transmitter and receiving end to prevent jitter, disguise packet loss, etc....You don't have that type of need with a land line.  Your phone doesn't process it, and the receivers phone doesn't have to compensate.


I'm willing to bet just about every land line is hitting an IP network at some stage of the game now, unless it never leaves the LEC.  And picking out compressed streams and applying the right codec doesn't seem difficult.

I doubt the NSA is recording all phone calls, it seems logistically hard to hide but...
 
2013-07-02 08:22:32 PM
Oh, is this the new game?  Publish things that have been common knowledge for 7 years and pretend they just started, recently, under Obama?
 
2013-07-02 08:22:51 PM

Marcus Aurelius: unlikely: This is simply technically impossible.

I'm not saying they wouldn't if they could, but the technology does not exist.

What parts don't exist?  And be specific.


Because if individual parts exist, they can be scaled infinitely without issue?
 
2013-07-02 08:22:57 PM
Oh no! I just called my mother and told her I couldn't meet her because I had the runs. And you're telling me they heard all that?
 
2013-07-02 08:26:06 PM

DustBunny: They might have built a giant data centre, it might have a lot of storage, but I'm pretty confident that the software and hardware in it is not more advance than that running Facebook or Google or the phone company itself


It's all geared toward identification and interception of the identified subject.  I doubt Aunt Ida's recipe for potato salad spends more than a millisecond on their server.  An exchange student from Saudi however might be an entirely different kettle of fish.  They might get the whole boat - cell phone, facebook, google, the works.

Then it comes down to how many subject you can track at once.  I'd guess it's well over a million by now.
 
2013-07-02 08:28:07 PM

1derful: Usually the burden of proof lies with the party attempting to justify their questionable activities, not the victims of that activity.


But for the sake or argument, lets say that a FISA court has secretly issued a warrant for every bit of information the NSA is gathering.


What would be the probable cause to support that warrant?



Actually, the burden of proof tends to fall to the person making wild, unfounded accusations without presenting any evidence.
 
2013-07-02 08:28:34 PM

PreMortem: whistleridge: Horseshiat.

RexTalionis: BS:


Shortly after the F-117 (stealth fighter) was unveiled, I read an interview (Aviation Leek i believe) with a VP of Northrup Grumman who said what is classified is 50 years ahead of what is public. If you think the NSA/CIA is using off the shelf gear or anything you can imagine, well you just shouldn't.


I recall from my short stint in the intel business that what the general public thinks is the world's most powerful supercomputer (Crays back then) is actually one generation behind the hardware running at NSA headquarters.
 
2013-07-02 08:29:28 PM

Mistymtnhop: Oh no! I just called my mother and told her I couldn't meet her because I had the runs. And you're telling me they heard all that?


Your government-issued pepto should appear at your front door within the hour, citizen.
 
2013-07-02 08:32:04 PM

Elmo Jones: Evil High Priest: intercept call.
translate to text file.
keep forever.

"I'm going to put the bum on the plan that will land in the basin street airport at behind gerty."


"Why yes, I would like the empty on my plantains"
 
2013-07-02 08:32:45 PM

cannotsuggestaname: There is a reason they have 5 ZB in SLC and another 2.5 in San Antonio. They didn't just put that crazy amount of storage capability out there because ZOMG SO COOL!

They did it because of intercepts.


Understood, but there are qualifiers missing.  Before you start claiming they can capture everything twice, you need to think about how long to store shiat so it can be at least lightly scanned.  Turns out, the data just keeps coming in.  So, maybe not everything.  Also, it's rather remarkable how much metadata gets generated on really big transactional databases.  Ask our friends at Google, it's  rather remarkable the hoops you need to jump through when you're dealing with data on these scales.

It's not going to be easy to have a rational discussion about whether or not the NSA is too intrusive if we don't really know how intrusive they are.  This article is not a good starting point.

Cheers.

//Never built anything transactional larger than 1 TB
//Bloody datawarehouses at 100TB are a major pain
 
2013-07-02 08:36:35 PM
Yeah, joining in on the "They would if they could, but they probably can't".

Does NSA have a culture that prizes knowing everything about everyone everywhere coupled with insane paranoia that would make them think this is a good idea?  Yeah.
Is is technically possible that they could record and store every phone call ever made, especially if they got them pre-compressed from the phone companies?  Yeah, it's mostly a matter of storage space and they've got a 5ZB datacenter.
Is it technically possible that they are storing "Person #175638 called Person #1875667 and they talked for 23 minutes and here's the recording if we need it."?  Yeah.
Is it technically possible that they can use other techniques (ala: http://kieranhealy.org/blog/archives/2013/06/09/using-metadata-to-fin d -paul-revere/) to find "calls of interest" and then run analysis on that exceedingly limited dataset?  Sure.

Is is technically possible that they could with 100% (or even 90%) accuracy transcribe all that data and do all the connecting, sorting, keyword searching, etc?  Not really.

It's not even a matter of processing power.  It's a matter of not knowing how to do it at all (barring NSA's X year advantage in all things tech having discovered some tremendous breakthrough).  It takes a TON of processing power to parse voice.  And even then, it's pretty bad.  So if you've got a .0001% false positive rate (and we'd kill for a 10% error rate last I heard) on 5ZB of data, then you've got to manually parse 5 exabytes of garbage.  And that's impossible.  We can't parse everything.
 
2013-07-02 08:37:04 PM

Nobodyn0se: Anyone got any evidence that any of it is being accessed without a warrant?

Still no? Ok. Forgive me if I hold my outrage.


How about collected and stored? Interesting you specifically chose the word accessed, instead of ya know search.
 
2013-07-02 08:37:08 PM
www.charlock.org
 
2013-07-02 08:37:38 PM
And we're supposed to believe that NOBODY is using this data to get inordinate rich?
 
2013-07-02 08:37:50 PM

Nobodyn0se: MrEricSir: Right, because why wouldn't you trust the government that's been hiding a massive spying program to be completely honest with you about the fake court system they operate in secret? Oh wait...

1. They haven't really done a bangup job of hiding this program, considering it's been all over the news for years now.

2. That "fake court system" was set up before most Farkers were even born. It's not anything new.

3. I will trust the evidence I have until I find evidence that contradicts it. So if you have some evidence that ANYONE is viewing this data without a warrant, or that the warrant process is less than ethical, I'd love to see it. Until then, I'll trust the evidence I have that warrants are required and not easy to get.


Um, WTF? You will never have evidence. Period. The government is hiding the evidence behind a fake court system for a reason.

You seem to be bending over backwards to come across as sounding reasonable, but what you're essentially saying is that you have 100% trust in a group of people who openly admit to deceiving you. That is not a reasonable position.
 
2013-07-02 08:38:04 PM

Marcus Aurelius: It's all geared toward identification and interception of the identified subject.  I doubt Aunt Ida's recipe for potato salad spends more than a millisecond on their server.  An exchange student from Saudi however might be an entirely different kettle of fish.  They might get the whole boat - cell phone, facebook, google, the works.

Then it comes down to how many subject you can track at once.  I'd guess it's well over a million by now.


That's fine, I have no problem with believing that...It's the whole 'every single phone call and piece of data including Aunt Ida's recipe for potato salad is hooverd up and stored forever so it can be indexed and searched' just in case Aunt Ida ever becomes a suspect that I have issues with...

Also the calculations being done to show that the data centre they're building has enough storage to capture and store all the current phone calls etc, are being done with no thought for future needs.... If the NSA was building such a massive data centre with the ability to store one or two years worth of data at the current production rate, then that's a MASSIVE fail on their part and shows such disregard for future storage needs that it practically invalidates all claims as to their competence.

IF the site has, say, 5zb of storage and the phone calls and date people say it will store add up to 2-3zb per year, and everyone says that's evidence they're doing it then doesn't that mean they'll have to have a new DC online in, say, 2015/16?

Considering how long it takes to build one, can anyone point to where they've broken ground on this new bespoke datacentre?
 
2013-07-02 08:38:16 PM
No chance. This would require yattabyte storage farms, which would take up more space than a small city and cost more money than the entire value of all the assets in the United States. The entire internet doesn't even have as much data on it as the phone calls of every American citizen made every year.
 
2013-07-02 08:40:56 PM
Hurry... Blame everyone except for Obama.  Everyone knows he's pretty much the best president ever yet somehow knows absolutely nothing about anything actually happening inside of the government.
 
2013-07-02 08:42:08 PM

whistleridge: 99.99% of those calls are utterly useless to anyone - hairdressers ordering pizza,


They won't listen to 99.99999% percent of the calls. But yes they want to record pizza orders.

1) If you are tracking someone then a pizza delivery order can tell you where he was.

2) If you know the guy had pizza delivered then the phone call can tell you what phone he used to order it.
 
2013-07-02 08:42:44 PM

PreMortem: whistleridge: Horseshiat.

RexTalionis: BS:


Shortly after the F-117 (stealth fighter) was unveiled, I read an interview (Aviation Leek i believe) with a VP of Northrup Grumman who said what is classified is 50 years ahead of what is public. If you think the NSA/CIA is using off the shelf gear or anything you can imagine, well you just shouldn't.


You know who invents that kind of stuff? Scientists. It takes huge teams of incredibly well educated people, the kind of people who would eventually contain at least one member that would go public. Classified state technology can only stay a few years ahead of civilian tech, especially in the modern information age.
 
2013-07-02 08:42:58 PM
I, for one, will trust and defend the guys who keep lying to us about all of this. It's legal! What more do you want?
 
2013-07-02 08:44:52 PM

Ontos: Hurry... Blame everyone except for Obama.  Everyone knows he's pretty much the best president ever yet somehow knows absolutely nothing about anything actually happening inside of the government.


Yes, Obama is clearly responsible for this program that was first made public 2 years before he took office.....
 
Displayed 50 of 382 comments

First | « | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | » | Last | Show all

View Voting Results: Smartest and Funniest


This thread is archived, and closed to new comments.

Continue Farking
Submit a Link »






Report