This bit both somewhat limits the impact and makes Greenwald et. al.'s claims that most everything is being Hoovered up a lot more credible:
"The XKeyscore system is continuously collecting so much internet data that it can be stored only for short periods of time. Content remains on the system for only three to five days, while metadata is stored for 30 days. One document explains: "At some sites, the amount of data we receive per day (20+ terabytes) can only be stored for as little as 24 hours.""
Of course, as the article goes on to detail, anything that's found to be of interest in that window can be saved permanently, and NSA analysis do that a lot.
>"To solve this problem, the NSA has created a multi-tiered system that allows analysts to store "interesting" content in other databases, such as one named Pinwale which can store material for up to five years."
It does not describe "interesting". Maybe metadata, encrypted sessions etc? Any conversations in threads linking to these articles?
...and so have the compression and filtering tech, I'm sure. If you strip out most attachments, the average email message is incredibly tiny when gzipped.
Tempora in the UK stores all internet data for 3 days and metadata for 30 days too. Wouldn't surprise me if most western governments use XKeyscore along with data sharing agreements with each other.
I would imagine that their top priority since 2008 has been improving storage systems. It's clear that their ultimate goal is a fully indexed archive of the entire digital universe.
"The XKeyscore system is continuously collecting so much internet data that it can be stored only for short periods of time. Content remains on the system for only three to five days, while metadata is stored for 30 days. One document explains: "At some sites, the amount of data we receive per day (20+ terabytes) can only be stored for as little as 24 hours.""
Of course, as the article goes on to detail, anything that's found to be of interest in that window can be saved permanently, and NSA analysis do that a lot.