The same government that invented the nuclear bomb and provided feedback that protected DES's S boxes from differential cryptanalysis before it was publicly known to exist?
I think you give it too little credit.
The government does a bad job with citizen-facing software engineering. It's not clear that the same is true for defense and intelligence applications.
The difference was that there weren't better jobs for those people back then, and there was a real world war back then and we were kinda obviously the good guys.
People with the skills to do the data mining needed at this scale already have better jobs at Google or on Wall Street. Are they really going to take a pay cut, worse benefits and "government job" bureaucracy for the privilege of spying on their friends and family?
I have family in military intelligence and the situation they are in right now is their branch of the armed service thinks they are just going to send their officers of below average mathematical talent to big data boot camp and expect them to just pick this stuff up, as if it's the same thing as learning Russian or flying a drone. Although the I'm as disappointed as everyone in that so many in the government seem think domestic spying is the wave of the future, I'm skeptical that the real motivations are anything beyond business as usual defense sector pork.
Sure but the big difference today is the internet. Good luck trying to hire an Oppenheimer today and setting up Los Alamos in secret.
Take a look at successful teams that already work with "big data" like at Wolfram Alpha or IBM's Watson. It is not at all easy to build such teams today. There aren't enough PhD's around, unless you are outsourcing to China. And then to step it up a level to Google's requirements we enter personal chefs and segway territory. I cant find the quote but Eric Schmidt has said multiple time what is hampering progress is not the lack of cash or infrastructure but talent.
More than the privacy issues I have to wonder about the waste. Cause getting the data and the infrastructure to handle it is the "easy" part. Extracting actionable intelligence I doubt highly they will succeed. Will turn into a big cash sink, that no one will talk about as the costs need to be justified...until they cant be.
I think you give it too little credit.
The government does a bad job with citizen-facing software engineering. It's not clear that the same is true for defense and intelligence applications.