At the end of last week, Hetzner technicians discovered a "backdoor" in one
of our internal monitoring systems (Nagios).
An investigation was launched immediately and showed that the administration
interface for dedicated root servers (Robot) had also been affected. Current
findings would suggest that fragments of our client database had been copied
externally.
As a result, we currently have to consider the client data stored in our Robot
as compromised.
To our knowledge, the malicious program that we have discovered is as yet
unknown and has never appeared before.
The malicious code used in the "backdoor" exclusively infects the RAM. First
analysis suggests that the malicious code directly infiltrates running Apache
and sshd processes. Here, the infection neither modifies the binaries of the
service which has been compromised, nor does it restart the service which has
been affected.
The standard techniques used for analysis such as the examination of checksum
or tools such as "rkhunter" are therefore not able to track down the malicious
code.
We have commissioned an external security company with a detailed analysis of
the incident to support our in-house administrators. At this stage, analysis
of the incident has not yet been completed.
The access passwords for your Robot client account are stored in our database
as Hash (SHA256) with salt. As a precaution, we recommend that you change your
client passwords in the Robot.
With credit cards, only the last three digits of the card number, the card type
and the expiry date are saved in our systems. All other card data is saved
solely by our payment service provider and referenced via a pseudo card number.
Therefore, as far as we are aware, credit card data has not been compromised.
Hetzner technicians are permanently working on localising and preventing possible
security vulnerabilities as well as ensuring that our systems and infrastructure
are kept as safe as possible. Data security is a very high priority for us. To
expedite clarification further, we have reported this incident to the data
security authority concerned.
Furthermore, we are in contact with the Federal Criminal Police Office (BKA) in
regard to this incident.
Naturally, we shall inform you of new developments immediately.
We very much regret this incident and thank you for your understanding and
trust in us.
> I just wish that they used bcrypt instead of a salted SHA256 but at least it's salted (and anyway in my case I never reuse passwords so no big deal).
If they fall under PCI:DSS, they might not be able to use bcrypt since it isn't an official recommended standard. (I am of course assuming that they mean PBKDF2 when they say salted SHA256.)
Thanks for that link. I think that means blowfish is approved as a standard encryption algorithm, but that doesn't mean it is approved as a standard way to store passwords.
> For any business required to comply with U.S. NIST or FIPS standards, bcrypt is not a valid option. Check every nation's laws and regulations separately if you do business there, of course.
> As far as I know, PCI doesn't specify actual technology, especially Encryption specifications. The legal cases I've seen support what's technically feasible, not technically ideal or perfect. I'm sure either one [bcrypt or PBKDF2] meets PCI requirements
An opinion shared by many people that deal with PCI, is that you follow PCI not to secure your systems, but to limit your exposure to PCI fines and remedies. It seems like it is a rock-solid defense for PCI compliance to choose the algorithm that is certified by NIST.
But it is also likewise plausible to say that you're using a newer technology that's understood by experts in the industry to be even better than those certified some time ago.
I am not a Hetzner customer, but the comment about Nagios perked up my ears. I work with some clients who have Nagios running in their environment, and I'm wondering if there is an exploit in Nagios or if it's just a coincidence that this was where they noticed the infection?
I'm kinda confused about why Nagios was even involved here. If you've got all your checks setup properly, the Nagios host doesn't have any write access to your monitored hosts. You can use NRPE to restrict what commands it runs. Combine that with a read only SNMP account, and even if your Nagios server is compromised it cannot access your server.
It makes me wonder if there's any connection with the recent Drupal Security problem (they cited a "third-party software installed on the Drupal.org server infrastructure" but they haven't - afaik - disclosed the software name yet)
I suspect it's simply out of date. Nagios does have a large surface area, but from what I've seen in the past, monitoring systems are very difficult for sysadmins to want to upgrade. :)
It's most likely that Nagios was simply used as a tool. It's easy enough to add a Nagios plugin that does something naughty and call it check_inode_usage or something.
..not that I'm a fan of Nagios' architecture though. I'm not.
"The standard techniques used for analysis such as the examination of checksum or tools such as "rkhunter" are therefore not able to track down the malicious code."
We use it as well but I'm not sure it is an example of a serious security practice when given with only one other "checksum or tools such as" type statement.
I can also confirm receiving the email (but I really need to tidy up my mail filters, I actually read about the hack here first).
Nice email, one improvement would be to host the wiki behind a proper https-site -- especially since we already know the attack appears somewhat sophisticated, and that the attackers gained control over servers on the Hetzner network. Now, the faq doesn't contain anything obviously bad (email your credit card info to... etc) -- but one small thing that could be improved.
Other than that I appreciate how Hetzner have handled this so far.
I don't understand, are they saying the backdoor might be because of the Nagios software or because something has implanted itself into the Nagios software?
Dear Client
At the end of last week, Hetzner technicians discovered a "backdoor" in one of our internal monitoring systems (Nagios).
An investigation was launched immediately and showed that the administration interface for dedicated root servers (Robot) had also been affected. Current findings would suggest that fragments of our client database had been copied externally.
As a result, we currently have to consider the client data stored in our Robot as compromised.
To our knowledge, the malicious program that we have discovered is as yet unknown and has never appeared before.
The malicious code used in the "backdoor" exclusively infects the RAM. First analysis suggests that the malicious code directly infiltrates running Apache and sshd processes. Here, the infection neither modifies the binaries of the service which has been compromised, nor does it restart the service which has been affected.
The standard techniques used for analysis such as the examination of checksum or tools such as "rkhunter" are therefore not able to track down the malicious code.
We have commissioned an external security company with a detailed analysis of the incident to support our in-house administrators. At this stage, analysis of the incident has not yet been completed.
The access passwords for your Robot client account are stored in our database as Hash (SHA256) with salt. As a precaution, we recommend that you change your client passwords in the Robot.
With credit cards, only the last three digits of the card number, the card type and the expiry date are saved in our systems. All other card data is saved solely by our payment service provider and referenced via a pseudo card number. Therefore, as far as we are aware, credit card data has not been compromised.
Hetzner technicians are permanently working on localising and preventing possible security vulnerabilities as well as ensuring that our systems and infrastructure are kept as safe as possible. Data security is a very high priority for us. To expedite clarification further, we have reported this incident to the data security authority concerned.
Furthermore, we are in contact with the Federal Criminal Police Office (BKA) in regard to this incident.
Naturally, we shall inform you of new developments immediately.
We very much regret this incident and thank you for your understanding and trust in us.
A special FAQs page has been set up at http://wiki.hetzner.de/index.php/Security_Issue/en to assist you with further enquiries.
Kind regards
Martin Hetzner