When you want to make a configuration change to your servers, and they aren't already in a "known state", why would you think any tool could put them into a known state? When a computer has a virus, say, you don't think it's been put into a "known state" after the anti-virus program gets done with it; the virus may have done any number of things you might be unaware of, altered any number of data or configuration files in subtle ways that the tool doesn't look for, but implicitly relies upon.
Example: what if, say, one of the provisioning requirements is "make sure this gem is installed", but on one of the servers /etc/gemrc has "install: --no-rdoc --no-ri" in it? Now on one server, the docs are missing, while everywhere else they're available. That sort of thing.
If you're already running on an IaaS like EC2, I think there's a simpler, better way: rather than trying to get "unknown state" to "known state", why not use the simplest known state of all--unprovisioned? Write an imperative script that reinitializes a freshly-provisioned IaaS node to your known state, and then do a rolling reprovision, terminating old nodes and provisioning new ones.
An equivalent comparison[1]: would you feel safe running an automated script that would SSH into a production machine and "git pull" a checked-out repo sitting on it, from whatever state it happens to be sitting at, up to refs/heads/master, so as to deploy code from it? Could you guarantee that that repo hadn't been moved to some state where refs/heads/master isn't a fast-forward commit? Or would you rather do a fresh "git clone"?
---
[1] A contrived comparison, though; although the former option is unreliable, the latter is terribly inefficient, and they're both horrible for keeping a .git directory inside of a directory that might very well be web-accessible.
Example: what if, say, one of the provisioning requirements is "make sure this gem is installed", but on one of the servers /etc/gemrc has "install: --no-rdoc --no-ri" in it? Now on one server, the docs are missing, while everywhere else they're available. That sort of thing.
If you're already running on an IaaS like EC2, I think there's a simpler, better way: rather than trying to get "unknown state" to "known state", why not use the simplest known state of all--unprovisioned? Write an imperative script that reinitializes a freshly-provisioned IaaS node to your known state, and then do a rolling reprovision, terminating old nodes and provisioning new ones.
An equivalent comparison[1]: would you feel safe running an automated script that would SSH into a production machine and "git pull" a checked-out repo sitting on it, from whatever state it happens to be sitting at, up to refs/heads/master, so as to deploy code from it? Could you guarantee that that repo hadn't been moved to some state where refs/heads/master isn't a fast-forward commit? Or would you rather do a fresh "git clone"?
---
[1] A contrived comparison, though; although the former option is unreliable, the latter is terribly inefficient, and they're both horrible for keeping a .git directory inside of a directory that might very well be web-accessible.