Yep, this is an endemic problem with end-to-end verifiable elections everywhere: it is easy to invalidate whole segments of votes. For example, in the wombat voting scheme, if one of the people who holds a decryption key refuses to divulge it, perhaps by "losing" their key, they can invalidate the whole election. Typically to minimize the chance of this happening you make sure that everyone with a key has a strong commitment to the election and make not divulging the key a public shaming event. "OMG, the Democratic Party refuses to decrypt the vote in this Republican-dominated area -- they must be corrupt!"
It's an awfully sticky situation, because the A (availability) in the CIA principle is so critical in elections, where it typically is compromised the most in encryption schemes. (i.e. Quantum Key Distribution ensures that Eve can't listen in on communication, but doesn't do much to stop Eve from making communication impossible).
Wombat is using threshold cryptography. Threshold cryptosystem, works like this (in short), in order to decrypt an encrypted message a number of parties exceeding a threshold is required to cooperate in the decryption protocol. Meaning, before the election has started a threshold is set, let's say there are 4 candidates, you can agree that 3 of the candidates is enough to decrypt the votes.
Also... Don't forget the paper backup, you can simply count those.
It's an awfully sticky situation, because the A (availability) in the CIA principle is so critical in elections, where it typically is compromised the most in encryption schemes. (i.e. Quantum Key Distribution ensures that Eve can't listen in on communication, but doesn't do much to stop Eve from making communication impossible).