Same reason I don't give out my real phone number unless I know how someone handles their phone. Doesn't take any malicious intent to grant Contacts access to all kinds of malicious apps. Other peoples' computers have to be assumed hostile.