I believe people should be able to downvote whatever they want. People should be allowed to go against the hivemind. PieFed implements pseudonymous downvoting. For anyone worried about spam, they have powerful tools to allow admins to filter out people who do not contribute to the community, e. g. just go around downvoting everything.
Sometimes though there can be cases of harassment based on downvotes. I won’t mention no names, but recently I saw someone on Lemmy reveal (with evidence) how they were sent death threats and were harassed for something FAR more petty. If people like that can exist, then I don’t even want to imagine what can happen when your disapproval of a person’s opinions is public.