I think I’ve settled on the latter. Disagreement is maybe best communicated by the absence of an upvote? And downvotes work best when they signal something that is just off base, and while not reportable, is not appreciated at a broad cultural level.
The nice thing about the fediverse is all my votes are displayed. I generally only down vote a post if it’s way off topic or it’s rife with hate. This also applies to comments.
I generally upvote most things as long as it fits the topic, even if I generally disagree with it.