Votes attract votes. Users feel more inclined to follow the positive or negative trend when voting. If given the option to hide vote scores it would give users the ability to vote without being influenced by how everyone else has voted, and maybe have an effect on what posts rise to the top.
If you want to get fancy it could have the option to show scores of things that a user has already voted on.
Have you tried turning off scores from your Lemmy profile?