Also even when some are trolling, the up/down vote will not gain/lose attention for the troll. Trolls love attention.

The lack of karma means participation is there not whether it is a troll or regular.

Unless it is spam, almost no one is punished/rewarded for participating.