This is an excellent point. There's the actual moderators who are, at minimum, filtering out spam. And then there are the human users voting, which I assume carries over in one way or another to Google's search ranking.