That's not correct. Condensed matter physicists will have a good handle on how plausible this is (but not certain). Other people will vote randomly so their votes cancel out, effectively leaving the final result as biased by the expert opinions. That's how the wisdom of the crowd works.