Not to excuse people, who provide no metrics or benchmarks for their "performance optimisations", I want to acknowledge that benchmarking is hard.
I hope something like NixOS will enable people to write and publish "reproducible" (in quotes because precise numbers are impossible) benchmarks by enabling you to setup your OS, user space and compilation flags exactly like in the benchmark.
Oh that would be really compelling. I only tried NixOS briefly a while ago, but I think I recall you can switch (actually I think literally `switch`) environment like that pretty easily and back again?
So your benchmark environment could actually be pretty minimal, or if you were sharing a bug reproduction it could be devoid of unrelated packages/config in your usual environment.
I hope something like NixOS will enable people to write and publish "reproducible" (in quotes because precise numbers are impossible) benchmarks by enabling you to setup your OS, user space and compilation flags exactly like in the benchmark.