It varies by course and college. My college had a relatively small number of maths students and would give a prize to the top performers each year. We all knew who got the prize and what grade everyone got. Some of my friends were top athletes in niche sports, but none of us really cared about the boat race, varsity rugby, or anything like that. Social life was centred much more around drinking than sport. I don't think our dining hall arrangement had anything to do with sports teams, official or otherwise.