I remember having this realization as a kid. I was standing in my driveway and realized there was an unbroken road from me to the other side of the country (and beyond!). It was one of those the world is so big and so small moments.
I think it's meant to be funny in part because it's almost certainly not literally true. (In some tellings, it could be mocking the rural person who says this for being ignorant of the larger world, or for feeling cut off from it.)
You could also think about borders that are closed. Like if you want to go from Haifa to Beirut, the shortest legal route goes through Jordan and Syria!
Or with the Darien Gap, people asking about how to drive from Panama City to Cartagena might be surprised -- you actually can't get there from there (by road).