I believe your experience. I just want to contribute the anecdata that I am a stay-at-home dad to my 2.5 year old, and I have spent most days for the past year and a half talking him out on walks, to parks, downtown, etc. in the Bay Area suburbs, more or less following him around and letting him do whatever he pleases. The only experience I have had where someone bothered me in public was some woman in a car who really wanted me to put a hat on him.

I would be interested to hear from other people to understand why this happens in some places and to some people, but not others. I always wonder if I am going to start getting harassed in public, since it's a common complaint, but so far -- nothing.