It sounds like there’s a torrent of gossip in your neighborhood about you, and I’m really sorry you have to suffer the consequences. Have any of the parents at the park ever just…chatted with you while the kids play? Its very strange that everyone seems to recognize you and have the same, unhinged opinion of you.
This has happened living in multiple places (in AU), we move lots. Yes I've had some nice people too, usually more curious chats about a stay-at-home dad being odd. I've run into other SAHD's and they've all shared similar stories.