I've never once felt obligated to get on the plane earlier just because I had a window. The person on the aisle knows someone will be getting into the empty seats, so they leave their seatbelt undone and get up when I arrive. I do the same if I have the aisle. Maybe this makes me an asshole, but I don't think this is one of those "common courtesy" things; after all, airlines are far from standard in boarding process; I don't think boarding groups usually favor window first; if anything they're random or start from the back of the plane first, or something like that.

I wouldn't worry about it so much if you can help it; I think it's just normal to have to get in to a window seat. If it bothers you, perhaps select the aisle and comfort yourself that you're being nice by boarding last :)