What's wrong with spending less time with co-workers? These are people you have to spend time with, regardless if you like them or not. I would personally much rather not spend time with people I didn't choose to spend time with.
I think it may perhaps not be a question of "right" or "wrong". It will probably be a jarring change for many people as it affects major chunks of their regular social contact.
