I think fixtures generally work fine. If a change to one breaks many tests, introduce a new one and start using that. I also think it's okay to make some manual changes to them in the test and it's distinct from wanting factories; needing factories only in test code feels like a waste.

100% agree with "Test only what you want to test".