Check out Mask Banana - you might have better luck with using masks to get image models to pay attention to what you want edited.