If you use the kustomize controller from the FluxCD project, you can set it to prune unmanaged objects.

In true GitOps, I think it's should be default on.