In my experience IPFS is only fast when you're using it through something like Cloudflare's IPFS proxy (which is basically a caching proxy). I haven't found IPFS to be actually usable any time I've tried running a node myself. Especially pinning anything beyond the most trivial of examples.
For a start, newer versions of IPFS work much better, but the network still had a significant amount of older nodes making the experience worse for everyone. Luckily, this is changing as of late, it seems.
IPFS shines when it can take advantage of content proximity (i.e. in your LAN, in your node etc.) or when there are few providers for the content within reach, at which point it just works regardless of connectivity to the outside world.
In a general scenario, comparing IPFS-delivery speed with a CDN-backed website is not very fair. Might be slower but offers much more versatility.
Hoping to be proven wrong though.