The handshake doesn't primarily depend on AES; it is typically a Diffie-Hellman variant (which doesn't have any acceleration) that takes time. Anyway, you're hopefully using TLS 1.3 by now, where you can use ChaCha20 instead of AES :-)