That's a whole field of study on its own, called perceptual hashing. I surveyed these a while for amusement and the TL;DR is that all immediately obvious approaches tend to have particularly bad corner cases.
I just wrote a quick function to compare visual similarity using the thumbhash and just adding up the difference at each byte position seems to work really well! (As long as the images are the same aspect ratio. I want to do more tests.)
I was thinking about that too. Can't answer the question, but I did come across this just the other day: https://github.com/cloudinary/ssimulacra2 Supposedly good for comparing image similarity. Might depend on your use-case, I think it's geared towards image quality moreso than similar photos.
