I see you tried private browsing and it works, so that would imply a bug in something that runs when not private browsing, so most likely a plugin that is blocking something.
Start doing a plugin bisection and see if you can find the offending plugin.
Well, the idea still stands. Private->ok, not private->not ok. That's your telltale sign, go with the evidence, as a famous TV show says. Maybe you have something on your normal profile, I dunno what, but I like bashing Apple and since you say you're on MacOS, see what Apple does with that profile. Who knows, maybe you encounter a bug and you can get money out of it from Apple's bug hunting program. Good luck!
Have you tried uninstalling Firefox, deleting any residual appdata, and installing again? Have you tried installing the same version of FF Nightly and FF on another macOS computer?
Much less radical is just to use the command line options to create a new profile and see what happens. I have fixed obscure problems in firefox by blowing away the profile. 99% of what I need is stored in firefox sync so it's an easy process to delete an old profile and sync back up. Sounds like OP wants to get to the bottom of this though as a haX0r-ng project.
Did you reuse existing profiles on both versions, or in either case did you start a completely fresh profile? If you haven't yet tested with a newly-created profile (which doesn't require deleting your existing profile), then you need to do so in order to triage whether the problem is with Firefox-by-default or Firefox-and-your-profile.
Since it exists in both versions of FF, could you have anything installed which could be interfering with FF's communication with the rest of the internet? Are you on a network with a router which is able to modify your traffic (requires custom SSL certificate) (long shot)?
I'd try installing your version of FF Nightly on another macOS computer to rule out that possibility. Maybe it's not a config issue but something to do with Nightly which is affecting your other, release FF
Start doing a plugin bisection and see if you can find the offending plugin.