Having worked with some MP4 demuxing for my extension [1], I feel the pain. Lots of times I would play the video only to find inexplicable issues such as drifting audio. I highly recommend using an mp4 inspector tool, such as mp4box [2], to debug these issues.

1: https://github.com/Andrews54757/FastStream

2: https://gpac.github.io/mp4box.js/test/filereader.html