The moov contains a list of byte offsets which the player can use to directly access media data. You can skip the moofs and other headers inside by using gaps in the offsets.