I wonder since this runs at real time framerate if it would be possible for someone to composite a regular rasterized frame on top of something like this (with correct depth testing) to make a game
For example a 3rd person game where the character you control and the NPCs/enemies is raster but the environment is all radiance fields
If you go to one of the demos the space bar will cycle through some debug modes. One shows a surface reconstruction. It comes from the usual structure from motion techniques I presume, so it's coarse and noisy, but I think the fundamental idea is viable.
This should absolutely be possible! The hard part is making it look natural: NeRF models (including SMERF) have no explicit materials or lighting. That means that any character inserted into the game will look out of place.
For example a 3rd person game where the character you control and the NPCs/enemies is raster but the environment is all radiance fields