Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

hdf5 is extremely slow however, pickle+zstd is faster and results in smaller files.


I kind of feel like you are doing something wrong with HDF5, since for my use cases it's the fastest solution by far.


hdf5+zstd would likely be comparable.

good luck loading those pickle files 5y from now.


hdf5+zstd is not a thing (or at least not a thing that's interoperable or usable 5y from now). I just wish there was a good off-the-shelf solution, this stuff is not difficult.


yeah, just pointing out that there's nothing inherently wrong with hdf5 and that the gains you speak of are likely from just from the use of modern compression standards.

maybe there's room for a simplified standard... or maybe just the addition of better compression to hdf5. (although they move slow for very good reason)


Why? I have successfully loaded pkl files that are much older than that.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: