I'm running a set of chains and saving them to HDF using HDFBackend, but it writes out gigantic (many Gb) files which can be externally compressed with gzip to tens of Mb.
I'd like emcee to write out compressed HDF files and I see that the code has a compressed option but it doesn't seem to be linked in the emcee code to the HDF routine.
I'm writing reproducible papers with showyourwork! and I'd like to store just the end set of walkers from a completed emcee run, so if HDF isn't working, I'd rather used a gzipped pickle approach instead - what objects should I store in a pickle in addition to the walker chains, or is there a transparent way to write compressed HDF files using the backend?
I'm running a set of chains and saving them to HDF using HDFBackend, but it writes out gigantic (many Gb) files which can be externally compressed with gzip to tens of Mb.
I'd like emcee to write out compressed HDF files and I see that the code has a compressed option but it doesn't seem to be linked in the emcee code to the HDF routine.
I'm writing reproducible papers with showyourwork! and I'd like to store just the end set of walkers from a completed emcee run, so if HDF isn't working, I'd rather used a gzipped pickle approach instead - what objects should I store in a pickle in addition to the walker chains, or is there a transparent way to write compressed HDF files using the backend?