Dataloader is the bottleneck in training?

[mengxuyiGit]

Thanks for the great work and code release!

However, I noticed a significant training speed difference between loading the original uncompressed GObjaverse data and reading from the processed h5 data, where loading h5 is about 6 times slower than the uncompressed one. The GPU util is also very low.

Is this normal?

[apchenstu]

Thank you for the kind words! How slow is it? It is supposed to be faster than the uncompressed version since it doesn't need to be decompressed, and the dataset is stored in a hierarchical structure. The GPU utilization is around 90%-100% on my side.