From the docstring:
#Notes
This is not a lazy operation — the entire parquet file is read into memory
and pivoted before writing to zarr. For large particle files this may
require significant memory. Performance improvements are welcome via PRs.
The performance here can be improved. @erikvansebille mentioned in #2811 (comment) about writing one variable at a time. This would be a significant improvement (cutting memory usage in 4 or 5 depending on number of variables)
I think we can also go further in the memory saving if we treat things on a per-chunk basis. This is something to be explored and benchmarked by people with a vested interest
From the docstring:
The performance here can be improved. @erikvansebille mentioned in #2811 (comment) about writing one variable at a time. This would be a significant improvement (cutting memory usage in 4 or 5 depending on number of variables)
I think we can also go further in the memory saving if we treat things on a per-chunk basis. This is something to be explored and benchmarked by people with a vested interest