Memory efficient config-parquet-metadata
I moved some code to make the job write the parquet metadata on disk as they are downloaded, instead of keeping them all in RAM and write them all at the end.

should help for https://github.com/huggingface/datasets-server/issues/1502
Memory efficient config-parquet-metadata: I moved some code to make the job write the parquet metadata on disk as they are downloaded, instead of keeping them all in RAM and write them all at the end.

should help for https://github.com/huggingface/datasets-server/issues/1502