 `config-parquet-and-info` crashes with OOM at pq files validation, presumably when there are a lot of files
[bigcode/the-stack-v2](https://huggingface.co/datasets/bigcode/the-stack-v2) keeps failing at config-parquet-and-info step and seems that the problem is that there are too many parquet files to validate (there are about 1k of them). 

It [starts](https://github.com/huggingface/datasets-server/blob/main/services/worker/src/worker/job_runners/config/parquet_and_info.py#L645) [ `retry_and_validate_get_parquet_file_and_size`](https://github.com/huggingface/datasets-server/blob/main/services/worker/src/worker/job_runners/config/parquet_and_info.py#L651): 

![image](https://github.com/huggingface/datasets-server/assets/16348744/a88733c6-145a-4bae-b6b3-8980e33b5d00)

and then fails with OOM in about a minute:

![image](https://github.com/huggingface/datasets-server/assets/16348744/bc8d7221-fa0a-4ff1-84c1-513b5e1ebc6e)

See also https://github.com/huggingface/datasets-server/pull/2544 which didn't help.

Note that [bigcode/the-stack-v2-dedup](https://huggingface.co/datasets/bigcode/the-stack-v2-dedup) which is about 2-4 times smaller works at this step.

cc @anton-l 
 `config-parquet-and-info` crashes with OOM at pq files validation, presumably when there are a lot of files: [bigcode/the-stack-v2](https://huggingface.co/datasets/bigcode/the-stack-v2) keeps failing at config-parquet-and-info step and seems that the problem is that there are too many parquet files to validate (there are about 1k of them). 

It [starts](https://github.com/huggingface/datasets-server/blob/main/services/worker/src/worker/job_runners/config/parquet_and_info.py#L645) [ `retry_and_validate_get_parquet_file_and_size`](https://github.com/huggingface/datasets-server/blob/main/services/worker/src/worker/job_runners/config/parquet_and_info.py#L651): 

![image](https://github.com/huggingface/datasets-server/assets/16348744/a88733c6-145a-4bae-b6b3-8980e33b5d00)

and then fails with OOM in about a minute:

![image](https://github.com/huggingface/datasets-server/assets/16348744/bc8d7221-fa0a-4ff1-84c1-513b5e1ebc6e)

See also https://github.com/huggingface/datasets-server/pull/2544 which didn't help.

Note that [bigcode/the-stack-v2-dedup](https://huggingface.co/datasets/bigcode/the-stack-v2-dedup) which is about 2-4 times smaller works at this step.

cc @anton-l 