Convert if too big row groups for copy
Before copying the parquet files I check that the row groups are not too big. Otherwise it can cause OOM for users that could like to use the parquet export, and also because it would make the dataset viewer too slow.

To do that, I check the first row group of the first parquet files and check their size.
If one row group is bigger than 500MB, we don't copy and `stream_to_parquet` is used instead.

Since we have the row group size, I also estimate an optimal `writer_batch_size` than I pass to `stream_to_parquet`. It must be a factor of 100 rows and have a row group size that is smaller than 500MB.

I currently set the limit to 500MB to fix https://huggingface.co/datasets/Open-Orca/OpenOrca, but if there are other problematic datasets that need to be handled this way we can adapt the limit.

close https://github.com/huggingface/datasets-server/issues/1491
Convert if too big row groups for copy: Before copying the parquet files I check that the row groups are not too big. Otherwise it can cause OOM for users that could like to use the parquet export, and also because it would make the dataset viewer too slow.

To do that, I check the first row group of the first parquet files and check their size.
If one row group is bigger than 500MB, we don't copy and `stream_to_parquet` is used instead.

Since we have the row group size, I also estimate an optimal `writer_batch_size` than I pass to `stream_to_parquet`. It must be a factor of 100 rows and have a row group size that is smaller than 500MB.

I currently set the limit to 500MB to fix https://huggingface.co/datasets/Open-Orca/OpenOrca, but if there are other problematic datasets that need to be handled this way we can adapt the limit.

close https://github.com/huggingface/datasets-server/issues/1491