Parquet export: ignore ALL datasets bigger than a certain size
Cases to ignore:
- [x] the dataset repository is >max_size
- [ ] the dataset uses a script that downloads more >max_size of data

To fix the second case I think we can pass a custom download manager to the dataset builder `_split_generators` to record the size of the files to donwload. It can also be implemented by checking the number of fiels passed as `gen_kwargs` and checking their sizes.

Currently max_size=5GB
Parquet export: ignore ALL datasets bigger than a certain size: Cases to ignore:
- [x] the dataset repository is >max_size
- [ ] the dataset uses a script that downloads more >max_size of data

To fix the second case I think we can pass a custom download manager to the dataset builder `_split_generators` to record the size of the files to donwload. It can also be implemented by checking the number of fiels passed as `gen_kwargs` and checking their sizes.

Currently max_size=5GB