Generate 5GB parquet files for big datasets
For datasets over 5GB, let's generate 5GB parquet files (with shards) instead of ignoring them. The fact that the dataset was truncated should be stored somewhere.

---

Currently, datasets server gets stores parquet files if the dataset size is less than PARQUET_AND_INFO_MAX_DATASET_SIZE config.
- `PARQUET_AND_INFO_MAX_DATASET_SIZE`: the maximum size in bytes of the dataset to pre-compute the parquet files. Bigger datasets, or datasets without that information, are ignored. Defaults to `100_000_000`.
For future implementations/features we would probably need a full dataset reading, big datasets won't be available for it.

There are a couple of suggestions to mitigate excluding big datasets from full reading:
- Copy at least first x GB of data
- If a dataset is already in parquet files copy them directly instead of processing
- To have a data lake? 

Generate 5GB parquet files for big datasets: For datasets over 5GB, let's generate 5GB parquet files (with shards) instead of ignoring them. The fact that the dataset was truncated should be stored somewhere.

---

Currently, datasets server gets stores parquet files if the dataset size is less than PARQUET_AND_INFO_MAX_DATASET_SIZE config.
- `PARQUET_AND_INFO_MAX_DATASET_SIZE`: the maximum size in bytes of the dataset to pre-compute the parquet files. Bigger datasets, or datasets without that information, are ignored. Defaults to `100_000_000`.
For future implementations/features we would probably need a full dataset reading, big datasets won't be available for it.

There are a couple of suggestions to mitigate excluding big datasets from full reading:
- Copy at least first x GB of data
- If a dataset is already in parquet files copy them directly instead of processing
- To have a data lake? 
