Store the list of parquet shards on the Hub?
We could reorganize the way we store the parquet files on the Hub:

- one directory per config (already done)
- then, one directory per split
- then, one file per shard, with name `n.parquet` (starting at `0.parquet`)
- plus: a file `shards.json` which is a JSON file with an "ordered" array of the shard URLs, ie: `["https://huggingface.co/datasets/LLMs/Alpaca-ShareGPT/parquet/train/0.parquet", "https://huggingface.co/datasets/LLMs/Alpaca-ShareGPT/parquet/train/1.parquet", ..., "https://huggingface.co/datasets/LLMs/Alpaca-ShareGPT/parquet/train/13.parquet"]`

Alternative: don't change the directory structure, nor the filenames, but use redirections. Anyway, add the shard.json files (one per split)

Store the list of parquet shards on the Hub?: We could reorganize the way we store the parquet files on the Hub:

- one directory per config (already done)
- then, one directory per split
- then, one file per shard, with name `n.parquet` (starting at `0.parquet`)
- plus: a file `shards.json` which is a JSON file with an "ordered" array of the shard URLs, ie: `["https://huggingface.co/datasets/LLMs/Alpaca-ShareGPT/parquet/train/0.parquet", "https://huggingface.co/datasets/LLMs/Alpaca-ShareGPT/parquet/train/1.parquet", ..., "https://huggingface.co/datasets/LLMs/Alpaca-ShareGPT/parquet/train/13.parquet"]`

Alternative: don't change the directory structure, nor the filenames, but use redirections. Anyway, add the shard.json files (one per split)
