Cache parquet metadata to optimize /rows
Step 1 of https://github.com/huggingface/datasets-server/issues/1186

I added a new job that gets the parquet metadata of each parquet file and write them on disk in the `assets_directory`.

These metadata will be used to optimize random access to rows, that I will implement in a subsequent PR.

The parquet metadata files are placed in `assets/<dataset>/-pq-meta/<config>/` and have the same filename as the parquet files in `refs/convert/parquet`. I chose `-pq-meta` as a dataset separator because it needs to start with a dash to differentiate with dataset names.

Usually these metadata files are supposed to be grouped into one `_metadata` sidecar file for all the parquet files, but I figured it was easier to have one per parquet file and it requires to load less data when doing a random access.

In the mongodb cache I store the lists of parquet metadata files and their `num_rows`, so that we can know in advance which parquet and metadata file to use.
Cache parquet metadata to optimize /rows: Step 1 of https://github.com/huggingface/datasets-server/issues/1186

I added a new job that gets the parquet metadata of each parquet file and write them on disk in the `assets_directory`.

These metadata will be used to optimize random access to rows, that I will implement in a subsequent PR.

The parquet metadata files are placed in `assets/<dataset>/-pq-meta/<config>/` and have the same filename as the parquet files in `refs/convert/parquet`. I chose `-pq-meta` as a dataset separator because it needs to start with a dash to differentiate with dataset names.

Usually these metadata files are supposed to be grouped into one `_metadata` sidecar file for all the parquet files, but I figured it was easier to have one per parquet file and it requires to load less data when doing a random access.

In the mongodb cache I store the lists of parquet metadata files and their `num_rows`, so that we can know in advance which parquet and metadata file to use.