feat: 🎸 add parquet worker
missing:
- [x] unit test: download the parquet files and ensure the data are the same
- [x] docker compose
- [x] e2e
- [x] Helm
- [x] README, DEVELOPER GUIDE
- [x] fix an issue: we receive an "update" webhook every time we write to the `ref/convert/parquet` branch, which generates a new job -> updates are looping. It's OK to receive the webhook, but we should skip the job because the git version of `main` and the worker version should be the same as the cached entry. Fixed with https://github.com/huggingface/datasets-server/pull/651/commits/56614570e15a1c089286ed70d1f7cb5b0904cac7
- [x] docs
- [x] openapi
- [x] fix e2e: the sum of all the docker images is too large for the Github Actions storage, e2e action breaks.
some examples:
- [x] monitor the storage used for datasets cache, and possibly delete cache after some time. Internal chat: https://huggingface.slack.com/archives/C01TETY5V8S/p1669805539127299. For now, prod has access to 4TB, which gives sufficient space to start. We should delete the cache data for every dataset after a job.

First datasets with parquet files:
- `duorc`: [files](https://huggingface.co/datasets/duorc/tree/refs%2Fconvert%2Fparquet), [API](https://datasets-server.huggingface.co/parquet?dataset=duorc)
- `rotten_tomatoes`: [files](https://huggingface.co/datasets/rotten_tomatoes/tree/refs%2Fconvert%2Fparquet), [API](https://datasets-server.huggingface.co/parquet?dataset=rotten_tomatoes)
- `severo/mnist`: [files](https://huggingface.co/datasets/severo/mnist/tree/refs%2Fconvert%2Fparquet), [API](https://datasets-server.huggingface.co/parquet?dataset=severo/mnist)
- `severo/glue`: [files](https://huggingface.co/datasets/severo/glue/tree/refs%2Fconvert%2Fparquet), [API](https://datasets-server.huggingface.co/parquet?dataset=severo/glue)
- `severo/danish-wit`: [files](https://huggingface.co/datasets/severo/danish-wit/tree/refs%2Fconvert%2Fparquet), [API](https://datasets-server.huggingface.co/parquet?dataset=severo/danish-wit)

Issues:
- [ ] https://huggingface.co/datasets/Elite35P-Server/EliteVoiceProject gives a 500 error (it's a gated dataset, and we get a "forbidden" error). It should have been ignored because it's not on the list of supported datasets. Update: `Elite35P-Server/EliteVoiceProject` is no more gated, btw.
  Same with `SciSearch/wiki-data`, `TREC-AToMiC/AToMiC-Images-v0.1`, `TREC-AToMiC/AToMiC-Qrels-v0.1`, `TREC-AToMiC/AToMiC-Texts-v0.1`, `bigcode/the-stack` (it has extra fields), `bigcode/the-stack-dedup` (same), `mitclinicalml/clinical-ie`
- [ ] `severo/embellishments` and `severo/wit`: "An error occurred while generating the dataset"
- [ ] `severo/winogavil`: `Repo id must be in the form 'repo_name' or 'namespace/repo_name': 'datasets/nlphuji/winogavil'. Use `repo_type` argument if needed.`???


once deployed:
- [x] allow parquet conversion for all the datasets (see https://github.com/huggingface/datasets-server/blob/worker-parquet/chart/env/prod.yaml#L197)
- [x] increase the resources (number of nodes) in production
- [x] rapid API: https://rapidapi.com/studio/api_0971eb45-d045-4337-be04-cec27c1122b0/client
- [x] postman: https://www.postman.com/winter-station-148777/workspace/hugging-face-apis/collection/23242779-d068584e-96d1-4d92-a703-7cb12cbd8053

feat: 🎸 add parquet worker: missing:
- [x] unit test: download the parquet files and ensure the data are the same
- [x] docker compose
- [x] e2e
- [x] Helm
- [x] README, DEVELOPER GUIDE
- [x] fix an issue: we receive an "update" webhook every time we write to the `ref/convert/parquet` branch, which generates a new job -> updates are looping. It's OK to receive the webhook, but we should skip the job because the git version of `main` and the worker version should be the same as the cached entry. Fixed with https://github.com/huggingface/datasets-server/pull/651/commits/56614570e15a1c089286ed70d1f7cb5b0904cac7
- [x] docs
- [x] openapi
- [x] fix e2e: the sum of all the docker images is too large for the Github Actions storage, e2e action breaks.
some examples:
- [x] monitor the storage used for datasets cache, and possibly delete cache after some time. Internal chat: https://huggingface.slack.com/archives/C01TETY5V8S/p1669805539127299. For now, prod has access to 4TB, which gives sufficient space to start. We should delete the cache data for every dataset after a job.

First datasets with parquet files:
- `duorc`: [files](https://huggingface.co/datasets/duorc/tree/refs%2Fconvert%2Fparquet), [API](https://datasets-server.huggingface.co/parquet?dataset=duorc)
- `rotten_tomatoes`: [files](https://huggingface.co/datasets/rotten_tomatoes/tree/refs%2Fconvert%2Fparquet), [API](https://datasets-server.huggingface.co/parquet?dataset=rotten_tomatoes)
- `severo/mnist`: [files](https://huggingface.co/datasets/severo/mnist/tree/refs%2Fconvert%2Fparquet), [API](https://datasets-server.huggingface.co/parquet?dataset=severo/mnist)
- `severo/glue`: [files](https://huggingface.co/datasets/severo/glue/tree/refs%2Fconvert%2Fparquet), [API](https://datasets-server.huggingface.co/parquet?dataset=severo/glue)
- `severo/danish-wit`: [files](https://huggingface.co/datasets/severo/danish-wit/tree/refs%2Fconvert%2Fparquet), [API](https://datasets-server.huggingface.co/parquet?dataset=severo/danish-wit)

Issues:
- [ ] https://huggingface.co/datasets/Elite35P-Server/EliteVoiceProject gives a 500 error (it's a gated dataset, and we get a "forbidden" error). It should have been ignored because it's not on the list of supported datasets. Update: `Elite35P-Server/EliteVoiceProject` is no more gated, btw.
  Same with `SciSearch/wiki-data`, `TREC-AToMiC/AToMiC-Images-v0.1`, `TREC-AToMiC/AToMiC-Qrels-v0.1`, `TREC-AToMiC/AToMiC-Texts-v0.1`, `bigcode/the-stack` (it has extra fields), `bigcode/the-stack-dedup` (same), `mitclinicalml/clinical-ie`
- [ ] `severo/embellishments` and `severo/wit`: "An error occurred while generating the dataset"
- [ ] `severo/winogavil`: `Repo id must be in the form 'repo_name' or 'namespace/repo_name': 'datasets/nlphuji/winogavil'. Use `repo_type` argument if needed.`???


once deployed:
- [x] allow parquet conversion for all the datasets (see https://github.com/huggingface/datasets-server/blob/worker-parquet/chart/env/prod.yaml#L197)
- [x] increase the resources (number of nodes) in production
- [x] rapid API: https://rapidapi.com/studio/api_0971eb45-d045-4337-be04-cec27c1122b0/client
- [x] postman: https://www.postman.com/winter-station-148777/workspace/hugging-face-apis/collection/23242779-d068584e-96d1-4d92-a703-7cb12cbd8053
