feat: 🎸 add MAX_JOBS_PER_DATASET to improve queue availability
Some datasets (https://huggingface.co/datasets/lvwerra/github-code) have
more than 100 splits. To avoid congesting the queue when such a dataset
is refreshed, the number of concurrent jobs for the same dataset is now
limited (to 2 jobs per default using MAX_JOBS_PER_DATASET environment
variable).
feat: 🎸 add MAX_JOBS_PER_DATASET to improve queue availability: Some datasets (https://huggingface.co/datasets/lvwerra/github-code) have
more than 100 splits. To avoid congesting the queue when such a dataset
is refreshed, the number of concurrent jobs for the same dataset is now
limited (to 2 jobs per default using MAX_JOBS_PER_DATASET environment
variable).