Aggregated JobRunners: Look for pending job if cache does not exist and create one if not
For aggregated JobRunners like https://github.com/huggingface/datasets-server/blob/main/services/worker/src/worker/job_runners/dataset/split_names_from_streaming.py https://github.com/huggingface/datasets-server/blob/main/services/worker/src/worker/job_runners/dataset_parquet.py and https://github.com/huggingface/datasets-server/blob/main/services/worker/src/worker/job_runners/dataset_size.py, we get the list of configs and iterate them to get the partial response. If no cache is found, we append it to a "pending" list. In that case we should look for pending Jobs in status WAITING/STARTED and if it does not exist, create a new Job.

See thread https://github.com/huggingface/datasets-server/pull/936#discussion_r1138298762 for more details
Aggregated JobRunners: Look for pending job if cache does not exist and create one if not: For aggregated JobRunners like https://github.com/huggingface/datasets-server/blob/main/services/worker/src/worker/job_runners/dataset/split_names_from_streaming.py https://github.com/huggingface/datasets-server/blob/main/services/worker/src/worker/job_runners/dataset_parquet.py and https://github.com/huggingface/datasets-server/blob/main/services/worker/src/worker/job_runners/dataset_size.py, we get the list of configs and iterate them to get the partial response. If no cache is found, we append it to a "pending" list. In that case we should look for pending Jobs in status WAITING/STARTED and if it does not exist, create a new Job.

See thread https://github.com/huggingface/datasets-server/pull/936#discussion_r1138298762 for more details