test priority small datasets vs big dataset
Add a test to get some clues about the next waiting job issue.
- From comment https://github.com/huggingface/datasets-server/pull/2141#issuecomment-1829822409, I couldn't reproduce the issue regarding small datasets being blocked by big ones.
- I was able to evaluate and measure the delay due to reading concurrency for the next waiting job that has already started. See comment https://github.com/huggingface/datasets-server/pull/2171#issuecomment-1833836921
What I would suggest regarding queue performance is:
- Try removing finished jobs in the queue so that db takes less time to retrieve the next waiting job as proposed in https://github.com/huggingface/datasets-server/pull/2137#issuecomment-1832674005
- Try to avoid reading concurrency in another PR  

Also.: Note that even though we have 46 active workers, we have about 15 started jobs most of the time.
test priority small datasets vs big dataset: Add a test to get some clues about the next waiting job issue.
- From comment https://github.com/huggingface/datasets-server/pull/2141#issuecomment-1829822409, I couldn't reproduce the issue regarding small datasets being blocked by big ones.
- I was able to evaluate and measure the delay due to reading concurrency for the next waiting job that has already started. See comment https://github.com/huggingface/datasets-server/pull/2171#issuecomment-1833836921
What I would suggest regarding queue performance is:
- Try removing finished jobs in the queue so that db takes less time to retrieve the next waiting job as proposed in https://github.com/huggingface/datasets-server/pull/2137#issuecomment-1832674005
- Try to avoid reading concurrency in another PR  

Also.: Note that even though we have 46 active workers, we have about 15 started jobs most of the time.