Avoid creating jobs as much as possible
We are often facing a full queue that blocks the viewer on the Hub from being available (https://github.com/huggingface/datasets-server/issues/725, https://github.com/huggingface/datasets-server/issues/704, etc.)

It's generally the /first-rows queue.

There are several ideas to improve this. One of them is to detect as soon as possible which jobs are useless because they will be skipped when processed by the worker. I suspect that a large number of the processed jobs is skipped (to do: get numbers), and that being able to remove that unnecessary burden on the workers would help.

This means, when adding a job, test if the job would be skipped or not. If it would be skipped, I think that we cannot just avoid creating it and forget, because a dependent processing step could still need to be computed (new step, error, outdated). I think that we should effectively avoid creating the job, but still run the post-processing action to try to add the dependent jobs (and do the same).
I think what this means is separating the "event" that triggers a possible refresh for a dataset, and the "materialization" step (see https://docs.dagster.io/concepts/assets/asset-materializations) where we look at which operations are needed, and run them.
Avoid creating jobs as much as possible: We are often facing a full queue that blocks the viewer on the Hub from being available (https://github.com/huggingface/datasets-server/issues/725, https://github.com/huggingface/datasets-server/issues/704, etc.)

It's generally the /first-rows queue.

There are several ideas to improve this. One of them is to detect as soon as possible which jobs are useless because they will be skipped when processed by the worker. I suspect that a large number of the processed jobs is skipped (to do: get numbers), and that being able to remove that unnecessary burden on the workers would help.

This means, when adding a job, test if the job would be skipped or not. If it would be skipped, I think that we cannot just avoid creating it and forget, because a dependent processing step could still need to be computed (new step, error, outdated). I think that we should effectively avoid creating the job, but still run the post-processing action to try to add the dependent jobs (and do the same).
I think what this means is separating the "event" that triggers a possible refresh for a dataset, and the "materialization" step (see https://docs.dagster.io/concepts/assets/asset-materializations) where we look at which operations are needed, and run them.