feat: 🎸 add "difficulty" field to JobDocument
Difficulty is an integer between 0 (easy) and 100 (hard). It aims at filtering the jobs in a specific worker deployment, ie, light workers will only run jobs with difficulty <= 40.

It should make the query to MongoDB quicker than currently (a filter `type: {$in: ALLOW_LIST}`). See #1486.

For now, all the jobs for a specific step have the same difficulty, ie: 20 for dataset-size, since it's only JSON manipulation, while split-duckdb-index is 70 because it can use a lot of RAM and take a lot of time.

Maybe, one day, we could estimate the difficulty of the job, based on previous stats for the same dataset, for example.
feat: 🎸 add "difficulty" field to JobDocument: Difficulty is an integer between 0 (easy) and 100 (hard). It aims at filtering the jobs in a specific worker deployment, ie, light workers will only run jobs with difficulty <= 40.

It should make the query to MongoDB quicker than currently (a filter `type: {$in: ALLOW_LIST}`). See #1486.

For now, all the jobs for a specific step have the same difficulty, ie: 20 for dataset-size, since it's only JSON manipulation, while split-duckdb-index is 70 because it can use a lot of RAM and take a lot of time.

Maybe, one day, we could estimate the difficulty of the job, based on previous stats for the same dataset, for example.