Create an orchestrator
## Proposal

We should add a new service: orchestrator.

On one side, it would receive events:
- webhooks when a dataset has changed (added, updated, deleted, changed to gated, etc.)
- manual trigger: refresh a dataset, update all the datasets for a specific step, etc.

On the other side, it would command the jobs:
- create only the jobs that are needed
- receive the heartbeat to ensure a job is still running
- receive the result of a job and:
  - store it, 
  - create the dependent jobs based on the result

The orchestrator would have access to the queue and to the cache.

This means that the job runners can be a lot dumber:
- they don't need to check if they should skip the job or not: they just run what they are given
- they don't need to create the dependent jobs or even know about the processing graph
- they don't have access to the cache (see #751) and to the queue (but the worker still has to access the queue; we could also make the orchestrator launch the jobs on demand instead of having workers that loop and look for jobs in the queue. It's an unrelated issue, though.)

See the related issues: #764, #736, #751, #741, #740
Create an orchestrator: ## Proposal

We should add a new service: orchestrator.

On one side, it would receive events:
- webhooks when a dataset has changed (added, updated, deleted, changed to gated, etc.)
- manual trigger: refresh a dataset, update all the datasets for a specific step, etc.

On the other side, it would command the jobs:
- create only the jobs that are needed
- receive the heartbeat to ensure a job is still running
- receive the result of a job and:
  - store it, 
  - create the dependent jobs based on the result

The orchestrator would have access to the queue and to the cache.

This means that the job runners can be a lot dumber:
- they don't need to check if they should skip the job or not: they just run what they are given
- they don't need to create the dependent jobs or even know about the processing graph
- they don't have access to the cache (see #751) and to the queue (but the worker still has to access the queue; we could also make the orchestrator launch the jobs on demand instead of having workers that loop and look for jobs in the queue. It's an unrelated issue, though.)

See the related issues: #764, #736, #751, #741, #740