Change the architecture
Currently: the API server, the cache/database, the assets, and the workers (that generate the data) are running on the same machine and share the same resources, which is the source of various issues:
- a worker that requires a lot of resources can block the server (https://grafana.huggingface.co/d/rYdddlPWk/node-exporter-full?orgId=2&refresh=1m&from=now-24h&to=now&var-DS_PROMETHEUS=HF%20Prometheus&var-job=node_exporter_metrics&var-node=datasets-preview-backend&var-diskdevices=%5Ba-z%5D%2B%7Cnvme%5B0-9%5D%2Bn%5B0-9%5D%2B)
- we have to kill the warming process if memory usage is too high to preserve the API resources, which requires manual supervision
- also related to resources limits: we currently run the warming and refreshing tasks on one dataset at a time, while they are logically independent and could be launched on different workers in parallel, reducing the duration of these processes
- also: I'm not sure if the current implementation of the database/cache (diskcache) really supports concurrent access (it does, but I'm not sure I used it adequately in the code, see http://www.grantjenks.com/docs/diskcache/tutorial.html / `cache.close()`)
- having everything in the same application also means that everything is developed in Python (since the workers have to be in Python), while managing a queue and async processes could be easier in node.js, for example

The architecture I imagine would have these components:
- API server
- queue
- database
- file storage
- workers

The API server would:
- deliver the data (`/rows`, `/splits`, `/valid`, `/cache-reports`, `/cache`, `/healthcheck`): directly querying the database. If not in the database, return an error.
- serve the assets from the storage
- command the queue (`/webhook`, `/warm`, `/refresh`) -> add authentication? Send new tasks to the queue

The queue would:
- manage the tasks sent by the API server
- launch workers for these tasks
- add/update/delete the data in the database and the assets in the storage

The database would:
- store the datasets' data

The storage would:
- store the assets (image files for example)

The workers would:
- compute the data for one dataset

Change the architecture: Currently: the API server, the cache/database, the assets, and the workers (that generate the data) are running on the same machine and share the same resources, which is the source of various issues:
- a worker that requires a lot of resources can block the server (https://grafana.huggingface.co/d/rYdddlPWk/node-exporter-full?orgId=2&refresh=1m&from=now-24h&to=now&var-DS_PROMETHEUS=HF%20Prometheus&var-job=node_exporter_metrics&var-node=datasets-preview-backend&var-diskdevices=%5Ba-z%5D%2B%7Cnvme%5B0-9%5D%2Bn%5B0-9%5D%2B)
- we have to kill the warming process if memory usage is too high to preserve the API resources, which requires manual supervision
- also related to resources limits: we currently run the warming and refreshing tasks on one dataset at a time, while they are logically independent and could be launched on different workers in parallel, reducing the duration of these processes
- also: I'm not sure if the current implementation of the database/cache (diskcache) really supports concurrent access (it does, but I'm not sure I used it adequately in the code, see http://www.grantjenks.com/docs/diskcache/tutorial.html / `cache.close()`)
- having everything in the same application also means that everything is developed in Python (since the workers have to be in Python), while managing a queue and async processes could be easier in node.js, for example

The architecture I imagine would have these components:
- API server
- queue
- database
- file storage
- workers

The API server would:
- deliver the data (`/rows`, `/splits`, `/valid`, `/cache-reports`, `/cache`, `/healthcheck`): directly querying the database. If not in the database, return an error.
- serve the assets from the storage
- command the queue (`/webhook`, `/warm`, `/refresh`) -> add authentication? Send new tasks to the queue

The queue would:
- manage the tasks sent by the API server
- launch workers for these tasks
- add/update/delete the data in the database and the assets in the storage

The database would:
- store the datasets' data

The storage would:
- store the assets (image files for example)

The workers would:
- compute the data for one dataset
