Separate the computation of metrics (queue, cache) from the exposition
The metrics about the queue and the cache require some processing, and queries to the MongoDB. It can take some time to process, sometimes more than 10s

<img width="522" alt="Capture d’écran 2023-03-21 à 15 55 30" src="https://user-images.githubusercontent.com/1676121/226646241-acbd6422-e962-44d1-8ea1-2973e4c391ad.png">

We should instead create the metrics with a dedicated process on one side, and just read the last metrics on the other side when the /metrics endpoint is polled.

Currently, the computation of the metrics is done live here: https://github.com/huggingface/datasets-server/blob/5c0e59f79d4a1f2f09b0b2ddef02b0b30a85f226/services/admin/src/admin/prometheus.py#L83
Separate the computation of metrics (queue, cache) from the exposition: The metrics about the queue and the cache require some processing, and queries to the MongoDB. It can take some time to process, sometimes more than 10s

<img width="522" alt="Capture d’écran 2023-03-21 à 15 55 30" src="https://user-images.githubusercontent.com/1676121/226646241-acbd6422-e962-44d1-8ea1-2973e4c391ad.png">

We should instead create the metrics with a dedicated process on one side, and just read the last metrics on the other side when the /metrics endpoint is polled.

Currently, the computation of the metrics is done live here: https://github.com/huggingface/datasets-server/blob/5c0e59f79d4a1f2f09b0b2ddef02b0b30a85f226/services/admin/src/admin/prometheus.py#L83