Storage disk metrics are not real-time
There is a lag of up to ~24h for some reason.
I noticed that when running long deletion jobs on the datasets cache

![image](https://github.com/huggingface/datasets-server/assets/42851186/6f5c3da8-df40-4811-8777-35bbe4e209e1)

When running the same python code as the one used to send the metric to prometheus I was able to get the right values.
Maybe the metrics refreshes are not frequent enough ?
Storage disk metrics are not real-time: There is a lag of up to ~24h for some reason.
I noticed that when running long deletion jobs on the datasets cache

![image](https://github.com/huggingface/datasets-server/assets/42851186/6f5c3da8-df40-4811-8777-35bbe4e209e1)

When running the same python code as the one used to send the metric to prometheus I was able to get the right values.
Maybe the metrics refreshes are not frequent enough ?