Avoid disk storage issues
The assets, cached-assets (for /rows) and datasets library cache are stored on a disk (might be the same or not). When the total used size reaches a  threshold, new jobs are not run (and the job queue increases).

But we don't check the inodes usage, which can also lead to "no space left" issues (occurred on 2023/04/21, see https://github.com/huggingface/datasets-server/issues/1071).

Some ideas we could implement.

Specific to inodes:
- add a metric about the inodes usage (and add a graph in grafana)
- add a test on the inodes usagebefore launching a new job

General:
- add alerts when disk related metrics reach a threshold
- have a periodical cleaning of the disk. Note that the disk was mostly filled by the datasets library cache for jobs that crashed (because otherwise the temporary directory is deleted at the end of the job). The directories names contain the date, so we can delete all the directories older than one week for example.
- use different disks for assets, cached-assets and cache (datasets library), and monitor them separately (and check them separately before launching a job)
- use S3, instead of a disk?
- move disk metrics to the metrics collector job, instead of checking on every call to /metrics
- add a check on the disk usage in cached-assets before storing new assets when /rows is called (even if we already have a mechanism to reduce the size)
Avoid disk storage issues: The assets, cached-assets (for /rows) and datasets library cache are stored on a disk (might be the same or not). When the total used size reaches a  threshold, new jobs are not run (and the job queue increases).

But we don't check the inodes usage, which can also lead to "no space left" issues (occurred on 2023/04/21, see https://github.com/huggingface/datasets-server/issues/1071).

Some ideas we could implement.

Specific to inodes:
- add a metric about the inodes usage (and add a graph in grafana)
- add a test on the inodes usagebefore launching a new job

General:
- add alerts when disk related metrics reach a threshold
- have a periodical cleaning of the disk. Note that the disk was mostly filled by the datasets library cache for jobs that crashed (because otherwise the temporary directory is deleted at the end of the job). The directories names contain the date, so we can delete all the directories older than one week for example.
- use different disks for assets, cached-assets and cache (datasets library), and monitor them separately (and check them separately before launching a job)
- use S3, instead of a disk?
- move disk metrics to the metrics collector job, instead of checking on every call to /metrics
- add a check on the disk usage in cached-assets before storing new assets when /rows is called (even if we already have a mechanism to reduce the size)