Setup prometheus + grafana
Related to #2

- [x] expose a `/metrics` endpoint using the Prometheus spec in the API, eg using https://github.com/prometheus/client_python - see #258. Beware: cache and queue metrics removed after https://github.com/huggingface/datasets-server/issues/279
- [x] Use a ServiceMonitor in the Helm chart: https://github.com/huggingface/tensorboard-launcher/blob/main/kube/templates/servicemonitor.yaml: see #260
- [x] create a dashboard in grafana. The recommended process is to:
  - [x] ensure Grafana can see the metrics: OK, at https://grafana.huggingface.tech/explore?orgId=1&left=%7B%22datasource%22:%22Prometheus%20EKS%20Hub%20Prod%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22instant%22:true,%22range%22:false,%22exemplar%22:false,%22expr%22:%22%7Bjob%3D%5C%22datasets-server-prod-api%5C%22,%20__name__%3D~%5C%22.%2B%5C%22%7D%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D
  - [x] read the [doc](https://github.com/huggingface/infra/blob/main/projects/monitoring/README.md#dashboard-development-workflow) written by @McPatate 
  - [x] create a new JSON in https://github.com/huggingface/infra/blob/main/projects/monitoring/grafana/dashboards/hub/, copying one of the existing ones
  - [x] create a PR: an ephemeral grafana will be available to test
  - [x] tweak the dashboard in the grafana frontend: https://grafana.huggingface.tech/?orgId=1 -> save: will give an error but allow to download the JSON to paste into the PR
- [x] add metrics about the cache and the queue. See discussion in https://github.com/huggingface/datasets-server/issues/279 and work on #310
Setup prometheus + grafana: Related to #2

- [x] expose a `/metrics` endpoint using the Prometheus spec in the API, eg using https://github.com/prometheus/client_python - see #258. Beware: cache and queue metrics removed after https://github.com/huggingface/datasets-server/issues/279
- [x] Use a ServiceMonitor in the Helm chart: https://github.com/huggingface/tensorboard-launcher/blob/main/kube/templates/servicemonitor.yaml: see #260
- [x] create a dashboard in grafana. The recommended process is to:
  - [x] ensure Grafana can see the metrics: OK, at https://grafana.huggingface.tech/explore?orgId=1&left=%7B%22datasource%22:%22Prometheus%20EKS%20Hub%20Prod%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22instant%22:true,%22range%22:false,%22exemplar%22:false,%22expr%22:%22%7Bjob%3D%5C%22datasets-server-prod-api%5C%22,%20__name__%3D~%5C%22.%2B%5C%22%7D%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D
  - [x] read the [doc](https://github.com/huggingface/infra/blob/main/projects/monitoring/README.md#dashboard-development-workflow) written by @McPatate 
  - [x] create a new JSON in https://github.com/huggingface/infra/blob/main/projects/monitoring/grafana/dashboards/hub/, copying one of the existing ones
  - [x] create a PR: an ephemeral grafana will be available to test
  - [x] tweak the dashboard in the grafana frontend: https://grafana.huggingface.tech/?orgId=1 -> save: will give an error but allow to download the JSON to paste into the PR
- [x] add metrics about the cache and the queue. See discussion in https://github.com/huggingface/datasets-server/issues/279 and work on #310