Store /valid and /is-valid in the cache
Pre-computing /is-valid (for each dataset) and /valid (for all the datasets) and storing the responses in that cache would help reduce the latency of these endpoints and the compute load on the API service. They are the only two endpoints that are calculated on the fly.

Another benefit: it would remove the particular field `required_by_dataset_viewer` in https://github.com/huggingface/datasets-server/blob/main/libs/libcommon/src/libcommon/processing_graph.py because the same property could be defined as dependencies between processing steps:
- /is-valid would depend on /splits and /first-rows
- /valid would depend on /is-valid

Note that we would have to add a value to [InputType](https://github.com/huggingface/datasets-server/blob/0539fbd1950bfbe89605ddd40db1aaebfc757438/libs/libcommon/src/libcommon/processing_graph.py#L10) for /valid. None? Optional?

Thanks, @AndreaFrancis, for the discussion that allowed rethinking this part.


---

- [x] create `dataset-is-valid` https://github.com/huggingface/datasets-server/pull/1032
- [ ] create `valid`
  - [ ] add `none` in InputType, and make the `dataset` parameter optional where it makes sense (it is currently mandatory nearly everywhere in the code)
  - [ ] add the `valid` step
- [ ] deploy and fill the cache. Done for `is-valid`
- [ ] switch the /valid and /is-valid endpoints. PR for `is-valid`: https://github.com/huggingface/datasets-server/pull/1160
Store /valid and /is-valid in the cache: Pre-computing /is-valid (for each dataset) and /valid (for all the datasets) and storing the responses in that cache would help reduce the latency of these endpoints and the compute load on the API service. They are the only two endpoints that are calculated on the fly.

Another benefit: it would remove the particular field `required_by_dataset_viewer` in https://github.com/huggingface/datasets-server/blob/main/libs/libcommon/src/libcommon/processing_graph.py because the same property could be defined as dependencies between processing steps:
- /is-valid would depend on /splits and /first-rows
- /valid would depend on /is-valid

Note that we would have to add a value to [InputType](https://github.com/huggingface/datasets-server/blob/0539fbd1950bfbe89605ddd40db1aaebfc757438/libs/libcommon/src/libcommon/processing_graph.py#L10) for /valid. None? Optional?

Thanks, @AndreaFrancis, for the discussion that allowed rethinking this part.


---

- [x] create `dataset-is-valid` https://github.com/huggingface/datasets-server/pull/1032
- [ ] create `valid`
  - [ ] add `none` in InputType, and make the `dataset` parameter optional where it makes sense (it is currently mandatory nearly everywhere in the code)
  - [ ] add the `valid` step
- [ ] deploy and fill the cache. Done for `is-valid`
- [ ] switch the /valid and /is-valid endpoints. PR for `is-valid`: https://github.com/huggingface/datasets-server/pull/1160