Add endpoint /hub-cache
The Hub's backend will use it to get the status of all the datasets. It will replace /valid, with the benefit that it's paginated, so the response should not timeout.

Also: the idea is to call this endpoint only at Hub's backend startup. Then, we plan to update the statuses with server-sent events (https://github.com/huggingface/datasets-server/issues/1718).

Missing:
- [x] an error when setting the header: `h11._util.LocalProtocolError: Illegal header value b'<http://api:9080/hub-cache?cursor=64ed29b48eba1f585b71776d>;rel="next" '`
- [x] btw: it's not the URL we want (we want the public URL, not the API pod's URL)
- [x] OpenAPI
Add endpoint /hub-cache: The Hub's backend will use it to get the status of all the datasets. It will replace /valid, with the benefit that it's paginated, so the response should not timeout.

Also: the idea is to call this endpoint only at Hub's backend startup. Then, we plan to update the statuses with server-sent events (https://github.com/huggingface/datasets-server/issues/1718).

Missing:
- [x] an error when setting the header: `h11._util.LocalProtocolError: Illegal header value b'<http://api:9080/hub-cache?cursor=64ed29b48eba1f585b71776d>;rel="next" '`
- [x] btw: it's not the URL we want (we want the public URL, not the API pod's URL)
- [x] OpenAPI