Warm the nginx cache after every dataset update?
Currently, when the list of splits, or the list of rows of a split, are updated, only the cache (MongoDB) is updated. But the cache in nginx still a) has the old value until it expires, or b) is still empty until the endpoint is requested for that dataset.

Case a) is not very important since the cache expiration is 2 minutes.

Case b) is more crucial because for large responses (https://datasets-preview.huggingface.tech/rows?dataset=edbeeching%2Fdecision_transformer_gym_replay&config=halfcheetah-expert-v2&split=train for example), the creation of the JSON response can take so long that the client timeouts (in moon-landing, the client timeouts at 1.5s, while the response might take a bit longer, 3-4s)

Warm the nginx cache after every dataset update?: Currently, when the list of splits, or the list of rows of a split, are updated, only the cache (MongoDB) is updated. But the cache in nginx still a) has the old value until it expires, or b) is still empty until the endpoint is requested for that dataset.

Case a) is not very important since the cache expiration is 2 minutes.

Case b) is more crucial because for large responses (https://datasets-preview.huggingface.tech/rows?dataset=edbeeching%2Fdecision_transformer_gym_replay&config=halfcheetah-expert-v2&split=train for example), the creation of the JSON response can take so long that the client timeouts (in moon-landing, the client timeouts at 1.5s, while the response might take a bit longer, 3-4s)
