Allow partial datasets
For the big datasets, it might be hard to get the rows for all the configs (common voice has 155 configs, one per language, and for each of their 5 splits, we have to download 100 audio files -> we get banned quickly).

An improvement would be to refresh the cache on a config basis, instead of a dataset basis. Possibly the atomicity could even be the split, instead of the config.

Doing this, we might lose the coherence of the dataset, and we increase the complexity of the state, but it would give a solution for the big datasets, that we expect to have more and more (audio, images).

It has implications on moonlanding too, because the errors would be per config, or per split, instead of per dataset.
Allow partial datasets: For the big datasets, it might be hard to get the rows for all the configs (common voice has 155 configs, one per language, and for each of their 5 splits, we have to download 100 audio files -> we get banned quickly).

An improvement would be to refresh the cache on a config basis, instead of a dataset basis. Possibly the atomicity could even be the split, instead of the config.

Doing this, we might lose the coherence of the dataset, and we increase the complexity of the state, but it would give a solution for the big datasets, that we expect to have more and more (audio, images).

It has implications on moonlanding too, because the errors would be per config, or per split, instead of per dataset.