Expand the purpose of this backend?
Depending on the evolution of https://github.com/huggingface/datasets, this project might disappear, or its features might be reduced, in particular, if one day it allows caching the data by self-generating:

- an arrow or a parquet data file (maybe with sharding and compression for the largest datasets)
- or a SQL database
- or precompute and store a partial list of known offsets (every 10MB for example)

It would allow getting random access to the data.
Expand the purpose of this backend?: Depending on the evolution of https://github.com/huggingface/datasets, this project might disappear, or its features might be reduced, in particular, if one day it allows caching the data by self-generating:

- an arrow or a parquet data file (maybe with sharding and compression for the largest datasets)
- or a SQL database
- or precompute and store a partial list of known offsets (every 10MB for example)

It would allow getting random access to the data.