[split-duckdb-index] OOM for big datasets
e.g. dataset=c4, config=en, split=train has a partial parquet export of 5GB

There also seems to be more than 12k cache entries with the same error code, over more than 3k datasets. This includes lots of big datasets (c4, oscar, wikipedia and all the variants by users) as well as many image and audio datasets.
[split-duckdb-index] OOM for big datasets: e.g. dataset=c4, config=en, split=train has a partial parquet export of 5GB

There also seems to be more than 12k cache entries with the same error code, over more than 3k datasets. This includes lots of big datasets (c4, oscar, wikipedia and all the variants by users) as well as many image and audio datasets.