try duckdb resources configuration
`split-duckdb-index` fails mostly for big datasets in the create index step (PRAGMA create_fts_index) because of OOM.
Same as in https://github.com/huggingface/datasets-server/pull/2081/files, I would like to try custom duckdb configurations to see if that helps.
Also I noticed the close condition was never closed when there was an error in parquet files downloading.
try duckdb resources configuration: `split-duckdb-index` fails mostly for big datasets in the create index step (PRAGMA create_fts_index) because of OOM.
Same as in https://github.com/huggingface/datasets-server/pull/2081/files, I would like to try custom duckdb configurations to see if that helps.
Also I noticed the close condition was never closed when there was an error in parquet files downloading.