descriptive-statistics job is sometimes stuck
Conversation on Slack (internal) https://huggingface.slack.com/archives/C04L6P8KNQ5/p1698325340414399?thread_ts=1698323528.797149&cid=C04L6P8KNQ5

> Note that stats jobs get stuck very often for some reason, I think Polina was investigating
> e.g. this job has been stuck for 30min at the duckdb data ingestion step (only 22MB of data)
```
INFO: 2023-10-26 12:44:15,764 - root - [split-descriptive-statistics] compute JobManager(job_id=653a5e8e2a3bd4b73b210849 dataset=onuralp/open-otter job_info={'job_id': '653a5e8e2a3bd4b73b210849', 'type': 'split-descriptive-statistics', 'params': {'dataset': 'onuralp/open-otter', 'revision': '17db84f97d83d3d782869fa3767b812ba4f7e407', 'config': 'default', 'split': 'train'}, 'priority': <Priority.NORMAL: 'normal'>, 'difficulty': 70}
INFO: 2023-10-26 12:44:15,771 - root - Compute descriptive statistics for dataset='onuralp/open-otter', config='default', split='train'
INFO: 2023-10-26 12:44:15,774 - root - Downloading remote parquet files to a local directory /storage/stats-cache/89781749453102-split-descriptive-statistics-onuralp-open-otter-776f2760. 
Downloading 0000.parquet: 100%|██████████| 22.3M/22.3M [00:01<00:00, 17.0MB/s]
INFO: 2023-10-26 12:47:00,397 - root - Original number of threads=2
INFO: 2023-10-26 12:47:00,398 - root - Current number of threads=8
INFO: 2023-10-26 12:47:00,398 - root - Original max_memory='105.7GB'
INFO: 2023-10-26 12:47:00,399 - root - Current max_memory='28.0GB'
INFO: 2023-10-26 12:47:00,399 - root - Loading data into in-memory table. 
INFO: 2023-10-26 12:47:00,399 - root - 
CREATE OR REPLACE TABLE data AS SELECT "input","output","instruction","data_source" FROM read_parquet('/storage/stats-cache/89781749453102-split-descriptive-statistics-onuralp-open-otter-776f2760/default/train/*.parquet');
```
> (running this duckdb command in a shell is instantaneous !!) (edited) 
> (and the subsequent duckdb-index job took a few seconds)

descriptive-statistics job is sometimes stuck: Conversation on Slack (internal) https://huggingface.slack.com/archives/C04L6P8KNQ5/p1698325340414399?thread_ts=1698323528.797149&cid=C04L6P8KNQ5

> Note that stats jobs get stuck very often for some reason, I think Polina was investigating
> e.g. this job has been stuck for 30min at the duckdb data ingestion step (only 22MB of data)
```
INFO: 2023-10-26 12:44:15,764 - root - [split-descriptive-statistics] compute JobManager(job_id=653a5e8e2a3bd4b73b210849 dataset=onuralp/open-otter job_info={'job_id': '653a5e8e2a3bd4b73b210849', 'type': 'split-descriptive-statistics', 'params': {'dataset': 'onuralp/open-otter', 'revision': '17db84f97d83d3d782869fa3767b812ba4f7e407', 'config': 'default', 'split': 'train'}, 'priority': <Priority.NORMAL: 'normal'>, 'difficulty': 70}
INFO: 2023-10-26 12:44:15,771 - root - Compute descriptive statistics for dataset='onuralp/open-otter', config='default', split='train'
INFO: 2023-10-26 12:44:15,774 - root - Downloading remote parquet files to a local directory /storage/stats-cache/89781749453102-split-descriptive-statistics-onuralp-open-otter-776f2760. 
Downloading 0000.parquet: 100%|██████████| 22.3M/22.3M [00:01<00:00, 17.0MB/s]
INFO: 2023-10-26 12:47:00,397 - root - Original number of threads=2
INFO: 2023-10-26 12:47:00,398 - root - Current number of threads=8
INFO: 2023-10-26 12:47:00,398 - root - Original max_memory='105.7GB'
INFO: 2023-10-26 12:47:00,399 - root - Current max_memory='28.0GB'
INFO: 2023-10-26 12:47:00,399 - root - Loading data into in-memory table. 
INFO: 2023-10-26 12:47:00,399 - root - 
CREATE OR REPLACE TABLE data AS SELECT "input","output","instruction","data_source" FROM read_parquet('/storage/stats-cache/89781749453102-split-descriptive-statistics-onuralp-open-otter-776f2760/default/train/*.parquet');
```
> (running this duckdb command in a shell is instantaneous !!) (edited) 
> (and the subsequent duckdb-index job took a few seconds)
