Limit max RAM that duckdb can use
Current value of `max_memory` in prod is 105.7Gb (despite the pod has only 34Gb but it's set automatically by duckdb as 80% of ram and ram of the node in total is 128Gb)
<img width="382" alt="image" src="https://github.com/huggingface/datasets-server/assets/16348744/b29d7829-43e3-431c-a453-546f80e722a5">
I want to try to lower it manually since we assume there might be some memory issues with duckdb causing the  `split-descriptive-statistics` process to stuck at the step of loading data into in-memory db.
(also want to try to load data to local db file, this is the only difference with `duckdb-index` step)
Limit max RAM that duckdb can use: Current value of `max_memory` in prod is 105.7Gb (despite the pod has only 34Gb but it's set automatically by duckdb as 80% of ram and ram of the node in total is 128Gb)
<img width="382" alt="image" src="https://github.com/huggingface/datasets-server/assets/16348744/b29d7829-43e3-431c-a453-546f80e722a5">
I want to try to lower it manually since we assume there might be some memory issues with duckdb causing the  `split-descriptive-statistics` process to stuck at the step of loading data into in-memory db.
(also want to try to load data to local db file, this is the only difference with `duckdb-index` step)