Add bonus difficulty if dataset is big
Add a bonus difficulty of 20 for `duckdb-index` if the dataset is bigger than 3GB (from the config info).
This is defined in the orchestrator.

This can be used to defined "heavy workers" that would have enough RAM to process them (up to 29GB of memory for datasets like C4 that have their first 5GB converted to parquet)

The main changes are in
libs/libcommon/src/libcommon/config.py
libs/libcommon/src/libcommon/constants.py
libs/libcommon/src/libcommon/orchestrator.py
libs/libcommon/src/libcommon/processing_graph.py

TODO:
- [x] tests

Related to https://github.com/huggingface/datasets-server/issues/1891
Add bonus difficulty if dataset is big: Add a bonus difficulty of 20 for `duckdb-index` if the dataset is bigger than 3GB (from the config info).
This is defined in the orchestrator.

This can be used to defined "heavy workers" that would have enough RAM to process them (up to 29GB of memory for datasets like C4 that have their first 5GB converted to parquet)

The main changes are in
libs/libcommon/src/libcommon/config.py
libs/libcommon/src/libcommon/constants.py
libs/libcommon/src/libcommon/orchestrator.py
libs/libcommon/src/libcommon/processing_graph.py

TODO:
- [x] tests

Related to https://github.com/huggingface/datasets-server/issues/1891