How to best manage the datasets that we cannot process due to RAM?
The dataset worker pod is killed (OOMKilled) for:

```
bigscience/P3
Graphcore/gqa-lxmert
echarlaix/gqa-lxmert
```

and the split worker pod is killed (OOMKilled) for:

```
imthanhlv/binhvq_news21_raw / started / train
openclimatefix/nimrod-uk-1km / sample / train/test/validation
PolyAI/minds14 / zh-CN / train
```

With the current jobs management (https://github.com/huggingface/datasets-server/issues/264) the killed jobs remain marked as "STARTED" in the mongo db. If we "cancel" them with

```
kubectl exec datasets-server-prod-admin-79798989fb-scmjw -- make cancel-started-dataset-jobs
kubectl exec datasets-server-prod-admin-79798989fb-scmjw -- make cancel-started-split-jobs
```

they are re-enqueue with the status "WAITING" until they are processed and killed again.

Possibly we should allow up to 3 attempts, for example, maybe increasing the dedicated RAM (see https://github.com/huggingface/datasets-server/issues/264#issuecomment-1158596143). Even so, we cannot have more RAM than the underlying node (eg: 32 GiB on the current nodes) and some datasets will still fail.

In that case, we should mark them as ERROR with a proper error message.
How to best manage the datasets that we cannot process due to RAM?: The dataset worker pod is killed (OOMKilled) for:

```
bigscience/P3
Graphcore/gqa-lxmert
echarlaix/gqa-lxmert
```

and the split worker pod is killed (OOMKilled) for:

```
imthanhlv/binhvq_news21_raw / started / train
openclimatefix/nimrod-uk-1km / sample / train/test/validation
PolyAI/minds14 / zh-CN / train
```

With the current jobs management (https://github.com/huggingface/datasets-server/issues/264) the killed jobs remain marked as "STARTED" in the mongo db. If we "cancel" them with

```
kubectl exec datasets-server-prod-admin-79798989fb-scmjw -- make cancel-started-dataset-jobs
kubectl exec datasets-server-prod-admin-79798989fb-scmjw -- make cancel-started-split-jobs
```

they are re-enqueue with the status "WAITING" until they are processed and killed again.

Possibly we should allow up to 3 attempts, for example, maybe increasing the dedicated RAM (see https://github.com/huggingface/datasets-server/issues/264#issuecomment-1158596143). Even so, we cannot have more RAM than the underlying node (eg: 32 GiB on the current nodes) and some datasets will still fail.

In that case, we should mark them as ERROR with a proper error message.