Monitor the RAM used by the workers
Since https://github.com/huggingface/datasets-server/pull/236/commits/12646ee6b7bd72e7563aeb0c16dfcf08eace9fb8, the workers loop infinitely (instead of being stopped after the first job, then restarted by pm2 or k8s).
This might lead to increased RAM use. We should:
- [ ] monitor the usage
- [ ] check the code for possible memory leaks (note that part of the code that is executed is out of our control: the datasets scripts)

Monitor the RAM used by the workers: Since https://github.com/huggingface/datasets-server/pull/236/commits/12646ee6b7bd72e7563aeb0c16dfcf08eace9fb8, the workers loop infinitely (instead of being stopped after the first job, then restarted by pm2 or k8s).
This might lead to increased RAM use. We should:
- [ ] monitor the usage
- [ ] check the code for possible memory leaks (note that part of the code that is executed is out of our control: the datasets scripts)
