Check if shared /cache is an issue for the workers
All the workers (in the kubernetes infra) share the same `datasets` library directory, both for the data and the modules. We must check if this shared access in read/write mode can lead to inconsistencies.
The alternative would be to create an empty cache for every new worker.
Check if shared /cache is an issue for the workers: All the workers (in the kubernetes infra) share the same `datasets` library directory, both for the data and the modules. We must check if this shared access in read/write mode can lead to inconsistencies.
The alternative would be to create an empty cache for every new worker.