Merge the workers that rely on the datasets library
The docker image size for a worker based on the datasets library is large, mainly due to two dependencies: PyTorch and TensorFlow (about 4GB).
We create /workers/datasets_based for all the workers that depend on datasets and chose the processing step with the `DATASETS_BASED_ENDPOINT` env var.
Merge the workers that rely on the datasets library: The docker image size for a worker based on the datasets library is large, mainly due to two dependencies: PyTorch and TensorFlow (about 4GB).
We create /workers/datasets_based for all the workers that depend on datasets and chose the processing step with the `DATASETS_BASED_ENDPOINT` env var.