Locking mechanism
Certain jobs like config-parquet-and-info write to a repository and may have concurrency issues (see e.g. https://github.com/huggingface/datasets-server/issues/1349)

For config-parquet-and-info we need to lock the job at the dataset level, so that no other job of the same type for the same dataset write at the same time.

This can be implemented using mongodb
Locking mechanism: Certain jobs like config-parquet-and-info write to a repository and may have concurrency issues (see e.g. https://github.com/huggingface/datasets-server/issues/1349)

For config-parquet-and-info we need to lock the job at the dataset level, so that no other job of the same type for the same dataset write at the same time.

This can be implemented using mongodb