adding revision for assets creation path
Fix for https://github.com/huggingface/datasets-server/issues/1981
As suggested by @lhoestq in https://github.com/huggingface/datasets-server/issues/1981#issuecomment-1764559346 , adding dataset version (revision) as part of assets path:
- Removing old code to periodically clean cached-assets storage
- Add revision as part of the asset path {dataset}-{revision}-{config}-{split} if no revision, default is "main" same as in https://github.com/huggingface/datasets-server/blob/main/jobs/mongodb_migration/src/mongodb_migration/migrations/_20230516101500_queue_job_add_revision.py#L22
- For cached-assets, we dont need to delete previous version since they will be automatically removed by the bucket TTL policy (After 1 day of inactivity)
- We can keep overwrite=False in /search, /fiter and /rows since a new version will be created in another folder
- For previous assets I think there is no problem with backwards compatibility since we stored the file URL in the db

Open questions for upcoming PRs:
- Since all our cache records in [cachedResponsesBlue](https://github.com/huggingface/datasets-server/blob/main/libs/libcommon/src/libcommon/simple_cache.py#L129) collection have a revision value, should we change it to a required and non Optional same as in [Jobs](https://github.com/huggingface/datasets-server/blob/main/libs/libcommon/src/libcommon/queue.py#L178)? -> Done in https://github.com/huggingface/datasets-server/pull/1993
- For assets, if a new version is processed, we will be storing the previous files in S3, we need to find a way to clean them or, maybe in https://github.com/huggingface/datasets-server/issues/1823 it will make more sense since a "versioned" approach was proposed.

adding revision for assets creation path: Fix for https://github.com/huggingface/datasets-server/issues/1981
As suggested by @lhoestq in https://github.com/huggingface/datasets-server/issues/1981#issuecomment-1764559346 , adding dataset version (revision) as part of assets path:
- Removing old code to periodically clean cached-assets storage
- Add revision as part of the asset path {dataset}-{revision}-{config}-{split} if no revision, default is "main" same as in https://github.com/huggingface/datasets-server/blob/main/jobs/mongodb_migration/src/mongodb_migration/migrations/_20230516101500_queue_job_add_revision.py#L22
- For cached-assets, we dont need to delete previous version since they will be automatically removed by the bucket TTL policy (After 1 day of inactivity)
- We can keep overwrite=False in /search, /fiter and /rows since a new version will be created in another folder
- For previous assets I think there is no problem with backwards compatibility since we stored the file URL in the db

Open questions for upcoming PRs:
- Since all our cache records in [cachedResponsesBlue](https://github.com/huggingface/datasets-server/blob/main/libs/libcommon/src/libcommon/simple_cache.py#L129) collection have a revision value, should we change it to a required and non Optional same as in [Jobs](https://github.com/huggingface/datasets-server/blob/main/libs/libcommon/src/libcommon/queue.py#L178)? -> Done in https://github.com/huggingface/datasets-server/pull/1993
- For assets, if a new version is processed, we will be storing the previous files in S3, we need to find a way to clean them or, maybe in https://github.com/huggingface/datasets-server/issues/1823 it will make more sense since a "versioned" approach was proposed.
