fsspec storage instead of boto3
Related to comment https://github.com/huggingface/datasets-server/pull/1882#issuecomment-1740688443

- Use a storage agnostic tool (fsspec) to store images/audios for assets and cached assets.
- Adding an abstraction layer for "file" and "s3" storage.
- For docker containers, we will use "file" storage but for staging/prod we will use "s3".
- I removed all the dependencies of boto3 and moto (and added s3fs)
- I removed all the dependencies of `/storage/assets` and `/storage/cached-assets` (We don't need it anymore since images and audios are being written directly to the file system)
- I removed the disk metrics (They need to be added again but getting the values from S3, maybe can be handled in another PR)
- Will close https://github.com/huggingface/datasets-server/issues/1887 and https://github.com/huggingface/datasets-server/issues/1885

Pending issue:

- For reverse-proxy, I am not able to get the files stored in /assets and /cached-assets volume (My generated URL is http://localhost:8100/assets/asoria/image/--/666e73a/--/default/train/99/image/image.jpg but it only works when I get http://localhost:8100/assets/assets/asoria/image/--/666e73a/--/default/train/99/image/image.jpg) maybe it is a configuration with the nginx redirect configuration).
fsspec storage instead of boto3: Related to comment https://github.com/huggingface/datasets-server/pull/1882#issuecomment-1740688443

- Use a storage agnostic tool (fsspec) to store images/audios for assets and cached assets.
- Adding an abstraction layer for "file" and "s3" storage.
- For docker containers, we will use "file" storage but for staging/prod we will use "s3".
- I removed all the dependencies of boto3 and moto (and added s3fs)
- I removed all the dependencies of `/storage/assets` and `/storage/cached-assets` (We don't need it anymore since images and audios are being written directly to the file system)
- I removed the disk metrics (They need to be added again but getting the values from S3, maybe can be handled in another PR)
- Will close https://github.com/huggingface/datasets-server/issues/1887 and https://github.com/huggingface/datasets-server/issues/1885

Pending issue:

- For reverse-proxy, I am not able to get the files stored in /assets and /cached-assets volume (My generated URL is http://localhost:8100/assets/asoria/image/--/666e73a/--/default/train/99/image/image.jpg but it only works when I get http://localhost:8100/assets/assets/asoria/image/--/666e73a/--/default/train/99/image/image.jpg) maybe it is a configuration with the nginx redirect configuration).