clean duckdb index files
Currently, we share the duckdb  EFS volume for:
- downloads (247 G)
- job runner index processing
![capture_d___e__cran_2023-10-03_a___15 14 59](https://github.com/huggingface/datasets-server/assets/5564745/85057f7d-32d4-4b2b-9b20-820716bad91c)



For downloads folder cleaning, job "delete-indexes" used to do the work but it looks like the volume is being filled because of remaining data when a job runner crashes because of long-running (zombie) and https://github.com/huggingface/datasets-server/blob/main/services/worker/src/worker/job_runners/_job_runner_with_cache.py#L66 is never called.

This PR add will apply same logic as for existing for hf_datasets_cache but:
- will change the subfolder where the job runner indexes are created under /storage/duckdb-index/job_runner
- will apply an expiration time for  storage/duckdb-index/job_runner folder to 3 hours in prod
- will apply an expiration time for  storage/duckdb-index/downloads folder to 3 days in prod
- will reuse logic in hf_datasets_cache for both cleaning (downloads and job_runner folders)

clean duckdb index files: Currently, we share the duckdb  EFS volume for:
- downloads (247 G)
- job runner index processing
![capture_d___e__cran_2023-10-03_a___15 14 59](https://github.com/huggingface/datasets-server/assets/5564745/85057f7d-32d4-4b2b-9b20-820716bad91c)



For downloads folder cleaning, job "delete-indexes" used to do the work but it looks like the volume is being filled because of remaining data when a job runner crashes because of long-running (zombie) and https://github.com/huggingface/datasets-server/blob/main/services/worker/src/worker/job_runners/_job_runner_with_cache.py#L66 is never called.

This PR add will apply same logic as for existing for hf_datasets_cache but:
- will change the subfolder where the job runner indexes are created under /storage/duckdb-index/job_runner
- will apply an expiration time for  storage/duckdb-index/job_runner folder to 3 hours in prod
- will apply an expiration time for  storage/duckdb-index/downloads folder to 3 days in prod
- will reuse logic in hf_datasets_cache for both cleaning (downloads and job_runner folders)
