remove finished jobs
See https://github.com/huggingface/datasets-server/pull/2137#issue-2005097421 for context.
The next waiting job operation could be affected because of useless job records (finished). This PR proposes removing the jobs in SUCCESS, ERROR, and CANCELLED status so that the query can improve performance and profile better indexing approaches.
- A waiting job becomes started (as usual)
- A started job no longer becomes a success/error/cancelled status, the record will be deleted from the job collection
- Since we don't need the finished job records, TTL index is no longer necessary and the same for `finished_at` field
-  In an upcoming PR, I will add metrics for finished jobs but using Prometheus Counter https://github.com/huggingface/datasets-server/pull/2137#issuecomment-1832674005

Depends on:
https://github.com/huggingface/datasets-server/pull/2178
https://github.com/huggingface/datasets-server/pull/2179
remove finished jobs: See https://github.com/huggingface/datasets-server/pull/2137#issue-2005097421 for context.
The next waiting job operation could be affected because of useless job records (finished). This PR proposes removing the jobs in SUCCESS, ERROR, and CANCELLED status so that the query can improve performance and profile better indexing approaches.
- A waiting job becomes started (as usual)
- A started job no longer becomes a success/error/cancelled status, the record will be deleted from the job collection
- Since we don't need the finished job records, TTL index is no longer necessary and the same for `finished_at` field
-  In an upcoming PR, I will add metrics for finished jobs but using Prometheus Counter https://github.com/huggingface/datasets-server/pull/2137#issuecomment-1832674005

Depends on:
https://github.com/huggingface/datasets-server/pull/2178
https://github.com/huggingface/datasets-server/pull/2179