Better handling of k8s SIGTERM when stopping workers
We've observed recently that worker pods can take more than 1h to get killed by kubernetes (see internal [slack thread](https://huggingface.slack.com/archives/C04L6P8KNQ5/p1697641144036899) for example).
According to the k8s docs, at one point it sends a TERM and then after a grace period a SIGKILL on all the running processes.
If this keeps happening we need to improve the way we handle those to kill the pods faster.

One idea is to catch SIGTERM in the worker executor (main python process of workers) that would stop the worker loop (its subprocess where the actual code runs).
Better handling of k8s SIGTERM when stopping workers: We've observed recently that worker pods can take more than 1h to get killed by kubernetes (see internal [slack thread](https://huggingface.slack.com/archives/C04L6P8KNQ5/p1697641144036899) for example).
According to the k8s docs, at one point it sends a TERM and then after a grace period a SIGKILL on all the running processes.
If this keeps happening we need to improve the way we handle those to kill the pods faster.

One idea is to catch SIGTERM in the worker executor (main python process of workers) that would stop the worker loop (its subprocess where the actual code runs).