what happened to the pods?
```
$ k get pods -w
...
datasets-server-prod-datasets-worker-776b774978-g7mpk   1/1     Evicted   0             73m                                             │DEBUG: 2022-06-16 18:42:46,966 - datasets_server.worker - try to process a split job
datasets-server-prod-datasets-worker-776b774978-cdb4b   0/1     Pending   0             1s                                              │DEBUG: 2022-06-16 18:42:47,011 - datasets_server.worker - job assigned: 62ab6804a502851c834d7e43 for split 'test' from dataset 'luozhou
datasets-server-prod-datasets-worker-776b774978-cdb4b   0/1     Pending   0             1s                                              │yang/dureader' with config 'robust'
datasets-server-prod-datasets-worker-776b774978-cdb4b   0/1     OutOfmemory   0             1s                                          │INFO: 2022-06-16 18:42:47,012 - datasets_server.worker - compute split 'test' from dataset 'luozhouyang/dureader' with config 'robust'
datasets-server-prod-datasets-worker-776b774978-7hw4j   0/1     Pending       0             0s                                          │Downloading builder script: 100%|██████████| 8.67k/8.67k [00:00<00:00, 4.85MB/s]
datasets-server-prod-datasets-worker-776b774978-7hw4j   0/1     Pending       0             0s                                          │Downloading metadata: 100%|██████████| 2.85k/2.85k [00:00<00:00, 1.43MB/s]
datasets-server-prod-datasets-worker-776b774978-7hw4j   0/1     OutOfmemory   0             0s                                          │Downloading builder script: 100%|██████████| 8.67k/8.67k [00:00<00:00, 5.07MB/s]
datasets-server-prod-datasets-worker-776b774978-qtmtd   0/1     Pending       0             0s                                          │Downloading metadata: 100%|██████████| 2.85k/2.85k [00:00<00:00, 1.18MB/s]
datasets-server-prod-datasets-worker-776b774978-qtmtd   0/1     Pending       0             0s                                          │Downloading builder script: 100%|██████████| 8.67k/8.67k [00:00<00:00, 4.52MB/s]
datasets-server-prod-datasets-worker-776b774978-qtmtd   0/1     OutOfmemory   0             0s                                          │Downloading metadata: 100%|██████████| 2.85k/2.85k [00:00<00:00, 1.76MB/s]
datasets-server-prod-datasets-worker-776b774978-54zr6   0/1     Pending       0             0s                                          │Downloading and preparing dataset dureader/robust (download: 19.57 MiB, generated: 57.84 MiB, post-processed: Unknown size, total: 77.4
datasets-server-prod-datasets-worker-776b774978-54zr6   0/1     Pending       0             0s                                          │1 MiB) to /cache/datasets/luozhouyang___dureader/robust/1.0.0/bdab4855e88c197f2297db78cfc86259fb874c2b977134bbe80d3af8616f33b1...
datasets-server-prod-datasets-worker-776b774978-54zr6   0/1     OutOfmemory   0             0s                                          │Downloading data:   1%|          | 163k/20.5M [01:45<3:40:25, 1.54kB/s]
datasets-server-prod-datasets-worker-776b774978-rxcb2   0/1     Pending       0             0s                                          │DEBUG: 2022-06-16 18:44:44,235 - datasets_server.worker - job finished with error: 62ab6804a502851c834d7e43 for split 'test' from datas
datasets-server-prod-datasets-worker-776b774978-rxcb2   0/1     Pending       0             0s                                          │et 'luozhouyang/dureader' with config 'robust'
datasets-server-prod-datasets-worker-776b774978-rxcb2   0/1     OutOfmemory   0             0s                                          │DEBUG: 2022-06-16 18:44:44,236 - datasets_server.worker - try to process a split job
datasets-server-prod-datasets-worker-776b774978-d8m42   0/1     Pending       0             0s                                          │DEBUG: 2022-06-16 18:44:44,281 - datasets_server.worker - job assigned: 62ab6804a502851c834d7e45 for split 'test' from dataset 'opencli
datasets-server-prod-datasets-worker-776b774978-d8m42   0/1     Pending       0             0s                                          │matefix/nimrod-uk-1km' with config 'sample'
datasets-server-prod-datasets-worker-776b774978-d8m42   0/1     OutOfmemory   0             0s                                          │INFO: 2022-06-16 18:44:44,281 - datasets_server.worker - compute split 'test' from dataset 'openclimatefix/nimrod-uk-1km' with config '
datasets-server-prod-datasets-worker-776b774978-xx7hv   0/1     Pending       0             0s                                          │sample'
datasets-server-prod-datasets-worker-776b774978-xx7hv   0/1     Pending       0             0s                                          │Downloading builder script: 100%|██████████| 15.2k/15.2k [00:00<00:00, 6.04MB/s]
datasets-server-prod-datasets-worker-776b774978-xx7hv   0/1     OutOfmemory   0             1s                                          │Downloading builder script: 100%|██████████| 15.2k/15.2k [00:00<00:00, 7.65MB/s]
datasets-server-prod-datasets-worker-776b774978-x7xzb   0/1     Pending       0             0s                                          │2022-06-16 18:44:46.305062: W tensorflow/core/platform/cloud/google_auth_provider.cc:184] All attempts to get a Google authentication b
datasets-server-prod-datasets-worker-776b774978-x7xzb   0/1     Pending       0             0s                                          │earer token failed, returning an empty token. Retrieving token from files failed with "NOT_FOUND: Could not locate the credentials file
datasets-server-prod-datasets-worker-776b774978-x7xzb   0/1     OutOfmemory   0             0s                                          │.". Retrieving token from GCE failed with "FAILED_PRECONDITION: Error executing an HTTP request: libcurl code 6 meaning 'Couldn't resol
datasets-server-prod-datasets-worker-776b774978-m5dqs   0/1     Pending       0             0s                                          │ve host name', error details: Could not resolve host: metadata".
datasets-server-prod-datasets-worker-776b774978-m5dqs   0/1     Pending       0             0s                                          │2022-06-16 18:44:46.389820: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.
datasets-server-prod-datasets-worker-776b774978-m5dqs   0/1     Init:0/3      0             0s                                          │1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
datasets-server-prod-datasets-worker-776b774978-g7mpk   0/1     Error         0             73m                                         │2022-06-16 18:44:46.389865: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
datasets-server-prod-datasets-worker-776b774978-m5dqs   0/1     Init:1/3      0             3s                                          │2022-06-16 18:44:46.390005: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on t
datasets-server-prod-datasets-worker-776b774978-m5dqs   0/1     Init:2/3      0             4s
```
what happened to the pods?: ```
$ k get pods -w
...
datasets-server-prod-datasets-worker-776b774978-g7mpk   1/1     Evicted   0             73m                                             │DEBUG: 2022-06-16 18:42:46,966 - datasets_server.worker - try to process a split job
datasets-server-prod-datasets-worker-776b774978-cdb4b   0/1     Pending   0             1s                                              │DEBUG: 2022-06-16 18:42:47,011 - datasets_server.worker - job assigned: 62ab6804a502851c834d7e43 for split 'test' from dataset 'luozhou
datasets-server-prod-datasets-worker-776b774978-cdb4b   0/1     Pending   0             1s                                              │yang/dureader' with config 'robust'
datasets-server-prod-datasets-worker-776b774978-cdb4b   0/1     OutOfmemory   0             1s                                          │INFO: 2022-06-16 18:42:47,012 - datasets_server.worker - compute split 'test' from dataset 'luozhouyang/dureader' with config 'robust'
datasets-server-prod-datasets-worker-776b774978-7hw4j   0/1     Pending       0             0s                                          │Downloading builder script: 100%|██████████| 8.67k/8.67k [00:00<00:00, 4.85MB/s]
datasets-server-prod-datasets-worker-776b774978-7hw4j   0/1     Pending       0             0s                                          │Downloading metadata: 100%|██████████| 2.85k/2.85k [00:00<00:00, 1.43MB/s]
datasets-server-prod-datasets-worker-776b774978-7hw4j   0/1     OutOfmemory   0             0s                                          │Downloading builder script: 100%|██████████| 8.67k/8.67k [00:00<00:00, 5.07MB/s]
datasets-server-prod-datasets-worker-776b774978-qtmtd   0/1     Pending       0             0s                                          │Downloading metadata: 100%|██████████| 2.85k/2.85k [00:00<00:00, 1.18MB/s]
datasets-server-prod-datasets-worker-776b774978-qtmtd   0/1     Pending       0             0s                                          │Downloading builder script: 100%|██████████| 8.67k/8.67k [00:00<00:00, 4.52MB/s]
datasets-server-prod-datasets-worker-776b774978-qtmtd   0/1     OutOfmemory   0             0s                                          │Downloading metadata: 100%|██████████| 2.85k/2.85k [00:00<00:00, 1.76MB/s]
datasets-server-prod-datasets-worker-776b774978-54zr6   0/1     Pending       0             0s                                          │Downloading and preparing dataset dureader/robust (download: 19.57 MiB, generated: 57.84 MiB, post-processed: Unknown size, total: 77.4
datasets-server-prod-datasets-worker-776b774978-54zr6   0/1     Pending       0             0s                                          │1 MiB) to /cache/datasets/luozhouyang___dureader/robust/1.0.0/bdab4855e88c197f2297db78cfc86259fb874c2b977134bbe80d3af8616f33b1...
datasets-server-prod-datasets-worker-776b774978-54zr6   0/1     OutOfmemory   0             0s                                          │Downloading data:   1%|          | 163k/20.5M [01:45<3:40:25, 1.54kB/s]
datasets-server-prod-datasets-worker-776b774978-rxcb2   0/1     Pending       0             0s                                          │DEBUG: 2022-06-16 18:44:44,235 - datasets_server.worker - job finished with error: 62ab6804a502851c834d7e43 for split 'test' from datas
datasets-server-prod-datasets-worker-776b774978-rxcb2   0/1     Pending       0             0s                                          │et 'luozhouyang/dureader' with config 'robust'
datasets-server-prod-datasets-worker-776b774978-rxcb2   0/1     OutOfmemory   0             0s                                          │DEBUG: 2022-06-16 18:44:44,236 - datasets_server.worker - try to process a split job
datasets-server-prod-datasets-worker-776b774978-d8m42   0/1     Pending       0             0s                                          │DEBUG: 2022-06-16 18:44:44,281 - datasets_server.worker - job assigned: 62ab6804a502851c834d7e45 for split 'test' from dataset 'opencli
datasets-server-prod-datasets-worker-776b774978-d8m42   0/1     Pending       0             0s                                          │matefix/nimrod-uk-1km' with config 'sample'
datasets-server-prod-datasets-worker-776b774978-d8m42   0/1     OutOfmemory   0             0s                                          │INFO: 2022-06-16 18:44:44,281 - datasets_server.worker - compute split 'test' from dataset 'openclimatefix/nimrod-uk-1km' with config '
datasets-server-prod-datasets-worker-776b774978-xx7hv   0/1     Pending       0             0s                                          │sample'
datasets-server-prod-datasets-worker-776b774978-xx7hv   0/1     Pending       0             0s                                          │Downloading builder script: 100%|██████████| 15.2k/15.2k [00:00<00:00, 6.04MB/s]
datasets-server-prod-datasets-worker-776b774978-xx7hv   0/1     OutOfmemory   0             1s                                          │Downloading builder script: 100%|██████████| 15.2k/15.2k [00:00<00:00, 7.65MB/s]
datasets-server-prod-datasets-worker-776b774978-x7xzb   0/1     Pending       0             0s                                          │2022-06-16 18:44:46.305062: W tensorflow/core/platform/cloud/google_auth_provider.cc:184] All attempts to get a Google authentication b
datasets-server-prod-datasets-worker-776b774978-x7xzb   0/1     Pending       0             0s                                          │earer token failed, returning an empty token. Retrieving token from files failed with "NOT_FOUND: Could not locate the credentials file
datasets-server-prod-datasets-worker-776b774978-x7xzb   0/1     OutOfmemory   0             0s                                          │.". Retrieving token from GCE failed with "FAILED_PRECONDITION: Error executing an HTTP request: libcurl code 6 meaning 'Couldn't resol
datasets-server-prod-datasets-worker-776b774978-m5dqs   0/1     Pending       0             0s                                          │ve host name', error details: Could not resolve host: metadata".
datasets-server-prod-datasets-worker-776b774978-m5dqs   0/1     Pending       0             0s                                          │2022-06-16 18:44:46.389820: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.
datasets-server-prod-datasets-worker-776b774978-m5dqs   0/1     Init:0/3      0             0s                                          │1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
datasets-server-prod-datasets-worker-776b774978-g7mpk   0/1     Error         0             73m                                         │2022-06-16 18:44:46.389865: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
datasets-server-prod-datasets-worker-776b774978-m5dqs   0/1     Init:1/3      0             3s                                          │2022-06-16 18:44:46.390005: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on t
datasets-server-prod-datasets-worker-776b774978-m5dqs   0/1     Init:2/3      0             4s
```