Parallel steps update incoherence
See the discussion https://huggingface.co/datasets/BelleGroup/multiturn_chat_0.8M/discussions/1#64c9e88a6a26cddbecd9bec6

Before the dataset update, the `split-first-rows-from-parquet` response was a success, and thus the `split-first-rows-from-streaming` response, computed later, is a `ResponseAlreadyComputedError` error.

But after the dataset update, the `split-first-rows-from-parquet` response was an error (due to a disk issue: ` FileSystemError`) and, due to a heavy load on the infra, the `split-first-rows-from-streaming` response has not been processed yet, so: it's still `ResponseAlreadyComputedError`.

Possibilities:
1. remove `ResponseAlreadyComputedError`, and copy the response (doubles storage)
2. change the model for parallel steps, and store only once. Let's say we have M+N parallel steps. If M steps are successful (normally with the same response) and N steps are erroneous, let's store the optional successful response content once, and all the responses, removing the success content for successful responses. It is a lot of complexity.
3. keep the logic, but if a parallel step gives an error whereas it had a successful response before AND the other parallel step is `ResponseAlreadyComputedError`, copy the successful answer to the other step. Seems brittle and overly complex.
4. keep the logic, but if a parallel step gives an error whereas it had a successful response before AND the other parallel step is `ResponseAlreadyComputedError`, delete the other answer

None seems like a good idea. Do you have better ideas @huggingface/datasets-server ?
Parallel steps update incoherence: See the discussion https://huggingface.co/datasets/BelleGroup/multiturn_chat_0.8M/discussions/1#64c9e88a6a26cddbecd9bec6

Before the dataset update, the `split-first-rows-from-parquet` response was a success, and thus the `split-first-rows-from-streaming` response, computed later, is a `ResponseAlreadyComputedError` error.

But after the dataset update, the `split-first-rows-from-parquet` response was an error (due to a disk issue: ` FileSystemError`) and, due to a heavy load on the infra, the `split-first-rows-from-streaming` response has not been processed yet, so: it's still `ResponseAlreadyComputedError`.

Possibilities:
1. remove `ResponseAlreadyComputedError`, and copy the response (doubles storage)
2. change the model for parallel steps, and store only once. Let's say we have M+N parallel steps. If M steps are successful (normally with the same response) and N steps are erroneous, let's store the optional successful response content once, and all the responses, removing the success content for successful responses. It is a lot of complexity.
3. keep the logic, but if a parallel step gives an error whereas it had a successful response before AND the other parallel step is `ResponseAlreadyComputedError`, copy the successful answer to the other step. Seems brittle and overly complex.
4. keep the logic, but if a parallel step gives an error whereas it had a successful response before AND the other parallel step is `ResponseAlreadyComputedError`, delete the other answer

None seems like a good idea. Do you have better ideas @huggingface/datasets-server ?