Remove the parallel steps?
I think it would be simpler to delete the concept of "parallel" steps.

Currently, we have two pairs of parallel steps:

1. `config-split-names-from-streaming` and `config-split-names-from-info`
2. `split-first-rows-from-streaming` and `split-first-rows-from-parquet`

It generates some complications:
- we don't want to store them both, in particular, the first-rows that have the heaviest content in the database
- the responses can be incoherent (see https://github.com/huggingface/datasets-server/issues/2460)
- it implies specific handling in the code (see `parallel_step_names` in the code)

The parallel steps are currently used to try to get a quick response from streaming the dataset, when it's faster than converting to parquet. But we should always trust the parquet ones more than the streaming ones. Also: computing from parquet (once we have parquet files) is super fast, while it can take some time from streaming.

So, I propose:

- to remove the concept of parallel steps (and raise an error if a processing graph has parallel steps)
- to have a conditional in the unique implementation of `config-split-names` and `split-first-rows`: use parquet/info preferentially, else use streaming
- (note: we have to ensure we don't have concurrency issue, and we always end up with the parquet version when possible, not the streaming one)

What do you think @huggingface/datasets-server?
Remove the parallel steps?: I think it would be simpler to delete the concept of "parallel" steps.

Currently, we have two pairs of parallel steps:

1. `config-split-names-from-streaming` and `config-split-names-from-info`
2. `split-first-rows-from-streaming` and `split-first-rows-from-parquet`

It generates some complications:
- we don't want to store them both, in particular, the first-rows that have the heaviest content in the database
- the responses can be incoherent (see https://github.com/huggingface/datasets-server/issues/2460)
- it implies specific handling in the code (see `parallel_step_names` in the code)

The parallel steps are currently used to try to get a quick response from streaming the dataset, when it's faster than converting to parquet. But we should always trust the parquet ones more than the streaming ones. Also: computing from parquet (once we have parquet files) is super fast, while it can take some time from streaming.

So, I propose:

- to remove the concept of parallel steps (and raise an error if a processing graph has parallel steps)
- to have a conditional in the unique implementation of `config-split-names` and `split-first-rows`: use parquet/info preferentially, else use streaming
- (note: we have to ensure we don't have concurrency issue, and we always end up with the parquet version when possible, not the streaming one)

What do you think @huggingface/datasets-server?