Full Spawning urls scan 
Currently we have urls scan implemented for first 100K rows [here ](https://github.com/huggingface/datasets-server/blob/main/services/worker/src/worker/job_runners/split/opt_in_out_urls_scan_from_streaming.py)
We need to do it for full dataset but there could be some concerns:
- Full scan could take too much time for big datasets (Like [laion](https://huggingface.co/datasets/laion/laion2B-en)) and we have a task configured [here](https://github.com/huggingface/datasets-server/blob/bdda4ed42c3f884995917ae6e7575ef67e338f48/services/worker/src/worker/executor.py#L121) that kills long running jobs that can prevent Full scan to do not finish
- We have a response size validation [here](https://github.com/huggingface/datasets-server/blob/main/services/worker/src/worker/job_runner.py#L470) which can lead to a database error in case of having big content (Too much opt in/out urls) to store

Approach 1:
Run full dataset scan in one single job
Will need the following actions:
- Implement custom timeout configurations to kill "long running jobs" i.e We could have a WORKER_MAX_JOB_DURATION_SECONDS value for split-opt-in-out-urls-scan Job set up to 10 hrs and for other jobs 20 min (as  it is currently)
- Store the content response (or maybe only opt in/out urls list) in an external storage. It was suggested by @severo  to store it on a parquet file (see thread https://github.com/huggingface/datasets-server/pull/1044#discussion_r1173625223). I think it could be a good idea but will need to think where to store the file (Maybe https://datasets-server.huggingface.co/assets? Another branch in each repository like we actually do for parquets https://huggingface.co/datasets/<dataset>/tree/refs%2Fconvert%2Fparquet? )
- In case we dont need yet to show the list of opt in/out urls, we can skip previous step and only store number of urls see sample: https://datasets-server.huggingface.co/opt-in-out-urls-scan?dataset=laion/laion2B-en&config=laion--laion2B-en&split=train

<div class="toolbar"><div class="devtools-separator"></div><div class="devtools-searchbox"><input class="searchBox devtools-filterinput" placeholder="Filter JSON" value=""></div></div><div class="panelContent" id="json-scrolling-panel" tabindex="0">

<div class="toolbar"><div class="devtools-separator"></div><div class="devtools-searchbox"><input class="searchBox devtools-filterinput" placeholder="Filter JSON" value=""></div></div><div class="panelContent" id="json-scrolling-panel" tabindex="0">

key  |  value
-- | --
urls_columns | […]
~~opt_in_urls~~ | []
~~opt_out_urls~~ | […]
num_opt_in_urls | 0
num_opt_out_urls | 1853
num_urls | 100000
num_scanned_rows | 100000
has_urls_columns | true

</div>

</div>

Approach 2:
Run full scan in separated Jobs and store results in separated cache entries
Will need the following actions:
- Add support in current datasets-server architecture to have another level of granularity https://github.com/huggingface/datasets-server/blob/main/libs/libcommon/src/libcommon/processing_graph.py#L12 maybe "batch" that will have an start_row and an end_row or offset and limit. So we scan the dataset by batches and store its result in a cache entry for each batch. At the end we could have a "dataset" aggregated Job that will give us the summarized result of previous responses e.g:

**Batch one:  offset:0 , limit:1000**

<div class="toolbar"><div class="devtools-separator"></div><div class="devtools-searchbox"><input class="searchBox devtools-filterinput" placeholder="Filter JSON" value=""></div></div><div class="panelContent" id="json-scrolling-panel" tabindex="0">

<div class="toolbar"><div class="devtools-separator"></div><div class="devtools-searchbox"><input class="searchBox devtools-filterinput" placeholder="Filter JSON" value=""></div></div><div class="panelContent" id="json-scrolling-panel" tabindex="0">

key  |  value
-- | --
urls_columns | […]
~~opt_in_urls~~ | []
~~opt_out_urls~~ | […]
num_opt_in_urls | 0
num_opt_out_urls | 10
num_urls | 1000
limit | 1000
offset | 0

</div>

</div>

**Batch two:  offset:1000 , limit:1000**

<div class="toolbar"><div class="devtools-separator"></div><div class="devtools-searchbox"><input class="searchBox devtools-filterinput" placeholder="Filter JSON" value=""></div></div><div class="panelContent" id="json-scrolling-panel" tabindex="0">

<div class="toolbar"><div class="devtools-separator"></div><div class="devtools-searchbox"><input class="searchBox devtools-filterinput" placeholder="Filter JSON" value=""></div></div><div class="panelContent" id="json-scrolling-panel" tabindex="0">

key  |  value
-- | --
urls_columns | […]
~~opt_in_urls~~ | []
~~opt_out_urls~~ | […]
num_opt_in_urls | 4
num_opt_out_urls | 5
num_urls | 50
limit | 1000
offset | 1000

</div>

</div>
`
Aggregated response:
`
<div class="toolbar"><div class="devtools-separator"></div><div class="devtools-searchbox"><input class="searchBox devtools-filterinput" placeholder="Filter JSON" value=""></div></div><div class="panelContent" id="json-scrolling-panel" tabindex="0">

<div class="toolbar"><div class="devtools-separator"></div><div class="devtools-searchbox"><input class="searchBox devtools-filterinput" placeholder="Filter JSON" value=""></div></div><div class="panelContent" id="json-scrolling-panel" tabindex="0">

key  |  value
-- | --
urls_columns | […]
~~opt_in_urls~~ | []
~~opt_out_urls~~ | […]
num_opt_in_urls | 4 (0 + 4)
num_opt_out_urls | 15 (10 + 5) 
num_urls | 1050 = (1000 from batch one and 50 from batch two)

</div>

</div>

- The only problem with this approach is that we could still have sometimes issues with db storage limitation (e.g a few opt in/out urls but really long text)
 


Full Spawning urls scan : Currently we have urls scan implemented for first 100K rows [here ](https://github.com/huggingface/datasets-server/blob/main/services/worker/src/worker/job_runners/split/opt_in_out_urls_scan_from_streaming.py)
We need to do it for full dataset but there could be some concerns:
- Full scan could take too much time for big datasets (Like [laion](https://huggingface.co/datasets/laion/laion2B-en)) and we have a task configured [here](https://github.com/huggingface/datasets-server/blob/bdda4ed42c3f884995917ae6e7575ef67e338f48/services/worker/src/worker/executor.py#L121) that kills long running jobs that can prevent Full scan to do not finish
- We have a response size validation [here](https://github.com/huggingface/datasets-server/blob/main/services/worker/src/worker/job_runner.py#L470) which can lead to a database error in case of having big content (Too much opt in/out urls) to store

Approach 1:
Run full dataset scan in one single job
Will need the following actions:
- Implement custom timeout configurations to kill "long running jobs" i.e We could have a WORKER_MAX_JOB_DURATION_SECONDS value for split-opt-in-out-urls-scan Job set up to 10 hrs and for other jobs 20 min (as  it is currently)
- Store the content response (or maybe only opt in/out urls list) in an external storage. It was suggested by @severo  to store it on a parquet file (see thread https://github.com/huggingface/datasets-server/pull/1044#discussion_r1173625223). I think it could be a good idea but will need to think where to store the file (Maybe https://datasets-server.huggingface.co/assets? Another branch in each repository like we actually do for parquets https://huggingface.co/datasets/<dataset>/tree/refs%2Fconvert%2Fparquet? )
- In case we dont need yet to show the list of opt in/out urls, we can skip previous step and only store number of urls see sample: https://datasets-server.huggingface.co/opt-in-out-urls-scan?dataset=laion/laion2B-en&config=laion--laion2B-en&split=train

<div class="toolbar"><div class="devtools-separator"></div><div class="devtools-searchbox"><input class="searchBox devtools-filterinput" placeholder="Filter JSON" value=""></div></div><div class="panelContent" id="json-scrolling-panel" tabindex="0">

<div class="toolbar"><div class="devtools-separator"></div><div class="devtools-searchbox"><input class="searchBox devtools-filterinput" placeholder="Filter JSON" value=""></div></div><div class="panelContent" id="json-scrolling-panel" tabindex="0">

key  |  value
-- | --
urls_columns | […]
~~opt_in_urls~~ | []
~~opt_out_urls~~ | […]
num_opt_in_urls | 0
num_opt_out_urls | 1853
num_urls | 100000
num_scanned_rows | 100000
has_urls_columns | true

</div>

</div>

Approach 2:
Run full scan in separated Jobs and store results in separated cache entries
Will need the following actions:
- Add support in current datasets-server architecture to have another level of granularity https://github.com/huggingface/datasets-server/blob/main/libs/libcommon/src/libcommon/processing_graph.py#L12 maybe "batch" that will have an start_row and an end_row or offset and limit. So we scan the dataset by batches and store its result in a cache entry for each batch. At the end we could have a "dataset" aggregated Job that will give us the summarized result of previous responses e.g:

**Batch one:  offset:0 , limit:1000**

<div class="toolbar"><div class="devtools-separator"></div><div class="devtools-searchbox"><input class="searchBox devtools-filterinput" placeholder="Filter JSON" value=""></div></div><div class="panelContent" id="json-scrolling-panel" tabindex="0">

<div class="toolbar"><div class="devtools-separator"></div><div class="devtools-searchbox"><input class="searchBox devtools-filterinput" placeholder="Filter JSON" value=""></div></div><div class="panelContent" id="json-scrolling-panel" tabindex="0">

key  |  value
-- | --
urls_columns | […]
~~opt_in_urls~~ | []
~~opt_out_urls~~ | […]
num_opt_in_urls | 0
num_opt_out_urls | 10
num_urls | 1000
limit | 1000
offset | 0

</div>

</div>

**Batch two:  offset:1000 , limit:1000**

<div class="toolbar"><div class="devtools-separator"></div><div class="devtools-searchbox"><input class="searchBox devtools-filterinput" placeholder="Filter JSON" value=""></div></div><div class="panelContent" id="json-scrolling-panel" tabindex="0">

<div class="toolbar"><div class="devtools-separator"></div><div class="devtools-searchbox"><input class="searchBox devtools-filterinput" placeholder="Filter JSON" value=""></div></div><div class="panelContent" id="json-scrolling-panel" tabindex="0">

key  |  value
-- | --
urls_columns | […]
~~opt_in_urls~~ | []
~~opt_out_urls~~ | […]
num_opt_in_urls | 4
num_opt_out_urls | 5
num_urls | 50
limit | 1000
offset | 1000

</div>

</div>
`
Aggregated response:
`
<div class="toolbar"><div class="devtools-separator"></div><div class="devtools-searchbox"><input class="searchBox devtools-filterinput" placeholder="Filter JSON" value=""></div></div><div class="panelContent" id="json-scrolling-panel" tabindex="0">

<div class="toolbar"><div class="devtools-separator"></div><div class="devtools-searchbox"><input class="searchBox devtools-filterinput" placeholder="Filter JSON" value=""></div></div><div class="panelContent" id="json-scrolling-panel" tabindex="0">

key  |  value
-- | --
urls_columns | […]
~~opt_in_urls~~ | []
~~opt_out_urls~~ | […]
num_opt_in_urls | 4 (0 + 4)
num_opt_out_urls | 15 (10 + 5) 
num_urls | 1050 = (1000 from batch one and 50 from batch two)

</div>

</div>

- The only problem with this approach is that we could still have sometimes issues with db storage limitation (e.g a few opt in/out urls but really long text)
 

