Opt-in / opt-out URLs scan with Spawning
Scan the datasets for URLs and get the number of opt-in / opt-out URLs from artists from Spawning.

Right now it's limited to the first 100K rows (runs in a few seconds) but we can have subsequent jobs to run it on the full datasets.

The rows are obtained from streaming in order to support LAION-2B - but later we could just do it on the parquet exports
Opt-in / opt-out URLs scan with Spawning: Scan the datasets for URLs and get the number of opt-in / opt-out URLs from artists from Spawning.

Right now it's limited to the first 100K rows (runs in a few seconds) but we can have subsequent jobs to run it on the full datasets.

The rows are obtained from streaming in order to support LAION-2B - but later we could just do it on the parquet exports