Support private datasets
This PR changes the logic a bit: operations.py replaces dataset.py and is the module that must be used to update or delete a dataset (ie: do not create jobs or cache entries directly or through DatasetOrchestrator). Indeed, operations.py checks if the dataset is supported (public, or private + pro user or enterprise org. Also checks if the repo is disabled, if the viewer is disabled, or if the dataset is in the list of blocked datasets). It also means that behind that layer, we don't check anymore if the dataset is supported. For example, DatasetOrchestrator is now free from these checks.

Remaining:
- [x] fix tests on hub-ci
  - a) the database has not been updated. See https://huggingface.slack.com/archives/C035N5UE2JU/p1703193635197549 (internal)
  - b) the app token has no right to access the private datasets (see https://github.com/huggingface/moon-landing/pull/8573 <- internal)
- [x] add e2e tests about the private datasets (normal, pro, enterprise). In particular: make sure both app tokens have the appropriate rights (read datasets, bypass gates, write in the refs/convert/parquet branch, etc. for private datasets)
- [x] change the logic of the backfill cronjob. Options:
  a) <strike>check support and completeness (backfill) of each public and private dataset on the Hub (requires getting the list of all the datasets). </strike>
  b) only check if the datasets we currently have in the datasets server are still supported and are complete (backfill). If a dataset is missing, we will create it when we receive the first request on an API endpoint for it. <- let's do this
- [x] change the logic of the obsolete cache cron job (it's currently disabled, see TODO in the code)
- [x] replace the assets/cached-assets URLs with S3 signed URLs (when the configured storage is S3). We will have to change a bit how we handle /first-rows, since it's the only endpoint that returns rows that is not computed on the fly. See https://github.com/huggingface/datasets-server/issues/2280
  - [x] cached-assets: done
  - [x] assets: see https://github.com/huggingface/datasets-server/issues/2280
    - [ ] change the configuration to restrict access to `/assets/*` to signed URLs.  
- [ ] change the settings of the webhook, to receive webhooks on the private datasets too. Otherwise, we will not update the dataset.

Not crucial:
- [x] when we finish a job: should we ensure the dataset still exists and is still supported? Not important: in the worst case, it will be cleaned in the next backfill cron job.
- [x] maybe? cache the info about the supported datasets to avoid calling the Hub all the time? See https://github.com/huggingface/datasets-server/issues/2208 for a proposal. -> I think it's not required.

Unrelated to this PR:
- when we delete a dataset, we should also delete the files (assets, cached_assets, parquet files, etc.) ->
  - [x] done for assets and cached_assets.
  - [x] invalidate the cloudfront cache? no need: the URLs are valid only for 1h...
  - [ ] The parquet files and duckdb index would be deleted if the repo has disappeared, but not if it went "unsupported" (public -> private, disabled repo, disabled viewer, or end of pro/enterprise subscription for private dataset).
  - [ ] the parquet metadata is not deleted either

Note that the following PR on moonlanding should not be needed anymore: https://github.com/huggingface/moon-landing/pull/8262
Support private datasets: This PR changes the logic a bit: operations.py replaces dataset.py and is the module that must be used to update or delete a dataset (ie: do not create jobs or cache entries directly or through DatasetOrchestrator). Indeed, operations.py checks if the dataset is supported (public, or private + pro user or enterprise org. Also checks if the repo is disabled, if the viewer is disabled, or if the dataset is in the list of blocked datasets). It also means that behind that layer, we don't check anymore if the dataset is supported. For example, DatasetOrchestrator is now free from these checks.

Remaining:
- [x] fix tests on hub-ci
  - a) the database has not been updated. See https://huggingface.slack.com/archives/C035N5UE2JU/p1703193635197549 (internal)
  - b) the app token has no right to access the private datasets (see https://github.com/huggingface/moon-landing/pull/8573 <- internal)
- [x] add e2e tests about the private datasets (normal, pro, enterprise). In particular: make sure both app tokens have the appropriate rights (read datasets, bypass gates, write in the refs/convert/parquet branch, etc. for private datasets)
- [x] change the logic of the backfill cronjob. Options:
  a) <strike>check support and completeness (backfill) of each public and private dataset on the Hub (requires getting the list of all the datasets). </strike>
  b) only check if the datasets we currently have in the datasets server are still supported and are complete (backfill). If a dataset is missing, we will create it when we receive the first request on an API endpoint for it. <- let's do this
- [x] change the logic of the obsolete cache cron job (it's currently disabled, see TODO in the code)
- [x] replace the assets/cached-assets URLs with S3 signed URLs (when the configured storage is S3). We will have to change a bit how we handle /first-rows, since it's the only endpoint that returns rows that is not computed on the fly. See https://github.com/huggingface/datasets-server/issues/2280
  - [x] cached-assets: done
  - [x] assets: see https://github.com/huggingface/datasets-server/issues/2280
    - [ ] change the configuration to restrict access to `/assets/*` to signed URLs.  
- [ ] change the settings of the webhook, to receive webhooks on the private datasets too. Otherwise, we will not update the dataset.

Not crucial:
- [x] when we finish a job: should we ensure the dataset still exists and is still supported? Not important: in the worst case, it will be cleaned in the next backfill cron job.
- [x] maybe? cache the info about the supported datasets to avoid calling the Hub all the time? See https://github.com/huggingface/datasets-server/issues/2208 for a proposal. -> I think it's not required.

Unrelated to this PR:
- when we delete a dataset, we should also delete the files (assets, cached_assets, parquet files, etc.) ->
  - [x] done for assets and cached_assets.
  - [x] invalidate the cloudfront cache? no need: the URLs are valid only for 1h...
  - [ ] The parquet files and duckdb index would be deleted if the repo has disappeared, but not if it went "unsupported" (public -> private, disabled repo, disabled viewer, or end of pro/enterprise subscription for private dataset).
  - [ ] the parquet metadata is not deleted either

Note that the following PR on moonlanding should not be needed anymore: https://github.com/huggingface/moon-landing/pull/8262