Implement Server-sent events to update the Hub cache
The Hub needs to know which datasets have a viewer, or only a preview. Currently, we publish the /valid endpoint, which returns a list of all the dataset names that has the search capability, the viewer, or just the preview.
It has two drawbacks:
1. it gives information about the gated datasets
2. it does not scale (the queries to the database take too much time, > 10s, even > 20s when the database is under load)

The solution we will implement is to send Server-sent events to the Hub.
Implement Server-sent events to update the Hub cache: The Hub needs to know which datasets have a viewer, or only a preview. Currently, we publish the /valid endpoint, which returns a list of all the dataset names that has the search capability, the viewer, or just the preview.
It has two drawbacks:
1. it gives information about the gated datasets
2. it does not scale (the queries to the database take too much time, > 10s, even > 20s when the database is under load)

The solution we will implement is to send Server-sent events to the Hub.