Self-hosting an instance of HF datasets on premises
Hi, 

Maybe I didn't read correctly, but I was thinking if it's possible to host an instance of the same middleware you use in our research institution so we could host our own datasets in a HF way.


The idea is that I am writing from the biggest research center in Europe, and we produce A LOT of data, and we use a lot of datasets from other sources as well - be it medical and confidential, or anything up to simple stuff like imagenet or common crawl. We host it all.

So far, this has been done in a very haphazard and ad-hoc way - people download everything into a folder visible to the supercomputers.

I would like to change that and do it in an organized way, so I would like to host the whole thing - the web server and the datasets.

Does my question even make sense? I can explain better if not.

If you have parts of your infra which are not open, I totally understand. Just let me know and I will search for an alternative.

Thanks!
Self-hosting an instance of HF datasets on premises: Hi, 

Maybe I didn't read correctly, but I was thinking if it's possible to host an instance of the same middleware you use in our research institution so we could host our own datasets in a HF way.


The idea is that I am writing from the biggest research center in Europe, and we produce A LOT of data, and we use a lot of datasets from other sources as well - be it medical and confidential, or anything up to simple stuff like imagenet or common crawl. We host it all.

So far, this has been done in a very haphazard and ad-hoc way - people download everything into a folder visible to the supercomputers.

I would like to change that and do it in an organized way, so I would like to host the whole thing - the web server and the datasets.

Does my question even make sense? I can explain better if not.

If you have parts of your infra which are not open, I totally understand. Just let me know and I will search for an alternative.

Thanks!