Create the prod infrastructure on Kubernetes
- [x] 4 nodes (4 machines t2 2xlarge, 8 vcpu, 32 gb ram)
- [x] NFS -> 4TB - To increase the size later: directly on AWS, eg https://us-east-1.console.aws.amazon.com/fsx/home?region=us-east-1#file-system-details/fs-097afa9688029b62a (terraform does not allow to change the size of a storage once created, to avoid deleting data by error). Also: see "Monitoring" on the same page to monitor and alert in case of low available storage. Alert: https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#alarmsV2:alarm/Low+disk+on+datasets+server?
- [x] Mongo (autoscale, size: 1GB to start) - URL in a secret: `mongodb-url`, with env var `MONGO_URL`: https://cloud.mongodb.com/v2/6239e2417155de3d798e9187#clusters/detail/datasets-server-prod. Beware: it uses a URL with `mongodb+srv://` which was not supported by the code -> fixed here: #263
- [x] secret `hf-token` with env var `HF_TOKEN`
- [x] domain `datasets-server.huggingface.tech`
- [x] own namespace: `datasets-server`
- [x] selector: `role-datasets-server`. Cluster de prod. See https://github.com/huggingface/tensorboard-launcher/blob/03a8c7c8d2f23fe21ec36c7a83ae2a5e7f7a3833/kube/env/prod.yaml#L22-L28
- [x] create the prod environment in the helm chart. resources: ensure the API and nginx have always reserved resources, by setting limit and request to the same value (1?). For the worker: spawn 6? workers per node, ie 24 in total, with resources: limit=1, request=0.01.
Create the prod infrastructure on Kubernetes: - [x] 4 nodes (4 machines t2 2xlarge, 8 vcpu, 32 gb ram)
- [x] NFS -> 4TB - To increase the size later: directly on AWS, eg https://us-east-1.console.aws.amazon.com/fsx/home?region=us-east-1#file-system-details/fs-097afa9688029b62a (terraform does not allow to change the size of a storage once created, to avoid deleting data by error). Also: see "Monitoring" on the same page to monitor and alert in case of low available storage. Alert: https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#alarmsV2:alarm/Low+disk+on+datasets+server?
- [x] Mongo (autoscale, size: 1GB to start) - URL in a secret: `mongodb-url`, with env var `MONGO_URL`: https://cloud.mongodb.com/v2/6239e2417155de3d798e9187#clusters/detail/datasets-server-prod. Beware: it uses a URL with `mongodb+srv://` which was not supported by the code -> fixed here: #263
- [x] secret `hf-token` with env var `HF_TOKEN`
- [x] domain `datasets-server.huggingface.tech`
- [x] own namespace: `datasets-server`
- [x] selector: `role-datasets-server`. Cluster de prod. See https://github.com/huggingface/tensorboard-launcher/blob/03a8c7c8d2f23fe21ec36c7a83ae2a5e7f7a3833/kube/env/prod.yaml#L22-L28
- [x] create the prod environment in the helm chart. resources: ensure the API and nginx have always reserved resources, by setting limit and request to the same value (1?). For the worker: spawn 6? workers per node, ie 24 in total, with resources: limit=1, request=0.01.