Add toleration to k8s pod config for new GPU nodes
GCP now adds a taint to GPU nodes to prevent non-GPU pods from scheduling on
them, so we need to add an explicit toleration to handle this.
@@ -103,6 +103,11 @@ spec:
value: ${BERT_CACHE}
nodeSelector:
cloud.google.com/gke-accelerator: nvidia-tesla-${GPU_TYPE}
+ tolerations:
+ - key: "nvidia.com/gpu"
+ operator: "Equal"
+ value: "present"
+ effect: "NoSchedule"
volumes:
- name: nfs-jsalt
persistentVolumeClaim:
