See https://github.com/kubernetes-client/python/issues/741

The Kubernetes module establishes some credentials at startup but never refreshes them during the life of the program.

When the credentials expire, (and we start getting

-s), we need to call

again (if that's how we got our credentials) and replace all the API clients.

Alternately, we could re-load the config and re-make the client at every use of Kubernetes, or just reload the config "sufficiently often" and remake the clients on every use, or something.

@glennhickey is seeing problems due to this (I think) on our Kubernetes cluster after an hour.

A workaround is running with service account credentials, which don't expire.

┆Issue is synchronized with a GitHub issue
┆Repository Name: toil
┆Issue Number: 2867