Keeping graphs of terminated Kubernetes pods in Prometheus/Grafana

I love using Prometheus/Grafana to monitor and tune resource usage on my Kubernetes clusters. When pods are terminated, however, the graphs are disposed of it seems.

I am looking for a solution to this problem.

For informed decisions on how to tune resource requests I need history to be kept.

Does anybody know how to keep graphs for deleted Kubernetes pods in Prometheus/Grafana.

I am currently using this deployment:
https://itnext.io/kubernetes-monitoring-with-prometheus-in-15-minutes-8e54d1de2e13

The metrics for deleted pods still exists on the Prometheus Time Series Database. The problem is that the Grafana dashboards imported from kubernetes-mixin create the pod list when loading the dashboards.

To change this in Grafana, open the Variables menu, then change $pod variable to refresh On Time Range Change:

Go back to the panels and change the time range (e.g., to "This Month"). The deleted pods for that period should be listed in the dropdown menu.

StackExchange DevOps Q#6308, answer score: 5