How To Determine Min/Max Resources For Kubernetes Pods?

Say I have 2 nodes for a kubernetes cluster. Both has

2 CPU → 4 Total
2G RAM → 4 Total

I have 4 pods to shcedule-- no replicas yet, 1 major pod that'll use 80% of resources, other use 5% each.
What resource limits should I specify for these pods?

CURENT USAGE STATS

# kubectl top pod --all-namespaces
NAMESPACE       NAME                        CPU(cores)   MEMORY(bytes)
default         pod1-64c455bcf8-mzxqc           1m          122Mi           
default         pod2-6f7b44478b-pn4tt           1m          33Mi            
default         pod3-867c956697-mf6zw           0m          48Mi            
default         pod4-844bd559dd-z96fh           0m          35Mi

The DevOps toolkit 2.3 book by Victor Farcic goes into some detail on this.

What You Should Understand

• You as a developer specify whatever requests/limits you like initially.

• These values have no baring on reality.

• In order to get "real" values, you must monitor your application's real resource usage.

• This is best done through a monitoring tool like prometheus, instana, datadog etc.

• E.g. in instana, I can see that I have 512MB set up for a pod's requests, 2G for its limits, but the pod only has one container currently taking 57MB.

• But I can't just use 60MB as a value now... I need to load test my application and see how it behaves under real/stressed circumstances.

• Then you review these values and you can adjust the requests/limits accordingly.

Determine Usage Without a Monitoring Tool

You can use some kubectl commands to get these values as a one off (it's annoying at scale). Here is an example of a different app I have running.

Notice that requests/limits are set to 1/4-2 CPUs and 4G-8G memory. But the container is really using almost no CPU and 1.198G memory. If I load tested this and those values did not move up, I would need to change them. But I have load tested it and know that they would go up towards these realistic requests/limits, so it's okay.

$ kubectl describe pod -n gazelle gazelle-ws-0-0-1-6d67cffcf-lvlhk

...
    Limits:
      cpu:     2
      memory:  8Gi
    Requests:
      cpu:      250m
      memory:   4Gi

$ kubectl top pod gazelle-ws-0-0-1-6d67cffcf-lvlhk -n gazelle --containers

POD                                NAME         CPU(cores)   MEMORY(bytes)   
gazelle-ws-0-0-1-6d67cffcf-lvlhk   gazelle-ws   5m           1198Mi

StackExchange DevOps Q#10913, answer score: 1