Questions about using pprof to obtain information related to k3s

[YYM0093]

Is your feature request related to a problem? Please describe.

I am exploring the performance of the k3s cluster, and I use the pprof tool for data collection. My work steps are as follows:

Start the proxy:

kubectl proxy --address='0.0.0.0' --accept-hosts='^*$'

Obtain performance metrics (I used the following two commands and got almost identical results):

go tool pprof http://127.0.0.1:8001/debug/pprof/heap

go tool pprof http://127.0.0.1:8001/api/v1/nodes/${NODENAME}/proxy/debug/pprof/heap

Describe the solution you'd like

I use the top command to obtain the memory usage of the entire k3s cluster, and I want to further analyze the memory usage of the internal components of k3s.

I would like to know the answers to the following questions:

1. Why do I get similar memory usage information through two different methods? If I want to analyze the performance of kubelet specifically, what is the correct approach?
2. If possible, could you please tell me the correct method to obtain pprof data for each component in k3s?
3. The above data was obtained on the server node; how can I obtain data on the agent nodes

Describe alternatives you've considered

Additional context

k3s version:
k3s version 1.28.2+k3s1
go version
go1.21.9

Node(s) CPU architecture, OS, and Version:
Linux 6.8.0-40-generic #40~22.04.3-Ubuntu SMP PREEMPT_DYNAMIC Tue Jul 30 17:30:19 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

[brandond]

You can't get individual info for separate components, because k3s runs all the Kubernetes components - including the kubelet - in a single process. This is the entire value proposition of k3s and is covered in the docs.

You can get it from agent in the same way you do on the server - by enabling pprof on the command line and hitting the pprof debug endpoint on the supervisor port.

[YYM0093]

You can't get individual info for separate components, because k3s runs all the Kubernetes components - including the kubelet - in a single process. This is the entire value proposition of k3s and is covered in the docs.

You can get it from agent in the same way you do on the server - by enabling pprof on the command line and hitting the pprof debug endpoint on the supervisor port.

Thank you very much for your reply. I understand that I cannot directly obtain the pprof data for each component, but theoretically, the pprof data I obtained in the manner described should reflect the overall memory usage of k3s. However, the actual values are significantly lower than those reported by the 'top' command. Could you please explain what the data I access at /debug/pprof represents?

[brandond]

The standard golang profiling data: https://pkg.go.dev/net/http/pprof

[YYM0093]

Thank you for your response. What I meant may not be the explanation for how pprof obtains the data, but rather the explanation of pprof's data in k3s. I have the following configuration file on an idle k3s-server node:

write-kubeconfig-mode: 644
enable-pprof: true
supervisor-metrics: true

I obtained memory usage through three different methods:

1. kubectl top nodes
   

2. top
   

3. go tool pprof http://127.0.0.1:8001/debug/pprof/heap
   

My questions are:
A. Why is there such a big difference in Method 3?

B. Based on the code related to /debug/pprof in “pkg/agent/run.go,” shouldn't the /debug/pprof method yield similar results to the "kubectl top nodes" method?

[brandond]

These are golang questions, not K3s questions. I don't know how the process memory shown by top, or the cadvisor metrics shown in kubectl top nodes, relate to the memory reported by golang's profiles. I suspect that pprof only shows the golang heap in use, while the process memory is going to include stuff that is mmaped, allocated but not in use heap memory that hasn't been freed back to the OS, and so on.

If you're trying to use pprof, you should probably understand enough about the golang memory model to figure this out - but I can't help you with that here.

[YYM0093]

Thanks for the reply, I will try you recommendations！