Linux Process Debugging with strace and lsof

A process is hanging, consuming too much CPU/memory, or behaving unexpectedly. Application logs don't provide enough information.

System-level process debugging tools:

# What files does a process have open?
lsof -p <PID>
lsof -p <PID> | grep -i socket  # Just network connections
lsof -i :8080  # What process is using port 8080?

# Trace system calls (what is the process DOING?)
strace -p <PID> -f -e trace=network  # Network calls only
strace -p <PID> -e trace=file        # File operations
strace -p <PID> -e trace=write       # What it's writing
strace -c -p <PID>                    # Summary of call counts

# Process tree
ps auxf  # Full tree view
pstree -p <PID>  # Tree from specific process

# What's using memory?
pmap -x <PID>  # Memory map
cat /proc/<PID>/status | grep -i vm  # Memory stats

# What's using CPU?
top -p <PID> -H  # Per-thread CPU usage
perf top -p <PID>  # CPU profiling (if available)

# Check file descriptors
ls -la /proc/<PID>/fd | wc -l  # Count open FDs
cat /proc/<PID>/limits | grep 'open files'  # FD limit

# Signals
kill -0 <PID>  # Check if process exists (no signal sent)
kill -USR1 <PID>  # Send custom signal (app-specific)

Common findings:

• Process stuck in futex() = deadlock
• Process stuck in read()/poll() = waiting for I/O
• Too many open files = FD leak

When application-level debugging isn't enough, system calls reveal exactly what the process is doing at the OS level. This is especially useful for hung processes or resource leaks.

Debugging misbehaving processes on Linux