HiveBrain v1.2.0
Get Started
← Back to all entries
debugkubernetesMinor

HAProxy Not Working with Kubernetes NodePort for Backend (Bare Metal)

Submitted by: @import:stackexchange-devops··
0
Viewed 0 times
backendbarewithkubernetesworkinghaproxynodeportmetalfornot

Problem

I have a host running HAProxy already. It's been up and running since before I did anything with Kubernetes. It works flawlessly as a reverse proxy and SSL terminator for various web things in various Docker containers on various other host machines.

Now I have a Kubernetes cluster up and running across some of those other machines. I've created the NodePort Service that exposes port 30080 on each worker node, as follows:

apiVersion: v1
kind: Service
metadata:
  name: snginx
  labels:
    app: nginx
spec:
  type: NodePort
  externalTrafficPolicy: Local    #Cluster or Local
  selector:
    app: nginx
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 80
    nodePort: 30080


From the machine running HAProxy (which is not part of the cluster), I can curl the NodePort successfully ( curl 10.0.0.20:30080 ), and I get "Welcome to nginx!..." However, if I set that NodePort as a backend in HAProxy, I get a 503 "No server is available", and HAProxy traffic log says:

localhost haproxy[18489]: [redactedIP]:49399 [30/Aug/2021:19:24:00.764] http-in~ load/load 66/0/-1/-1/67 503 212 - - SC-- 1/1/0/0/3 0/0 "GET / HTTP/1.1"


The haproxy admin log says:

Aug 30 20:07:13 localhost haproxy[18839]: Server load/load is DOWN, reason: Layer4 connection problem, info: "General socket error (Permission denied)"


However, I've disabled the firewall with

sudo systemctl disable --now firewalld


and verified the status is not running. Also, SELinux was disabled when I installed the cluster. Also, I can ping 10.0.0.20 just fine.

"load" is the hostname I'm using for testing load balancing (i.e. load.mydomain.com).

Also, if I use static NAT on my physical router to route directly to that NodePort, from outside the building, it works as expected.

What gives? What's the difference between the proxied request and curl?

Thank you.

Solution

I figured it out, and SELinux is the difference. That is, SELinux on the HAProxy host (not a cluster node):

"SELinux only allows the web server to make outbound connections to a limited set of ports"

That is, you can't make an outbound http request to any port in the NodePort range (30000-32768) without opening that port on the "client", which is the HAProxy server in this case.

sudo semanage port --add --type http_port_t --proto tcp 30080


More specifically, I guess, the security context that HAProxy runs under is different than curl.

Code Snippets

sudo semanage port --add --type http_port_t --proto tcp 30080

Context

StackExchange DevOps Q#14597, answer score: 1

Revisions (0)

No revisions yet.