Server state file reading does not respect server address #3265
Description
Detailed Description of the Problem
When show servers state is polled while set server / addr ... port ... is executed repeatedly (e.g., via the CLI socket), the output can show a server address from one update and a port from another. This produces lines where the IP address and the port do not match (e.g., 10.10.10.10 with port 500 even though 10.10.10.10 was set with port 100).
This suggests the dump path reads fields that can be updated independently without a consistent snapshot. The issue is reproducible with a tight loop that alternates set server ... addr ... port ... and another loop that runs show servers state (optionally with reloads). The result is inconsistent output in the state dump, which can mislead operators and tooling that consume show servers state.
Thanks @benedictweis for the debugging setup!
Expected Behavior
The address and the port should never diverge.
Steps to Reproduce the Behavior
- Create this script:
#!/usr/bin/env bash
set -o errexit
set -o nounset
set -o pipefail
if [[ "${TRACE-0}" == "1" ]]; then set -o xtrace; fi
HAPROXY_VERSION="3.0.14"
CONTAINER_NAME="haproxy-issue"
HAPROXY_ADMIN_SOCK="/tmp/haproxy-admin.sock"
HAPROXY_MASTER_SOCK="/tmp/haproxy-master.sock"
HAPROXY_SERVER_STATE_FILE="/tmp/server-state-file"
CHANGE_SERVER_CONFIG_PID=-1
RELOAD_HAPROXY_PID=-1
WATCH_STATE_PID=-1
function cleanup() {
kill $CHANGE_SERVER_CONFIG_PID || echo "Failed to kill change server config process with PID $CHANGE_SERVER_CONFIG_PID"
kill $RELOAD_HAPROXY_PID || echo "Failed to kill reload haproxy process with PID $RELOAD_HAPROXY_PID"
kill $WATCH_STATE_PID || echo "Failed to kill watch state process with PID $WATCH_STATE_PID"
docker rm --force ${CONTAINER_NAME}
exit 0
}
trap 'cleanup' EXIT SIGINT SIGTERM
docker run -d --name ${CONTAINER_NAME} --volume ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro haproxy:${HAPROXY_VERSION} haproxy -f /usr/local/etc/haproxy/haproxy.cfg -S ${HAPROXY_MASTER_SOCK} -W
docker exec ${CONTAINER_NAME} sh -c "
i=0
while true; do
i=\$((i + 1))
if [ \$((i % 2)) -eq 0 ]; then
echo \"set server backend-1/server-1 addr 10.10.10.10 port 100\" | socat ${HAPROXY_ADMIN_SOCK} -
else
echo \"set server backend-1/server-1 addr 50.50.50.50 port 500\" | socat ${HAPROXY_ADMIN_SOCK} -
fi
done
" > change_server_config.log 2>&1 &
CHANGE_SERVER_CONFIG_PID=$!
docker exec ${CONTAINER_NAME} sh -c "
while true; do
echo 'show servers state' | socat ${HAPROXY_ADMIN_SOCK} - > ${HAPROXY_SERVER_STATE_FILE}
echo 'reload' | socat ${HAPROXY_MASTER_SOCK} -
sleep 0.5
done" > reload_haproxy.log 2>&1 &
RELOAD_HAPROXY_PID=$!
docker exec ${CONTAINER_NAME} sh -c "while true; do echo 'show servers state' | socat ${HAPROXY_ADMIN_SOCK} - | grep 'backend-1 1 server-1'; done" > watch_state.log 2>&1 &
WATCH_STATE_PID=$!
wait
- Use config below as haproxy.cfg
- Execute the script and run
cat watch_state.log | grep "10.10.10.10" | grep "500"
Do you have any idea what may have caused this?
The show servers state dump reads multiple fields from struct server that can be updated independently (address vs. service port). With concurrent set server ... addr/port updates, the dump can pick up a new address and an old port (or vice versa), producing a mismatched pair. The dump path doesn’t take a consistent snapshot of these related fields, so race windows show up under rapid updates.
Do you have an idea how to solve the issue?
The safest fix is to read all related server fields under the per‑server lock and format from those captured values. Additionally, derive the output port from the server address (get_host_port(&srv->addr)) so IP/port are always consistent, and fall back to srv->svc_port only if no port is present in the address. A deeper fix is to ensure set server updates keep srv->svc_port synchronized with the address update in the same critical section, so any consumer that uses svc_port still gets a consistent value.
One working solution is here: sap-contributions#8
However, this was a bit rushed and needs more thought.
What is your configuration?
global
server-state-file /tmp/server-state-file
stats socket /tmp/haproxy-admin.sock mode 600 level admin
stats timeout 2m
defaults
load-server-state-from-file global
backend backend-1
mode tcp
server server-1 10.10.10.10:100Output of haproxy -vv
HAProxy version 3.4-dev4 2026/02/04 - https://haproxy.org/
Status: development branch - not safe for use in production.
Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open
Running on: Linux 5.14.21-150500.55.130-default #1 SMP PREEMPT_DYNAMIC Fri Nov 28 01:22:57 UTC 2025 (c32feab) x86_64
Build options :
TARGET = linux-glibc
CC = cc
CFLAGS = -O2 -g -fwrapv
OPTIONS = USE_OPENSSL=1 USE_PCRE2=1
DEBUG =
Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ECH -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE +KTLS -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY -LUA -MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 -PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL -PROMEX -PTHREAD_EMULATION -QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB +ACME +HAVE_TCP_MD5SIG
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_TGROUPS=32, MAX_THREADS=1024, default=40).
Built with SSL library version : OpenSSL 3.0.8 7 Feb 2023
Running on SSL library version : OpenSSL 3.0.8 7 Feb 2023
SSL library supports TLS extensions : yes
SSL library supports SNI : yes
SSL library default verify directory : /var/lib/ca-certificates/openssl
SSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.39 2021-10-29
PCRE2 library supports JIT : no (USE_PCRE2_JIT not set)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 7.5.0
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
<default> : mode=SPOP side=BE mux=SPOP flags=HOL_RISK|NO_UPG
spop : mode=SPOP side=BE mux=SPOP flags=HOL_RISK|NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
Available services : none
Available filters :
[BWLIM] bwlim-in
[BWLIM] bwlim-out
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
Last Outputs and Backtraces
`cat watch_state.log | grep "10.10.10.10" | grep "500"`
2 backend-1 1 server-1 10.10.10.10 2 0 1 1 1 1 0 4 0 0 0 0 - 500 - 0 0 - - 0
2 backend-1 1 server-1 10.10.10.10 2 0 1 1 3 1 0 4 0 0 0 0 - 500 - 0 0 - - 0
2 backend-1 1 server-1 10.10.10.10 2 0 1 1 4 1 0 4 0 0 0 0 - 500 - 0 0 - - 0
2 backend-1 1 server-1 10.10.10.10 2 0 1 1 5 1 0 4 0 0 0 0 - 500 - 0 0 - - 0
2 backend-1 1 server-1 10.10.10.10 2 0 1 1 6 1 0 4 0 0 0 0 - 500 - 0 0 - - 0
2 backend-1 1 server-1 10.10.10.10 2 0 1 1 6 1 0 4 0 0 0 0 - 500 - 0 0 - - 0
Additional Information
No response