CVE-2023-25725: The Reliable, High Performance TCP/HTTP Load Balancer

Quick links****Quick News
Recent News
Description
Performance
Reliability
Security
Documentation
Project on GitHub
Download
Live demo
They use it!
Enterprise Features
Third party extensions
Commercial Support
Contacts
External links
Discussions
Slack channel
Mailing list
Coding style
Open Issues
Known bugs

HATop: Ncurses Interface
Herald: load feedback agent
haproxystats: stats collection
Alpine-based Docker images
Debian-based Docker images
RHEL-based Docker images
Debian/Ubuntu packages

visitors online

Thanks for your support !

Latest versions

Branch

Release date

End of life

Latest version

Changelog

Links

2.8-dev

~2023-Q2

2028-Q2 (dev » LTS)

2.8-dev4

2023/02/14

git / web / dir / announce

2.7

2022-12-01

2024-Q1 (stable)

2.7.3

2023/02/14

git / web / dir / announce / bugs

2.6

2022-05-31

2027-Q2 (LTS)

2.6.9

2023/02/14

git / web / dir / announce / bugs

2.5

2021-11-23

2023-Q1 (stable)

2.5.12

2023/02/14

git / web / dir / announce / bugs

2.4

2021-05-14

2026-Q2 (LTS)

2.4.22

2023/02/14

git / web / dir / announce / bugs

2.2

2020-07-07

2025-Q2 (LTS)

2.2.29

2023/02/14

git / web / dir / announce / bugs

2.0

2019-06-16

2024-Q2 (critical fixes only)

2.0.31

2023/02/14

git / web / dir / announce / bugs

Hide/Show unmaintained

Quick News

February, 14th, 2023 : CVE-2023-25725 fixed!

We’ve been notified of a vulnerability in HAProxy that can be exploited to build some request smuggling attacks. It affects all currently supported branches, all the details are here on the mailing list announce. Please make sure to update either to your latest distro package or to latest version if you build from the sources (2.0.31, 2.2.29, 2.4.22, 2.5.12, 2.6.9, 2.7.3 or 2.8-dev4).

December, 1st, 2022 : HAProxy 2.7.0 release

HAProxy 2.7.0 is now released and available for download, opening the way to 2.8-dev. 2.7 provides traffic shaping, many QUIC improvements, eases the switch to alternate SSL libraries, and improves user experience with everything related to troubleshooting and issue reporting. Please see the announnce for more details and/or consult the HAProxyTech blog article for more details.

June, 16th, 2022 : HAProxyConf: Call for Papers

This year, HAProxyConf 2022 will be held physically so that we can meet in person, just as we did in 2019! It will be held in Paris from November 8th to 9th. The Call for Papers is now open and ends on September 5th, just after vacation. Better not wait too much if you already have an idea in mind. There’s a simplified form to fill to propose a talk. It doesn’t ask much, just your contact and a quick abstract. If you have no idea yet, think about some great things or tricks you’ve achieved using HAProxy, or all the stuff tha made your friends tell you "you should really blog about it". Remember, you have one month and it’s counting…

May, 31st, 2022 : HAProxy 2.6.0 release

HAProxy 2.6 is now the latest long-term supported release. It further improves reliability, and focused on making future contributions easier by simplifying some of the internals, and completing the native HTTP client, allowing easier interactions with external services. And the star of this release is undoubtly the long-awaited support for the QUIC protocol! The full details are detailed here in the announce.

Mar, 26th, 2022 : QUIC experimentation

One front that made impressive progress over the last few months is QUIC. While a few months ago we were counting the number of red boxes on the interop tests at https://interop.seemann.io/ to figure what to work on as a top priority, now we’re rather counting the number of tests that report a full-green state, and haproxy is now on par with other servers in these tests. Thus the idea emerged, in order to continue to make progress on this front, to start to deploy QUIC on haproxy.org so that interoperability issues with browsers and real-world traffic can be spotted. A few attempts were made and already revealed issues so for now it’s disabled again. Be prepared to possibly observe a few occasional hiccups when visiting the site (and if so, please do complain to us). The range of possible issues would likely be frozen transfers and truncated responses, but these should not happen.

From a technical point, the way it’s done is by having a separate haproxy process listening to QUIC on UDP port 1443, and forwarding HTTP requests to the existing process. The main process constantly checks the QUIC one, and when it’s seen as operational, it appends an Alt-Svc header that indicates the client that an HTTP/3 implementation is available on port 1443, and that this announce is valid for a short time (we’ll leave it to one minute only so that issues can resolve quickly, but for now it’s only 10s so that quick tests cause no harm):

http-response add-header alt-svc 'h3=":1443"; ma=60' if { var(txn.host) -m end haproxy.org } { nbsrv(quic) gt 0 }

As such, compatible browsers are free to try to connect there or not. Other tools (such as git clone) will not use it. For those impatient to test it, the QUIC process’ status is reported at the bottom of the stats page here: http://stats.haproxy.org/. The “quic” socket in the frontend at the top reports the total traffic received from the QUIC process, so if you’re seeing it increase while you reload the page it’s likely that you’re using QUIC to read it. In Firefox I’m having this little plugin loaded: https://addons.mozilla.org/en-US/firefox/addon/http2-indicator/. It displays a small flash on the URL bar with different colors depending on the protocol used to load the page (H1/SPDY/H2/H3). When that works it’s green (H3), otherwise it’s blue (H2). For Chrome there is HTTP Indicator which does the same but displays an orange symbol when using H3. Chrome only accepts H3 on port 443 (which we enabled as well for it). Note that H2 and H3 are only served when the site is browsed in HTTPS at https://haproxy.org/.

At this point I’d still say "do not reproduce these experiments at home". Amaury and Fred are still watching the process’ traces very closely to spot bugs and stop it as soon as a problem is detected. But it’s still too early for being operated by non-developers. The hope is that by 2.6 we’ll reach the point where enthousiasts can deploy a few instances on not-too-sensitive sites with sufficient confidence and a little dose of monitoring.

Older news…

Description

HAProxy is a free, very fast and reliable reverse-proxy offering high availability, load balancing, and proxying for TCP and HTTP-based applications. It is particularly suited for very high traffic web sites and powers a significant portion of the world’s most visited ones. Over the years it has become the de-facto standard opensource load balancer, is now shipped with most mainstream Linux distributions, and is often deployed by default in cloud platforms. Since it does not advertise itself, we only know it’s used when the admins report it :-)

The HAProxy core team maintains multiple versions in parallel. Since version 1.8, two major version are emitted every year. The first digit usually indicates a breaking change (config format etc) but in practice rarely changes. The second digit indicates new features. Both constitute a branch. One extra number appears after these digits to indicate the bug fix release.

The core team deploys a lot of efforts backporting fixes to older releases while being extremely careful not to break anything. For this reason, it is really important to stay up to date within one branch, i.e. having the highest possible number on the last digits.

Branches with an even number are called “LTS” (for “long term support”) and area maintained for 5 years after their release. During this time they will receive fixes for bugs that are discovered after the release. These branches are aimed at general users who seek extreme stability and do not want to qualify a new version too often but still want to receive fixes.

Branches with an odd number are only called "stable", they’re aimed at highly skilled users who prefer to upgrade often to benefit from modern features, and who are also able to roll back in case of problem. These versions are maintained between 12 and 18 months. The duration is short and purposely not strict so that the maintenance cycle is decided with users based on feedback, and so that these versions do not end up in embedded products. It may happen that a few features are backported to these version if there is some reasonable demand and the operation is considered riskless enough.

Everyone used to dealing with production knows that it’s difficult to upgrade components in field when one has to plan and advertise upwards of any operation. For this reason, the HAProxy core team doesn’t insist on users to upgrade, will not ask someone to switch to a new branch (unless they ask for a feature that is part of that other branch), but will often ask the user to re-check with the latest version of their branch before reporting a problem, because nobody likes to troubleshoot a problem a second time. It’s often suggested to use the versions that come with the operating system when it follows the official maintenance cycle, and depending on the expected level of stability or exposure, some users may want to update as soon as an update is available while others may prefer to wait a few weeks to a month to be sure the update is reliable enough for them.

The currently supported versions are :

• version 2.7 : Traffic shaping, QUIC improvements, thread groups, easier switch to alternate SSL libraries, improved debugging
• version 2.6 : QUIC/HTTP3, OpenSSL 3.0, better usability, improved code accessibility and maintenance
• version 2.5 : runtime server addition/removal, runtime CA/CRL updates, native HTTP client, simplified HTTPS logging, default TCP/HTTP rulesets, JWT validation, and more
• version 2.4 : syslog and DNS over TCP, multi-threaded Lua, full sharing of idle conns, lower latency, server-side dynamic SSL update, Opentracing, WebSocket over H2, atomic maps, Vary support, new debugging tools, even more user-friendly CLI and configuration, lots of cleanups
• version 2.3 : syslog forwarding, better idle conn management, improved balancing with large queues, simplified SSL managment, more stats metrics, stricter config checking by default, general performance improvements
• version 2.2 : runtime certificate additions, improved idle connection management, logging over TCP, HTTP “return” directive, errorfile templates, TLSv1.2 by default, extensible health-checks
• version 2.1 : improved I/Os and multi-threading, FastCGI, runtime certificate updates, HTX-only, improved debugging, removal of obsolete keywords
• version 2.0 : gRPC, layer 7 retries, process manager, SSL peers, log load balancing/sampling, end-to-end TCP fast-open, automatic settings (maxconn, threads, HTTP reuse, pools), …
• version 1.9 : improved multi-threading, end-to-end HTTP/2, connection pools, queue priority control, stdout logging, …
• version 1.8 : multi-threading, HTTP/2, cache, on-the fly server addition/removal, seamless reloads, DNS SRV, hardware SSL engines, …
• version 1.7 : added server hot reconfiguration, content processing agents, multi-type certs, …
• version 1.6 : added DNS resolution support, HTTP connection multiplexing, full stick-table replication, stateless compression, …
• version 1.5 : added SSL, IPv6, keep-alive, DDoS protection, …

Performance

As shown in this test run on AWS ARM-based Graviton2, HAProxy scales very well with threads and was shown to be able to reach 2 million requests/s over SSL and 100 Gbps for forwarded traffic.

This is made possible thanks to its event-driven architecture that allows to react extremely quickly to I/O events, its parallelism on SMP machines provided by light multi-threading, a task scheduler that permanently composes between low-latency and high throughput, and generally speaking a permanent quest of resource savings at every single architecture layer. These efforts tend to cost a bit in development time but are immediately valued by users who are able to reduce their number of machines upgrade after upgrade. For the vast majority of common loads, the HAProxy process is simply not noticed, which tends to make its users forget it, sometimes resulting in questions regarding extremely old versions.

Please consult this section for more information on the architecture details and some performance test results.

Reliability - keeping high-traffic sites online since 2002

HAProxy is first known for being extremely robust. The core team developers tend to be irritated by certain bugs they fix, but this is because their job is to see them all. Most users report having never ever faced any single crash and claim that HAProxy is the most solid part of their infrastructure. Finding machines with HAProxy processes being up for more than 3 years is not exceptional at all!

All this is not an accident, though. A lot of efforts are made in that direction, to provide excellent observability on what is happening, and an amazing number of protections against bad behaviors. HAProxy is built with many checks for unacceptable situations (impossible conditions, endless loops, etc) that in other products might result in service outages or data corruption, but in HAProxy will immediately result in a crash with a dump of the problem. This rigor pays off since most users have never faced such an issue, thanks to the few who faced them and provided useful reports allowing to fix the problem early.

The development process also encourages quality, with a long term maintenance cycle: versions are maintained for 5 years by the same developers who code the new features. This encourages them to write high quality code and commit messages that correspond to the highest standards. A regression testing suite is used and run along development by all developers and before merging code, as well as after on a wide variety of platforms thanks to the continuous integration (CI) system.

The principle of “eating one’s dog’s food” applies here as well: haproxy.org runs on the latest development release. This usually helps spot a bug or two per major version before it hits a release. But in addition it maintains a permanent pressure on the development team to release something they’re confident in.

The program having been designed from its early age to be extremely conservative on resource usage, a significant number of settings are calculated at startup time and enforce many limits on number of sockets, connections, streams etc, guaranteeing that any processing that was started will complete.

Security - Hardened by default

Security is a very important concern when deploying a software load balancer, because it runs at the edge and takes all the dirty traffic. It is possible to harden the OS, to limit the number of open ports and accessible services, but the load balancer itself stays exposed. The unified and non-fantasist coding style aims at avoiding common traps when writing or reviewing code. Some high standards are sought when it comes to dealing with unvalidated data. Non-portable functions and those having unreliable behaviors are avoided or replaced. Input data gets sanitized very early in the lower layers. Resource usage is carefully controlled. Dangling pointers are forbidden in the code via careful release functions. These standards already help eliminate a great deal of uncertainty in the code itself.

Since zero-bug is not reasonable, the product embarks a number of defensive measures, such as chroot, privilege drops, fork prevention, strict protocol validation, checks for impossible states and detailed traces in case of violation detection, etc. All these usually result in an attempt to exploit a real bug in a failure or possibly a crash. These measures have to be purposely disabled by the user using sufficiently evocative commands so that the reason for doing so has to be regularly questioned.

Complete Download Matrix

Here you will find a quick access to downloadable contents by type and version. Just click on the desired format to access the content in that format.

dev

2.7

2.6

2.5

2.4

2.3

2.2

2.1

2.0

1.9

1.8

1.7

1.6

1.5

1.4

1.3

1.2

1.1

1.0

Git repository

Git / Web

Git / Web

Git / Web

Git / Web

Git / Web

Git / Web

Git / Web

Git / Web

Git / Web

Git / Web

Git / Web

Git / Web

Git / Web

Git / Web

Git / Web

Git / Web

Git / Web

Git / Web

Git / Web

Latest snapshot

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz

tar.gz

-

Latest release

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz / Log

tar.gz / Log

-

Browsable dir

Dir

Dir

Dir

Dir

Dir

Dir

Dir

Dir

Dir

Dir

Dir

Dir

Dir

Dir

Dir

Dir

Dir

Dir

Dir

Known bugs

Web

Web

Web

Web

Web

Web

Web

Web

Web

Web

Web

Web

Web

Web

Web

Web

Web

Web

Web

Starter guide

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

-

-

-

-

-

-

Configuration manual

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

txt

txt

txt

txt

Management guide

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

-

-

-

-

-

-

Lua ref. manual

html

html

html

html

html

html

html

html

html

html

html

html

html

-

-

-

-

-

-

Lua arch. guide

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

html / txt

Browsable doc

Dir

Dir

Dir

Dir

Dir

Dir

Dir

Dir

Dir

Dir

Dir

Dir

Dir

Dir

Dir

Dir

Dir

Dir

Dir

Please note that official docs are the pure-text ones and directly come from the project, except for the Lua reference manual that is maintained by Thierry Fournier. The HTML versions are direct translations from the text version automatically performed by Cyril Bont�’s excellent documentation converter, dconv. A TeX-oriented variant able to produce PDFs was also created by Pavel Lang for versions 1.4 and 1.5 but it is not maintained anymore.

Commercial Support and availability

If you think you don’t have the time and skills to setup and maintain a free load balancer, or if you’re seeking for commercial support to satisfy your customers or your boss, you have the following options :

2. contact HAProxy Technologies to hire some professional services or subscribe a support contract ;
3. install HAProxy Enterprise Edition (HAPEE), which is a long-term maintained HAProxy package accompanied by a well-polished collection of software, scripts, configuration files and documentation which significantly simplifies the setup and maintenance of a completely operational solution ; it is particularly suited to Cloud environments where deployments must be fast.
4. try an ALOHA appliance (hardware or virtual), which will even save you from having to worry about the system, hardware and from managing a Unix-like system.

I also find it important to credit Loadbalancer.org. I am not affiliated with them at all but like us, they have contributed a fair amount of time and money to the project to add new features and they help users on the mailing list, so I have some respect for what they do. They’re a UK-based company and their load balancer also employs HAProxy, though it is somewhat different from the ALOHA.Contacts

Feel free to contact us for any questions or comments :

• mailing-list :
  Read the list archives on mail-archive
  Read the list pre-2009 archives on Marc.info

  Subscribe to the list :
  Unsubscribe from the list :

• HAProxy site in HTTPS (needed for HTTP/3 and HTTP/2) : https://www.haproxy.org/

• Willy’ main site : http://1wt.eu/

• e-mail :

Some people regularly ask if it is possible to send donations, so I have set up a Paypal account for this. Click here if you want to donate.

An IRC channel for HAProxy has been opened on Libera.Chat:

irc://irc.libera.chat/%23haproxy

A Slack Workspace for HAProxy exists as well:

https://slack.haproxy.org/

External links

Here are some links to possibly useful external contents I gathered on the net. I have found most of them due to their link to haproxy’s site ;-)

• Health Checks and Graceful Degradation in Distributed Systems
• haproxy-auth-request: HTTP access control using subrequests
• Utiliser HAProxy pour profiter d’HTTP/2 (FR)
• HAProxy in 2018 - A load balancer with HTTP/2 and dynamic reconfig
• GlusterFS: Configuration and Setup w/ NFS-Ganesha for an HA NFS Cluster
• Service discovery at Stripe
• High Availability with HAProxy and Keepalived in AWS
• How we fine-tuned HAProxy to achieve 2,000,000 concurrent SSL connections
• Speeding up SSL - All you need to know about haproxy
• Improving load balancing with a new consistent-hashing algorithm
• ACME validation plugin for HAProxy / Let’s Encrypt
• Use HAProxy to load balance 300k concurrent tcp socket connections
• Accelerating SSL Load Balancers with Intel� Xeon� v3 Processors
• Linux networking stack from the ground up (parts 1-5) (useful to whoever wants to know what happens below haproxy)
• Loadbalance your website with haproxy and varnish
• How TubeMogul Handles over One Trillon HTTP Requests a Month
• HAProxy in the era of Microservices
• Making HAProxy 1.5 replication lag aware in MySQL
• MaxCDN: How to Use HAProxy to Handle Traffic Spikes
• Varnish, SSL and HAProxy
• True Zero Downtime HAProxy Reloads
• HAProxy Is Still An Arrow in the Quiver for Those Scaling Apps
• How To Set Up SQL Load Balancing with HAProxy (Webinar)
• HAProxy running on Ubuntu Cloud on Power8, featured by Mark Shuttleworth at IBM Impact 2014 Keynote
• Guidelines for HAProxy termination in AWS
• Marcus Rueckert’s talk at osc14
• How Stack Exchange gets the most out of HAProxy
• Open Source Windows service for reporting server load back to HAProxy (load balancer feedback agent).
• Load Balancing Amazon RDS Read Replica’s using HAProxy
• Installing HAProxy on pfSense
• MySQL Load Balancing with HAProxy - Tutorial
• HAProxy vs Nginx benchmark for the Eucalyptus Cloud computing Platform
• WebSocket Over SSL: HAProxy, Node.js, Nginx
• Comparison Analysis:Amazon ELB vs HAProxy EC2
• Simple SPDY and NPN Negotiation with HAProxy
• Using HAProxy to Build a More Featureful Elastic Load Balancer
• 3 ways to configure haproxy for websockets
• Segregating services at bitbucket
• Load balancing FTP, by Ben Timby
• Howto setup a haproxy as fault tolerant / high available load balancer for multiple caching web proxies on RHEL/Centos/SL
• Load balancing @Tuenti, by Ricardo Bartolom�
• Benchmarking SSL performance
• Smart Content Switching for News Website
• HA Proxy for Exchange 2010 Deployment & SMTP Restriction
• A more stable MySQL with HAProxy
• Benchmarking HAProxy under VMware : Ubuntu vs FreeBSD
• Stack Overflow: Better rate limiting for all with HAProxy
• Benchmarking Load Balancers in the Cloud
• Using HAProxy for MySQL failover and redundancy
• Setting up a high availability load blancer with haproxy and keepalived on debian lenny
• Configure HAProxy with TPROXY kernel for full transparent proxy
• HAProxy, X-Forwarded-For, GeoIP, KeepAlive
• Load Balancing in Amazon EC2 with HAProxy
• CouchDB Load Balancing and Replication using HAProxy
• Zero-Downtime restarts with HAProxy
• Free your port 80 with HAProxy
• Another comparison of HAProxy and Nginx
• Scaling on EC2
• HAProxy on Opensolaris 2008.05
• Load-Balancing and QoS with HAProxy
• Reviewing Application Health with HAProxy Stats