====== [HOWTO] Balancing Gateways (keepalived+haproxy) ======
^ Documentation ^|
^Name:| [HOWTO] Balancing Gateways (keepalived+haproxy) |
^Description:| A production-ready way to balance gateways |
^Modification date :|28/07/2019|
^Owner:|dodger|
^Notify changes to:|Owner |
^Tags:|ceph, object storage |
^Scalate to:|The_fucking_bofh|
====== Pre-Requirements ======
* [[linux:ceph:howtos:using_amazon_dns_bucket_naming|Setup S3 naming method]]
====== Instructions ======
===== Deploy lb nodes =====
For example:
bash CloneWars.sh -c nuciberterminal -h AVMLP-OSLB-001 -i 10.20.54.1 -m 20 -O -r 8192 -v 2 -o 4 -F
bash CloneWars.sh -c nuciberterminal2 -h AVMLP-OSLB-002 -i 10.20.54.2 -m 20 -O -r 8192 -v 2 -o 4 -F
===== Apply basic states =====
export THESERVER="avmlp-oslb-0*"
salt "${THESERVER}" state.apply
salt "${THESERVER}" state.apply nsupdate
===== Install required packages =====
salt "${THESERVER}" pkg.install haproxy
salt "${THESERVER}" pkg.install keepalived
===== Setup HAproxy =====
Sample from clover:
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2
#log /var/log/haproxy.log
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 2048
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 2048
frontend http_web *:80
mode http
default_backend rgw
#frontend rgw-https
# bind :443 ssl crt /etc/ssl/private/example.com.pem
# default_backend rgw
backend rgw
balance roundrobin
mode http
server rgw1 avmlp-osgw-001.ciberterminal.net:80 check maxconn 512
server rgw2 avmlp-osgw-002.ciberterminal.net:80 check maxconn 512
server rgw3 avmlp-osgw-003.ciberterminal.net:80 check maxconn 512
server rgw4 avmlp-osgw-004.ciberterminal.net:80 check maxconn 512
===== Setup rsyslog =====
Necessary for haproxy logging:
--- rsyslog.conf 2018-10-30 15:49:15.000000000 +0100
+++ rsyslog.conf.new 2019-08-13 17:43:26.004833747 +0200
@@ -12,12 +12,12 @@
#$ModLoad immark # provides --MARK-- message capability
# Provides UDP syslog reception
-#$ModLoad imudp
-#$UDPServerRun 514
+$ModLoad imudp
+$UDPServerRun 514
# Provides TCP syslog reception
-#$ModLoad imtcp
-#$InputTCPServerRun 514
+$ModLoad imtcp
+$InputTCPServerRun 514
#### GLOBAL DIRECTIVES ####
@@ -72,6 +72,8 @@
# Save boot messages also to boot.log
local7.* /var/log/boot.log
+# haproxy logging
+local2.* /var/log/haproxy.log
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
===== Setup keepalived =====
global_defs {
notification_email {
dodger@ciberterminal.net
}
notification_email_from clover@ciberterminal.net
smtp_server mta4.bavel.biz
smtp_connect_timeout 30
! router_id LVS_DEVEL
! vrrp_skip_check_adv_addr
! vrrp_strict
! vrrp_garp_interval 0
! vrrp_gna_interval 0
}
vrrp_script chk_haproxy {
script "killall -0 haproxy" # check the haproxy process
interval 2 # every 2 seconds
weight 2 # add 2 points if OK
}
vrrp_instance VI_1 {
interface eth0 # interface to monitor
state MASTER # MASTER on haproxy, BACKUP on haproxy2
virtual_router_id 51
priority 101 # 101 on haproxy, 100 on haproxy2
virtual_ipaddress {
10.20.54.0 # virtual ip address
}
track_script {
chk_haproxy
}
smtp_alert
}
On the secondary node, you'll have to chante the line:
state MASTER # MASTER on haproxy, BACKUP on haproxy2
===== setup pmta to allow sending un-authenticated emails =====
# avmlp-oslb-001
# avmlp-oslb-002
# clover.ciberterminal.net
===== Restart & enable all =====
systemctl restart rsyslog
systemctl restart haproxy
systemctl restart keepalived.service
systemctl enable rsyslog
systemctl enable haproxy
systemctl enable keepalived.service
====== Official documentation ======
* [[https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/load_balancer_administration/ceph_example|Load Balancing Ceph Object Gateway Servers with HAProxy and Keepalived]]