LVS

-
Work on layer 4 

Cluster
Scale Up: Use better hardware resource to improve performance; --> the efficent is low. High cost but low improvement.
Scale Out: parallel extention

Linux Cluster Mode
1. Load Balance
2. High Available: 99.999% --> 315 seconds for one year; 99.99% --> 3150 seconds for one year; 
3. High Performance: Distributed storage

Website Distribution
A. Level: Application level / Web level / Database level
B. Separation: depending on business or file type
C. Distribution: application / static resource / data storage / calculation

Load balance Realiziation
Sofeware: LVS (Linux Virtual Server)  -> Transfer Level
Hardware: F5 BIG-IP / Citrix Netsacler / A10

LVS Knowledge
LVS logic: unpack the package from client and find out the IP and PORT, distribute requests to real servers;
ipvsadm: distribution policies
ipvs: program to apply policies

Director: lvs server
Real Server: real servers

Client IP: CIP
Direct Virtual IP: VIP (lvs server IP listen client requests)
Direct IP: DIP (lvs server IP send requests to real servers)
Real Server IP: RIP

LVS work mode
1. lvs-nat: tranfer request by changing the target IP address; real server feedback the response to lvs server; Lvs server feedback to client;
Request: change the target IP from VIP to RIP
Response: change the source IP from RIP to VIP

Character
A. The gateway setting of real server must be lvs server DIP
B. The request and response must go through lvs server; it may be the bottleneck
C. The real servers could be windows or linux
D. Support Port Map;
E. RIP and DIP must in same subnet;

2. lvs-dr: Change to mac address to distribut to real servers; All real server lo IP is same as VIP in order to process the request; Directly feedback to client
Request: change the traget mac address from VIP to that of RIP
Response: directly response from real server to client

Character
A. The lvs server only use mac address to distribute request to real servers
B. The lo IP setting of real server must be lv server VIP
C. real server must be in same physical network with lvs server
D. not support port map;
E. If real servers have private IP address, it need nat server to transfer response to client;
If backed servers have public IP address, it can directly transfer response to client
F. Request need lvs server; Response no need lvs server
G. real server only can support Linux server
H. real server gateway must not point to lvs server

3. lvs-tun: use public network to connect lvs server and real servres; Add RIP as target and DIP as source to the request package; Directly feedback to client

Character
A. RIP, DIP, VIP all public IP address
B. real server directly feedback to client
C. real server gateway must not point to lvs server
D. Not support port map
E. real server need support tunnel feature

4. lvs-fullnat: tranfer request by changing the target IP address; real server feedback the response to lvs server; Lvs server feedback to client;
Request: change source IP from CIP to DIP and target IP from VIP to RIP
Response: change source IP from RIP to VIP and target IP from DIP to CIP

LVS Scheduler
Check LVS configuration: grep -A 10 -i "ipvs"  /boot/config-2.6.32-504.el6.x86_64

Session binding
1. Session bind: source IP address
2. Session cluster: every real server save session information --> only suitable for small scale system
3. Session server: save session server directly --> need HA to secure


Distribution Methods
1. Static: not consider the current situaion; apply the real servers are different
A. RR: Round robin
B. WRR: Weighted RR overhead=current conn number / weight 
Example session list 1,2,....
weight=2 1 3 4   6 7   
weight=1 2      5      8 
C. SH: Source hashing: session binding
D. DH: Destination hashing: session binding

2. Dynamic: consider the current situation; apply the real servers are same
A. LC: Less Connection
overhead=active conn*256+inactive conn
B. WLC: Weighted LC
overhead=(active conn*256+inactive conn)/weight
C. SED: Shortest Expection Delay
overhead=(active conn+1)*256/weight

LVS client: ipvsadm
LVS kernel: ipvs (default in kernel)

LVS Setup
1. LVS Installation: yum install ipvsadm
2. LVS Config
A. Cluster Service Management
i. Format
ipvsadm -A|E -t|u|f service-address [-s scheduler] [-p [timeout]] [-M netmask]
ipvsadm -D -t|u|f service-address

ii. Description
ipvsadm -A: Create
ipvsadm -E: Modify
ipvsadm -D: Delete

-t: tcp protocol
-u: udp protocol
-f: firewall

service-address: VIP address
-t|u: IP:PORT
-f: #

-s: default WLC (Weighted Less Connection)

B. Add real servers
i. Format
ipvsadm -a|e -t|u|f service-address -r server-address [-g|i|m] [-w weight] 
ipvsadm -d -t|u|f service-address -r server-address

ii. Description
-a: Add 
-e: Modify
-d: delete

-t|u|f service-address： 引用此前定义过的集群服务 

-r server-address：RS的地址

IP[:PORT]: PORT省略时表示使用与对应的CS相同的端口；使用不同的端口时，需明确指定，且意味使用了端口映射功能，只有NAT类型可使用；

[-g|i|m]：LVS work mode
-g: GATEWAY(dr)
-i: IPIP(tun)
-m: MASQUERADE(nat)

[-w weight]: 权重，数字表示；

iii. Example
ipvmadm -a -t 172.16.100.6:80 -r 192.168.10.7 -m -w 2

C. Save the configuration to file
ipvsadm -S > <file name>

D. Read the config file
ipvsadm -R < <file name>

E. Clean policy
ipvsadm -C

F. Check policy
ipvsadm -c: list all the connection;
ipvsadm --stats: list CS & RS connection number;

Example1: Define web cluster lvs-nat work mode
ENV: (lvs server using host-only/bridge network; real servers using internal network)
LVS Server: 
A. VIP: 12.11.0.101 eth0
B. DIP: 172.16.1.1 eth1
Real Servers: 
Real Server1: RIP1: 172.16.1.2 GATEWAY: 172.16.1.1 
Real Server2: RIP2: 172.16.1.3 GATEWAY: 172.16.1.1

1. Sync all server time 

2. LVS Server
A. Network configuration
VIP: ifconfig eth0 12.11.0.101/24 up

DIP: ifconfig eth1 172.16.1.1/24 up

service network restart

B. Check status: cat /proc/sys/net/ipv4/ip_forward
0: disable
1: enable

C. Change config
Temporary Change: echo 1 > /proc/sys/net/ipv4/ip_forward
Permanent Change system config: vim /etc/sysctl.conf
change from
net.ipv4.ip_forward=0
to
net.ipv4.ip_forward=1

D. Reload config: sysctl -p

E. Stop firewall: service iptables stop

F. Install ipvsadm: yum install ipvsadm

3. Real Servers
A. Stop firewall
B. Network configuration
i. Real Server 1
ifconfig eth0 172.16.1.3/24 up
route add default gw 172.16.1.1

ii. Real Server 2
ifconfig eth0 172.16.1.4/24 up
route add default gw 172.16.1.1

C. Install & Config LAMP service
i. yum install httpd mysql-server mysql php php-mysql
ii. config index.php & conn.php

4. LVS Config (LVS Server)
i. Create cluster service: ipvsadm -A -t 12.11.0.101:80 -s rr
ii. Add Real server1: ipvsadm -a -t 12.11.0.101:80 -r 172.16.1.2 -m
iii. Add Real server2: ipvsadm -a -t 12.11.0.101:80 -r 172.16.1.3 -m
iv. Check the config: ipvsadm -L -n

5. LVS Check

6. Additional
A. Change from rr to weight rr
ipvsadm -E -t 12.11.0.101:80 -s wrr
ipvsadm -e -t 12.11.0.101:80 -r 172.16.1.2 -m -w 2

B. Shutdown one server to test 
need manually create schedule job to monitor the server status, if the server down(ping or curl), remove the configuration of the server

Example2: Define web cluster lvs-dr work mode
ENV: (all three server using host-only/bridge network)
LVS Server: 
A. VIP: 12.11.0.101 eth1
B. DIP: 12.11.0.102 eth0
Real Servers: 
Real Server1: RIP1:12.11.0.103
Real Server2: RIP2:12.11.0.104

1. Sync all server time 
2. Network configuration
ifconfig eth0 12.11.0.102/24 up
ifconfig  eth1 12.11.0.101 broadcast 12.11.0.101 netmask 255.255.255.255  up
route add -host 12.11.0.101 dev eth1

3. LVS Server
A. Check status: cat /proc/sys/net/ipv4/ip_forward
0: disable
1: enable

B. Change config
Temporary Change: echo 1 > /proc/sys/net/ipv4/ip_forward
Permanent Change system config: vim /etc/sysctl.conf
change from
net.ipv4.ip_forward=0
to
net.ipv4.ip_forward=1

C. Reload config: sysctl -p

D. Stop firewall

4. Real Servers
A. Stop firewall
B. Network configuration
i. Real Server 1
ifconfig eth0 12.11.0.103 up
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
ifconfig  lo:0 12.11.0.101 broadcast 12.11.0.101 netmask 255.255.255.255  up
route add -host 12.11.0.101 dev lo:0

ii. Real Server 2
ifconfig eth0 12.11.0.104 up
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
ifconfig  lo:0  12.11.0.101 broadcast 12.11.0.101 netmask 255.255.255.255  up
route add -host 12.11.0.101 dev lo:0

C. Config httpd service

5. LVS Config (LVS Server)
i. Create cluster service: ipvsadm -A -t 12.11.0.101:80 -s rr
ii. Add Real server1: ipvsadm -a -t 12.11.0.101:80 -r 12.11.0.103 -g
iii. Add Real server2: ipvsadm -a -t 12.11.0.101:80 -r 12.11.0.104 -g
iv. Check the config: ipvadm -L -n

6. LVS Check

Comments

Post a Comment

Popular posts from this blog

Nginx Proxy & Load Balance & LNMP

-
Nginx Proxy Service E-commerce website Server 1: provide image request response (  apache / nginx ) Server 2: provide dynamic request response ( php-fpm / apache-php ) Server 3: provide static request response ( apache / nginx ) Server 4: provide data request response ( mysql ) It need one additional nginx server 1. Install nginx 2. proxy to specific server Example location / {      proxy_pass http://192.168.199.187:80/; } location /bbs/ {      proxy_pass http://192.168.199.187:80/bbs/; } location ~* \.(jpg|png|gif)$  {      proxy_pass http://172.16.100.7; } Enhance the log information and specific the hostname location / {      proxy_pass  http://192.168.199.187:80/;      proxy_set_header <parameter name>  $host;  (client requst the hostname )      proxy_set_header <parameter name> $remote_addr; ...
Read more

Snort+barnyard2+Snorby CentOS 6.5_64 Installation

-
#download packages: daq-2.0.2.tar.gz ImageMagicK-6.8.9-8.tar.gz oinkmaster-2.0.tar.gz ruby-1.9.3-p547.tar.gz snort-2.9.6.2.tar.gz snortrules-snapshot-2962.tar.gz MySQL-server-5.5.40-1.rhel5.x86_64.rpm MySQL-client-5.6.21-1.rhel5.x86_64.rpm MySQL-devel-5.6.21-1.rhel5.x86_64.rpm MySQL-shared-5.6.21-1.rhel5.x86_64.rpm epel-release-6-8.noarch.rpm barnyard2-2.1.13.tar.gz #install yum package rpm -ivh epel-release-6-8.noarch.rpm yum install gcc-c++ patch readline readline-devel zlib zlib-devel flex git yum install libyaml-devel libffi-devel openssl-devel make  yum install bzip2 autoconf automake libtool bison iconv-devel libxml2-devel libxslt  yum install gcc g++ build-essential libssl-devel libreadline5-devel zlib1g-devel linux-headers-generic libsqlite3-devel libxslt-devel curl-devel sqlite-devel yum --enablerepo=epel install jasper jasper-libs jasper-devel #Mysql install yum remove mysql mysql-libs rpm -ivh MySQL-server-5.5.40-1.rh...
Read more

ORACLE Error

-
Image
oracle official error explanation  http://docs.oracle.com/cd/E11882_01/server.112/e17766/toc.htm 从服务器下载ALERT日志文件，传输方式需要选择"ASCII" ORA-01078: failure in processing system parameters 发生场景：新建数据库第一次启动时 SQL> startup ORA-01078: failure in processing system parameters LRM-00109: could not open parameter file ‘/home/oracle/oracle/product/10.2.0/db_1/dbs/initorcl.ora’ 原因：oracle_sid没有配置正确 解决方案： 1. 修改oracle用户.bash_profile文件中ORACLE_SID后的内容； 2. 重新已oracle用户登录； ORA-01157: cannot identify/lock data file string - see DBWR trace file  Cause: The background process was either unable to find one of the data files or failed to lock it because the file was already in use. The database will prohibit access to this file but other files will be unaffected. However the first instance to open the database will need to access all online data files. Accompanying error from the operating system describes why the file could not be identified. Action: Have operating system make file ...
Read more