Network¶
Netgroups¶
For convenience configured on the head1. Following ones are available
- heads
- All heads
- computeheads
- Compute heads (head3-head6)
- storageheads
- Heads carrying storage, main or backup (head1, head2)
Networks¶
Interfaces configured to use 10.0.0.0/16. So separation is
- 10.0.0.0/24
- Compute nodes + backup node + main node
- 10.0.1.0/24
- Associated IPMI cards
- 10.0.2.0/24
- Misc. hardware to support. E.g. router has 10.0.2.1 (static),
- APC1 unit has 10.0.2.2 (via dhcp)
Bonding¶
All nodes have both interfaces bonded and powered by bonding module using balance-rr mode which theoretically allows to double throughput. Parameters for the module were provided by Microway and reside under /etc/modprobe.d/arch-aliases.
Link Aggregation¶
On the _router ports for head1-head3 were combined into LAG (link aggregation) ports. With TCP though no benefit was observed. Only UDP seems to get advantage but since order of packets is not guaranteed – there might be problems. Sample test outputs with LAG Up and “Flow control” enabled at the router:
root@head1:/var/yp# netperf -P 1 -c -C -n 8 -f M -l 10 -H head2 -t TCP_STREAM
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to head2.hydra.dartmouth.edu (10.0.0.2) port 0 AF_INET : demo
Recv Send Send Utilization Service Demand
Socket Socket Message Elapsed Send Recv Send Recv
Size Size Size Time Throughput local remote local remote
bytes bytes bytes secs. MBytes /s % S % S us/KB us/KB
87380 16384 16384 10.04 111.59 7.72 8.32 5.403 5.822
root@head1:/var/yp# netperf -P 1 -c -C -n 8 -f M -l 10 -H head2 -t UDP_STREAM
UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to head2.hydra.dartmouth.edu (10.0.0.2) port 0 AF_INET : demo
Socket Message Elapsed Messages CPU Service
Size Size Time Okay Errors Throughput Util Demand
bytes bytes secs # # MBytes/sec % SS us/KB
124928 65507 10.01 18363 0 114.6 2.37 1.618
124928 10.01 18362 114.6 4.39 2.992
and with LAG enabled with “Flow control” off:
root@head1:/var/yp# netperf -P 1 -c -C -n 8 -f M -l 10 -H head2 -t TCP_STREAM
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to head2.hydra.dartmouth.edu (10.0.0.2) port 0 AF_INET : demo
Recv Send Send Utilization Service Demand
Socket Socket Message Elapsed Send Recv Send Recv
Size Size Size Time Throughput local remote local remote
bytes bytes bytes secs. MBytes /s % S % S us/KB us/KB
87380 16384 16384 10.13 89.37 5.53 6.11 4.833 5.341
root@head1:/var/yp# netperf -P 1 -c -C -n 8 -f M -l 10 -H head2 -t UDP_STREAM
UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to head2.hydra.dartmouth.edu (10.0.0.2) port 0 AF_INET : demo
Socket Message Elapsed Messages CPU Service
Size Size Time Okay Errors Throughput Util Demand
bytes bytes secs # # MBytes/sec % SS us/KB
124928 65507 10.01 36700 0 229.1 2.87 8969.761
124928 10.01 4 0.0 2.93 9158.813
As we see, UDP manages to fill up both ports in such cases, but TCP is hurt. Without LAG on router, like head4,head5 there is actually less penalty for TCP:
head4:~# netperf -P 1 -c -C -n 8 -f M -l 10 -H head5 -t UDP_STREAM
UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to head5.hydra.dartmouth.edu (10.0.0.5) port 0 AF_INET : demo
Socket Message Elapsed Messages CPU Service
Size Size Time Okay Errors Throughput Util Demand
bytes bytes secs # # MBytes/sec % SS us/KB
124928 65507 10.01 36537 0 228.1 7.11 4943.666
124928 10.01 18 0.1 3.14 2184.322
head4:~# netperf -P 1 -c -C -n 8 -f M -l 10 -H head5 -t TCP_STREAM
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to head5.hydra.dartmouth.edu (10.0.0.5) port 0 AF_INET : demo
Recv Send Send Utilization Service Demand
Socket Socket Message Elapsed Send Recv Send Recv
Size Size Size Time Throughput local remote local remote
bytes bytes bytes secs. MBytes /s % S % S us/KB us/KB
87380 16384 16384 10.01 110.16 8.22 9.03 5.827 6.403
So LUGs were disabled. Advantage of bonding for TCP comes whenever multiple boxes access the same storage (and are lucky to be assigned to different ports based on MAC):
root@head1:/var/yp# netperf -P 1 -c -C -n 8 -f M -l 10 -H head2 -t TCP_STREAM & netperf -P 1 -c -C -n 8 -f M -l 10 -H head3 -t TCP_STREAM
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to head2.hydra.dartmouth.edu (10.0.0.2) port 0 AF_INET : demo
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to head3.hydra.dartmouth.edu (10.0.0.3) port 0 AF_INET : demo
Recv Send Send Utilization Service Demand
Socket Socket Message Elapsed Send Recv Send Recv
Size Size Size Time Throughput local remote local remote
bytes bytes bytes secs. MBytes /s % S % S us/KB us/KB
87380 16384 16384 10.01 98.72 12.63 6.89 9.995 5.449
Recv Send Send Utilization Service Demand
Socket Socket Message Elapsed Send Recv Send Recv
Size Size Size Time Throughput local remote local remote
bytes bytes bytes secs. MBytes /s % S % S us/KB us/KB
87380 16384 16384 10.01 107.86 12.58 7.70 9.113 5.576