週四 17 11月 2011

週四 17 11月 2011

10G test

Lenovo M80 7521 CPU i7 Memory 4G PCI Express 16X with Intel 10G Gigabits dual port

Operation System: Linux Ubuntu 11.10 ssh server no graphic card, no X11, text mode 3.0.0-12 generic amd64 thomas@ubuntu2:\~\$ cat /etc/issue Ubuntu 11.10 \n \l

thomas@ubuntu2:\~\$ sudo lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 11.10 Release: 11.10 Codename: oneiric

thomas@ubuntu2:\~\$ uname -a Linux ubuntu2 3.0.0-12-generic #20-Ubuntu SMP Fri Oct 7 14:56:25 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

MTU:1500

lspci old convert log: ./17/screenshot_2011-11-17-150007.png)

ifconfig -a old convert log: ./17/screenshot_2011-11-17-150544.png)

[ 184.646085] ixgbe 0000:01:00.0: eth1: NIC Link is Up 10 Gbps, Flow Control: RX/TX [ 184.646774] ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready [ 184.953305] ixgbe 0000:01:00.0: eth1: NIC Link is Down [ 185.100439] ixgbe 0000:01:00.0: eth1: NIC Link is Up 10 Gbps, Flow Control: RX/TX

Machine A 192.168.5.1 Machine B 192.168.5.3 10G switch

test ping: thomas@ubuntu:\~\$ ping 192.168.5.3 PING 192.168.5.3 (192.168.5.3) 56(84) bytes of data. 64 bytes from 192.168.5.3: icmp_req=1 ttl=64 time=0.240 ms 64 bytes from 192.168.5.3: icmp_req=2 ttl=64 time=0.256 ms 64 bytes from 192.168.5.3: icmp_req=3 ttl=64 time=0.252 ms 64 bytes from 192.168.5.3: icmp_req=4 ttl=64 time=0.271 ms 64 bytes from 192.168.5.3: icmp_req=5 ttl=64 time=0.269 ms 64 bytes from 192.168.5.3: icmp_req=6 ttl=64 time=0.275 ms 64 bytes from 192.168.5.3: icmp_req=7 ttl=64 time=0.259 ms 64 bytes from 192.168.5.3: icmp_req=8 ttl=64 time=0.276 ms 64 bytes from 192.168.5.3: icmp_req=9 ttl=64 time=0.279 ms 64 bytes from 192.168.5.3: icmp_req=10 ttl=64 time=0.276 ms \^C --- 192.168.5.3 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 8997ms rtt min/avg/max/mdev = 0.240/0.265/0.279/0.017 ms

test ncat tcp:

  1. [TCP protocol]

============================================== (1) recive firstly at B ncat -l 60000 > /dev/null

​(2) then send at A cat /dev/zero | ncat 192.168.5.3 60000 ==============================================

Summary Result: send rate 4456937 kbits receive rate 2844327 kbits

cat /dev/zero | ncat -u 192.168.5.3 60000 old convert log: ./17/screenshot_2011-11-17-155814.png)

ncat -l 60000 > /dev/null old convert log: ./17/screenshot_2011-11-17-155846.png)

test ncat udp:

  1. [UDP protocol]
=======================================

​(1) recive firstly ncat -u -l 60000 > /dev/null

​(2) then send cat /dev/zero | ncat -u [IP_Number] 60000

=======================================

Summary Result: send rate 2344620 kbits receive rate 2054655 kbits

cat /dev/zero | ncat -u [IP_Number] 60000 old convert log: ./17/screenshot_2011-11-17-160538.png)

ncat -u -l 60000 > /dev/null old convert log: ./17/screenshot_2011-11-17-160611.png)

test iperf: thomas@ubuntu:\~\$ iperf -s


Server listening on TCP port 5001 TCP window size: 85.3 KByte (default)


[ 4] local 192.168.5.1 port 5001 connected with 192.168.5.3 port 51698 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 4.97 GBytes 4.26 Gbits/sec

thomas@ubuntu2:\~\$ iperf -c 192.168.5.1


Client connecting to 192.168.5.1, TCP port 5001 TCP window size: 16.0 KByte (default)


[ 3] local 192.168.5.3 port 51698 connected with 192.168.5.1 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 4.97 GBytes 4.27 Gbits/sec

test scp: trabsfer 8G file via scp B to A rsync -P file a:\~/ thomas@ubuntu2:\~\$ dd if=/dev/zero of=file bs=1M count=8192 8192+0 records in 8192+0 records out 8589934592 bytes (8.6 GB) copied, 62.7039 s, 137 MB/s thomas@ubuntu2:\~\$ rsync -P file 192.168.5.1:\~/ The authenticity of host '192.168.5.1 (192.168.5.1)' can't be established. ECDSA key fingerprint is f0:36:69:7a:d9:76:9a:78:a1:9c:fd:5e:89:0a:d7:fc. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '192.168.5.1' (ECDSA) to the list of known hosts. thomas@192.168.5.1's password: file 1796669440 20% 49.37MB/s 0:02:14 \^C rsync error: unexplained error (code 130) at rsync.c(549) [sender=3.0.8]

test After TCP TUNING echo 262144 > /proc/sys/net/core/rmem_default echo 8388608 > /proc/sys/net/core/wmem_max echo 8388608 > /proc/sys/net/core/rmem_max echo "4096 87380 4194304" > /proc/sys/net/ipv4/tcp_rmem echo "4096 65536 4194304" > /proc/sys/net/ipv4/tcp_wmem

thomas@ubuntu2:\~\$ sudo iperf -c 192.168.5.1


Client connecting to 192.168.5.1, TCP port 5001 TCP window size: 64.0 KByte (default)


[ 3] local 192.168.5.3 port 51699 connected with 192.168.5.1 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 6.87 GBytes 5.90 Gbits/sec

test chang cable 5e -> 6 thomas@ubuntu2:\~\$sudo iperf -c 192.168.5.1


Client connecting to 192.168.5.1, TCP port 5001 TCP window size: 64.0 KByte (default)


[ 3] local 192.168.5.3 port 51700 connected with 192.168.5.1 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 10.9 GBytes 9.34 Gbits/sec

test netcat TCP:

cat /dev/zero | ncat --send-only 192.168.5.3 60000 send: TX: cumm: 82.4GB peak: 3.86Gb rates: 2.97Gb 3.45Gb 2.80Gb RX: 112MB 4.80Mb 4.53Mb 4.60Mb 4.07Mb total: 82.5GB 3.87Gb 2.97Gb 3.46Gb 2.81Gb

ncat --recv-only -l 60000 > /dev/nul receive: TX: cumm: 105MB peak: 4.79Mb rates: 4.46Mb 4.56Mb 4.02Mb RX: 76.8GB 3.85Gb 3.08Gb 3.42Gb 2.77Gb Total: 76.9GB 3.85Gb 3.08Gb 3.42Gb 2.77Gb

test netcat UDP: cat /dev/zero | ncat -u --send-only 192.168.5.3 60000 TX: cumm: 34.7GB peak: 2.67Gb rates: 2.67Gb 2.36Gb 2.30Gb RX: 1.11KB 1.11Kb 0b 0b 57b thomas@ubuntu:\~\$ 34.7GB 2.67Gb 2.67Gb 2.36Gb 2.30Gb

ncat -u --recv-only -l 60000 > /dev/nul TX: cumm: 0B peak: 0b rates: 0b 0b 0b RX: 36.4GB 3.13Gb 3.13Gb 2.54Gb 2.34Gb thomas@ubuntu2:\~\$ 36.4GB 3.13Gb 3.13Gb 2.54Gb 2.34Gb

test dd| netcat : ncat -l 60000 > /dev/nul dd if=/dev/zero bs=512 count=1024000000 | ncat 192.168.5.3 60000 iftop result

test using dd: 128-dd.log 160000000+0 records in 160000000+0 records out 20480000000 bytes (20 GB) copied, 57.4505 s, 356 MB/s

256-dd.log 80000000+0 records in 80000000+0 records out 20480000000 bytes (20 GB) copied, 38.8449 s, 527 MB/s

512-dd.log 40000000+0 records in 40000000+0 records out 20480000000 bytes (20 GB) copied, 42.393 s, 483 MB/s

1024-dd.log 20000000+0 records in 20000000+0 records out 20480000000 bytes (20 GB) copied, 41.1297 s, 498 MB/s

2048-dd.log 0000000+0 records in 10000000+0 records out 20480000000 bytes (20 GB) copied, 40.6217 s, 504 MB/s

4096-dd.log 5000000+0 records in 5000000+0 records out 20480000000 bytes (20 GB) copied, 41.2186 s, 497 MB/s

8192-dd.log 2500000+0 records in 2500000+0 records out 20480000000 bytes (20 GB) copied, 41.0573 s, 499 MB/s

big server

memory 24G raid 1+0 (20T->14T)

zero to null

dd if=/dev/zero bs=256 count=80000000 80000000+0 records in 80000000+0 records out 20480000000 bytes (20 GB) copied, 41.6271 s, 492 MB/s

dd if=/dev/zero bs=512 count=40000000 40000000+0 records in 40000000+0 records out 20480000000 bytes (20 GB) copied, 28.4953 s, 719 MB/s

dd if=/dev/zero bs=1024 count=20000000 20000000+0 records in 20000000+0 records out 20480000000 bytes (20 GB) copied, 26.1946 s, 782 MB/s

dd if=/dev/zero bs=2048 count=10000000 10000000+0 records in 10000000+0 records out 20480000000 bytes (20 GB) copied, 25.3085 s, 809 MB/s

dd if=/dev/zero bs=4096 count=5000000 5000000+0 records in 5000000+0 records out 20480000000 bytes (20 GB) copied, 25.4756 s, 804 MB/s

iftop peek =\~ 6.3Gb

raw to null dd if=raw bs=256 80000000+0 records in 80000000+0 records out 20480000000 bytes (20 GB) copied, 59.171 s, 346 MB/s

dd if=raw bs=512 40000000+0 records in 40000000+0 records out 20480000000 bytes (20 GB) copied, 29.9193 s, 685 MB/s

dd if=raw bs=1024 20000000+0 records in 20000000+0 records out 20480000000 bytes (20 GB) copied, 27.893 s, 734 MB/s

dd if=raw bs=2048 10000000+0 records in 10000000+0 records out 20480000000 bytes (20 GB) copied, 26.8301 s, 763 MB/s

dd if=raw bs=4096 5000000+0 records in 5000000+0 records out 20480000000 bytes (20 GB) copied, 27.0086 s, 758 MB/s

peek 5.8Gb

raw to raw

dd if=raw bs=256 80000000+0 records in 80000000+0 records out 20480000000 bytes (20 GB) copied, 54.2189 s, 378 MB/s

dd if=raw bs=512 40000000+0 records in 40000000+0 records out 20480000000 bytes (20 GB) copied, 48.417 s, 423 MB/s

dd if=raw bs=1024 20000000+0 records in 20000000+0 records out 20480000000 bytes (20 GB) copied, 54.0252 s, 379 MB/s

dd if=raw bs=2048 dd if=raw bs=2048 10000000+0 records in 10000000+0 records out 20480000000 bytes (20 GB) copied, 46.5879 s, 440 MB/s

Ncat: Connection refused. dd if=raw bs=4096 5000000+0 records in 5000000+0 records out 20480000000 bytes (20 GB) copied, 46.3077 s, 442 MB/s

=================================================== test note about lenovo M80 7521 there is only one pci express 16x port, remove original graphic card and install intel 10g net card. boot get 5 bee warrning, skip it, it will keep booting without graphic card old convert log: ./17/tag%3E%20calendar%202011%2011)

Comments