APAN Servers Relocation

Lee, Jeahwa ANAN-KR NOC Pensri A.

APAN-TH NOC Yoshitaka Hattori APAN-JP NOC

Summary • Replacing servers on Tokyo XP

– Old NS and NS2 servers on Tokyo XP were replaced by three new servers – Some operation tools were restructured

• APAN Servers Relocation

– New servers were installed on KR, TH and JP NOC – How to make the redundancy and synchronization method for APAN web service

Replacing servers on Tokyo XP

• Replaced old NS and NS2 on TokyoXP by three new ones – NS, NS2 and TOOLS – They were built from bare-bone kit with some generic server parts – New OS (FreeBSD 5.2.1R) were installed on them • Some operating tools were restructured and rebuilt on new TOOLS server – MRTG, cricket+RRDTool, smokeping, Nagios… NS DNS(Pri jp.apan.net, Sec apan.net, kr.apan.net) Operation Tools, Web(www.jp.apan.net) NS2 DNS(Sec jp.apan.net), Operation Tools, Web(Backup of apan.net) OLD NS DNS(Pri jp.apan.net, Sec apan.net, kr.apan.net), Web (jp.apan.net) NS2 DNS(Sec jp.apan.net), Web (Backup of apan.net) Tools Operation Tools, Web(Tools) NEW

Specification of the Servers

• Here is the JP servers specification. KR and TH NOC installed same class of servers.

• SuperMicro 8023P-8R bare-bone kit – 2U, Redundant PS, 1000BaseT x2, PCI-X – http://www.supermicro.com/products/system/2U/6023/SYS-6023P-8R.cfm

• Generic server parts – Intel Xeon 3.0GHz (Single), Memory 2GBytes – RAID card (Adaptec Zero Channel SCSI RAID) – HDD 70Gbytes (Seagate TokyoXP • FreeBSD 5.2.1R

• Apache, BIND… ST373453LC) • 2 disks installed on NS and NS2 with RAID1(mirror) - total disk size 70Gbytes, 3 disks installed on TOOLS with RAID5(N+1) - total disk size 140Gbytes • The spare HDDs can be shared with some existing SuperMicro’s servers on • Future upgradeable - CPU, memory, HDD

Effect of replacing servers on TokyoXP

• Decreasing Load average – NS 1.60->0.00, NS2 0.20->0.00, Tools 0.10

• Decreasing Swap space – NS and NS2 5~6Mbytes -> 0, Tools 0 • Decreasing Disk usage – NS and NS2 90% -> 15% (9/70Gbytes) – Tools 45% (60/140Gbtyes) • Hard ware redundancy – Changing Disk controller (IDE->SCSI RAID) • Can provide disk redundancy • Can minimize down time, when disk fails – Redundant power supply • Can zero down time, when power supply fails • Quick Web response

Photos of the Servers on Tokyo XP

Redundant Hotswapable Power Supply They were temporarily placed on the NOC room for burn-in test and OS installation Observatory servers by Ikeda-san 1000BaseT x2 The servers ware installed on the housing rack with rail mount kit

APAN Servers Relocation

• APAN-KR – JHLee-san replaced and relocated the DNS server of apan.net and installed a new one for web redundancy • APAN-TH – Pensri-san and Tatum-san installed a new server for master web server • APAN-JP – APAN TokyoXP replaced old NS and NS2 by three new servers • JHLee-san(KR) sent the master web contents of http://apan.net/ to Pensri-san(TH) by air-mail with burned CD-R media

Building TH-JP Synchronization

• The bandwidth of TH-JP is several Mbps – Each destination route is via Abilene at that time – Bottle neck is TH local link – The existing TH-JP direct link was upgraded to 135Mbps(UBR 68Mbps) at just one week ago • Using SSH-wrapped rsync with data compression option for web contents synchronization – Destination is from TH to JP – Running every 4 hours (0:20, 4:20, 8:20…) now • Is it too long interval for quick update?

Rsync Test on TH-JP

• Used SSH-wrapped rsync with data compression option( z) • Tested on each destination, TH->JP and JP->TH • Tested with total 10 Mbytes of dummy files • The data transfer ratio on the test was up to nearly

600kbps ns2% rsync -azv -e ssh [email protected]::test/ ~/test/ receiving file list ... done : ap_jointsec/ wrote 6067 bytes read 6677109 bytes 76379.15 bytes/sec total size is 10107667 speedup is 1.51

Rsync Test on TH-JP (cont.)

• The result has proved enough basis of using rsync for synchronizing the web contents – Rsync enable up to find out updated contents and transfer only such contents – Most of the space is filled up by presentation materials of past meetings, which will hardly ever be modified – The HTML files, which will be modified frequently will be small enough compared to the whole contents (6 Gbytes)

Rsync Test on TH-JP (cont.)

• Considering the worst case – The amount of the update will be the largest before and after an APAN meeting – There is about 2.4Gbytes of contents for the past Busan meeting • http://www.apan.net/meetings/busan03/ – The overall update is not expected to be done at one time – Assuming the result of past test ( 600kbits/sec=75kbytes/sec ) is maintained, the time to check the updated files by rsync is about 15 sec, which can be disregard • The time to synchronize whole of contents = 2,400,000,000 / 75,000 = 32,000 sec (nearly 9 hours) – It exceeds the update interval (4 hours), but multiple rsync processes will not run because rsync has lock capability

Others… • Tatum-san of the Secretariat rewrote all of absolute links on the contents with relative ones • There was Japanese Kanji code problem

– It was solved by correcting httpd.conf

TO-DO

• Additional rsync test will be done on new TH-JP link to compare the result with that of the past test – Can synchronization interval be shortened?

– 1 hour? 15 minutes? Real-time???

• Primary DNS server for apan.net should be changed from 192.249.24.62 to 203.255.248.57

– It will be done by JHLee-san soon • Redundancy test should be done • Anycast is one of good ideas for enabling more redundancy, and web redirection also

Thanks