Transcript LIGO Data Grid and KISTI - Korean Gravitational Wave Group

June 27, 2015 at 8th J-K Joint Workshop on KAGRA, Gwangju in Korea
LIGO Data Grid and KISTI
Gungwon Kang & Jiwoong Kim (KISTI)
2
OUTLINE
I. KISTI GSDC Overview
II. KISTI LDG (LIGO Data Grid)
III. Conclusion
3
I. KISTI GSDC Overview
KISTI: Korea Institute of Science and Technology Information
• National research institute for information technology since 1962
• About 600 people working for Supercomputing & Networking and
National Information Service (development & analysis)
• Running High-Performance Computing Facility
• Total 3,398 Nodes (30,592 CPUs, 360 TFlops at peak), 1,667 TB
storage (introduced from 2008)
Rpeak: 300TF
CPU: 25,408
Memory: 76.8TB
Storage: 1,061TB
Intel Xeon X5570 2.93GHz (Nehalem)
4
GSDC: Global Science experiment Data hub Center
• National project to promote data-based science research experiments by
providing computing and storage resources: HEP and other fields
• Running Data-Intensive Computing Facility
• ~20 Staffs: system administration, experiment support, external-relations,
administration and students
• CPU: ~5,900 cores. Storage: ~6.8PB
• Budget: ~ 6M$/year
• Supporting experiments: ALICE, CMS, Belle, LIGO, RENO, Genomic Medicine, etc.
GSDC Facility
HP Servers
Hitachi VSP Storage
Resource allocations
2014.12
Field / Experiment
CPU (cores)
Storage (TB)
CERN (ALICE)
2,520
~2,000
KiAF
108
100
CERN (T3)
240
110
KEK
224
110
BNL (STAR)
540
110
RENO
280
252
HCP
346
180
LIGO
420
152
Genom
144
150
G-brain
208
52
PCMI
0
380
Etc.
870
126
Total
5,900
3,722
Particle/Nuclear
Physics
Astrophysics
Medical Science
Meteorology
National Institute of Supercomputing and Networking 2013. 10 . 28.
6
Model
Physical size
Usable size
104TB
50TB
NetApp FAS2050
GSDC CPU/Storage
(2014.12.30)
구분
(SAN only, RAID6)
세부사항
수량
☐ 모델명 : HP DL360G8(1U)
NetApp FAS6080
334TB
200TB
(SAN & NAS, RAID6)
☐ 사양 :
Comput
1,100 C
- E5-2680v2 2.8GHz
ing
Hitachi USP-V
960TB
600TB
(SAN & NAS, RAID6)
ore
* 2P(20core)
Server
- 128GB DDR3 1600Mhz SDRAM
(10Core
(Purcha
- 600GB 10K SAS * 4EA
EMC CX4-960C(SAN, RAID6)
1,920TB
1,250TB
x 55 No
sed in
- 1GbE 2Port, 10GbE 2Port
des)
2014)
- 8G HBA
EMC Isilon 108NL
2013
1,620TB
758TB
1,400TB
(SAN, NAS, RAID6)
320TB
구분
내역
Model
Hitach VSP, HNAS4080(4Node)
Disk
Usable 700TB
RAID
RAID6(6D+2P)
500TB
Hitach VSP 1
2014
- Redundant Power Supply
214TB
Hitach VSP 2
857TB
570TB
(SAN, NAS, RAID6)
Cache Memor
512GB
y
Total
6,873TB
4,784TB
Front-end Inter
8Gbps FC 32 ports
face
7
II. LIGO Data Grid (LDG)
+ KISTI LDG T3
8
Result Analysis
Discussion
Login Node
Web Server
User Authentication
based on GSI
ldas.ligo.kisti.re.kr
ui04.sdfarm.kr
(Condor)
lgm.sdfarm.kr (Intel-Compiler-License)
ce04.sdfarm.kr (Condor)
Storage
Cluster
LIGO / VIRGO Data
Tier1/2/3
ldr.sdfarm.kr
(GridFTP server)
155 TBs
KGWG F2F Meeting
576 cores (780)
wn3076~3110.sdfarm.kr (Condor)
8
• System Configuration in more detail:
ldas.ligo.kisti.re.kr
{web publication}
Connection/
Job Submission
ui test
{Condor}
ui04.sdfarm.kr
{Condor}
Central storage
150TB
lgm.sdfarm.kr
{Intel-Compiler-License}
ldr.ligo.caltech.edu
ldr.sdfarm.kr
{GridFTP server}
….
wn3076~3110.sdfarm.kr
{Condor}
9
+ KISTI LDG Resources (2015)
10
• Computation Resources(Worker Node)
• 48 Node : 780cores (Hyperthread-17node) / 576 cores (Physical)
Cores
RAM
Worker Node
780 / 576
72GB(Hyperthread)
48GB(W/O Hyperthread)
UI,CE,LGM,LDAS,LDR
60(12core per server)
24GB
Total
840 / 636
• Storage Resources(Only Data - /data/ligo/archive)
• 155 TB (Expandable to 200 TB)
Size
Used
Avail
Use (%)
/data/ligo/home
786GB
471GB
315GB
60
/data/ligo/lib
100GB
71GB
20GB
71
/data/ligo/scratch
4.73TB
3.96TB
784G
84
/data/ligo/archive
150TB
123TB
28TB
82
Total
155TB
127TB
29TB
82
KGWG F2F Meeting
10
+ Stored Data
- hoft frame: LIGO S5~6. Virgo VSR1~4
- RDS L1 frame: LIGO S6
11
+ KISTI LDG Usage:
12
• Work Node Usage: CPU 384 cores
- 2014.11~2015.02: Max 78.9%, Ave 13%
- 2015.01~2015.02: Max 60%, Ave 35%
(※ Monitored by KISTI)
기간
1월
2월
3월
2011
4월
5월
6월
7월
8월
9월
10월
11월
12월
건수
(컴퓨팅
시간, 일)
124
(81)
362
(309)
80
(171)
70
(726)
44
(500)
30
(4)
187
(193)
합계
128
1,025
(159) (2,143)
2012
62
55
22
10
19
30
30
1
0
0
?
?
229
2013
1,287
1,951
927
268
1
9
5,430
1,129
26
?
?
?
11,028
2014
331
918
921
703
367
881
1214
545
120
124
?
?
6,124
21,683
4,257
109
1
14
2015
9,335
35,400
* Job 건수 만이 아닌 새로운 모니터링 Measure 필요 (예, 2011년 6월, 8월 비교)
KGWG F2F Meeting
12
13
User groups (1)
• KGWG Korean Gravitational Wave Group (2008~):
•
한국중력파연구협력단
~30 people working in 8 universities and 3 government-funded institutes
• LIGO-Virgo and KAGRA
소속
이름
서울대
이형목(PI)
연세대
김정리
오상훈
한양대
이현규
손재주
김경민
김환선
이철훈
추형석
KISTI
오정근
서강대
조규만
부산대
이창환
장행진
김영민
김지웅
김명국
윤희준
이형원
조희석
인제대
고려대
KGWG
NIMS
명지대
강궁원
김정초
KAERI
차용호
윤태현
GIST
강훈수
조동현
경북대
박명구
김재완
군산대
김상표
+ User groups (2)
14
- Mostly used by domestic researchers!
Parameter estimation:
Chunglee Kim(Yonsei U/ KISTI GSDC), Hyungwon Lee, Chungcho Kim (Inje U)
+ LSC collaborators (Caltech, NU, UWM, Monclair State Univ.)
+ KAGRA collaborators (Osaka University)
중력파 분석을 통한 모수추정(parameter estimation)
 보다 천체물리학적으로 “실제와 가깝고”, 계산적으로 효율적인 중력파형 개발

CBC Signal and Noise Identification:
오정근, 오상훈, 손재주, 김환선, 추형석 (NIMS), 이창환, 김영민(부산대)
iDQ pipeline의 개선 (Deep Learning) 및 인공신경망 모듈 이식
 Bank chisq를 이용한 detection statistic의 개선연구
 중력파 채널과 보조채널간의 Correlation Analysis (CAGMon)
 HHT를 이용한 Trigger generation 연구

※ KISTI 연구원: 강궁원, 장행진, 김지웅, 윤희준, 조희석 (KISTI-GSDC)
+
User groups (3)
이름
소속
Kazuhiro
Osaka
Hayama
University
Tatsusya
Osaka
Narikawa
University
Hideyuki
Osaka
Tagoshi
e-mail
계정
[email protected]
…
[email protected]
…
[email protected]
…
[email protected]
…
[email protected]
…
University
Osaka
Koh Ueno
University
Hirotaka
Yuzurihara
Osaka
University
+
이름
Kazuhiro Hayama
접속기록
-
narikawa pts/7 :pts/8:S.0 Wed Jun 3 16:49 - 16:51 (00:01)
narikawa pts/8 pascal.hep.osaka Wed Jun 3 16:49 - 16:51 (00:01)
narikawa pts/7 :pts/6:S.0 Thu May 21 12:12 - 12:43 (00:30)
narikawa pts/7 :pts/6:S.0 Thu May 21 11:20 - 11:51 (00:30)
narikawa pts/6 pascal.hep.osaka Thu May 21 11:20 - 12:45 (01:24)
narikawa pts/7 :pts/6:S.0 Thu May 21 10:52 - 10:52 (00:00)
Tue May 12 18:07 - 18:11 (00:04)
narikawa pts/18 :pts/14:S.0 Tue May 12 17:48 - 17:57 (00:09)
narikawa pts/14 pascal.hep.osaka Tue May 12 17:48 - 17:57 (00:09)
Tatsusya Narikawa
narikawa pts/19 :pts/16:S.0 Mon Apr 27 17:29 - 18:02 (00:32)
(588분)
narikawa pts/19 :pts/16:S.0 Mon Apr 27 15:50 - 16:54 (01:04)
narikawa pts/16 pascal.hep.osaka Mon Apr 27 15:50 - 18:02 (02:11)
narikawa pts/19 :pts/16:S.0 Mon Apr 27 14:59 - 14:59 (00:00)
narikawa pts/19 :pts/16:S.0 Mon Apr 27 12:38 - 13:10 (00:31)
narikawa pts/16 pascal.hep.osaka Mon Apr 27 12:38 - 14:59 (02:20)
narikawa pts/5 :pts/1:S.0 Thu Apr 23 17:59 - 18:04 (00:05)
narikawa pts/1 pascal.hep.osaka Thu Apr 23 17:59 - 18:04 (00:05)
narikawa pts/5 :pts/1:S.0 Thu Apr 23 17:56 - 17:56 (00:00)
narikawa pts/1 pascal.hep.osaka Thu Apr 23 17:55 - 17:56 (00:00)
narikawa pts/1 pascal.hep.osaka Thu Apr 23 17:54 - 17:55 (00:00)
+
이름
접속기록
Kazuhiro Hayama narikawa pts/7 :pts/8:S.0 Wed Jun 3 16:49 - 16:51 (00:01)
narikawa pts/8 pascal.hep.osaka Wed Jun 3 16:49 - 16:51 (00:01)
Tatsusya Narikawa
…
(588분)
narikawa pts/1 pascal.hep.osaka Thu Apr 23 17:55 - 17:56 (00:00)
narikawa pts/1 pascal.hep.osaka Thu Apr 23 17:54 - 17:55 (00:00)
Hideyuki Tagoshi ueno pts/19 :pts/18:S.0 Sat May 2 19:37 - 20:21 (00:43)
Koh Ueno
(250분)
ueno pts/18 pascal.hep.osaka Sat May 2 19:37 - 20:21 (00:43)
…
ueno pts/8 pascal.hep.osaka Tue Apr 21 21:00 - 21:07 (00:06)
yuzu pts/19 :pts/3:S.0 Tue Jun 9 06:09 - 06:09 (00:00)
Hirotaka Yuzurihara yuzu pts/3 pascal.hep.osaka Tue Jun 9 06:09 - 06:09 (00:00)
(28분)
…
yuzu pts/24 pascal.hep.osaka Mon Apr 27 17:39 - 17:48 (00:09)
18
LDG WatchTower
• LDG central monitoring system
• Ganglia installed on all KISTI
resources
• ce04.sdfarm.kr, a condor server,
gathers ganglia information of
LDG WNs.
•
8649 port is open to the watchtower.
•
129.89.57.50(watchtower.phys.uwm.edu)
19
Soft Wares Deployed
•
•
•
•
OS: Scientific Linux 6.1 ( ?)
Batch system for managing compute jobs: Condor-7.8.7 ( ?)
LIGO Data Grid : 5.2.2 https://www.lsc-group.phys.uwm.edu/daswg/download/repositories.html
More than 200 packages
Packages
FrameL
MetaIO
LAL Suite
Related Project:
PYLAL
GLUE
FrameCPP
(deprecated)
LDAS-TOOLS
GDS
NDS2-Client
Related Project:
PYNDS
Matappsutilitites
LVAlert
GRACEdb
LARS
LIGO-common
GSTLAL
GST-Plugins
Low-Latency
VOEvent
Description
for data_frame manipulation
for LIGO_LW files metadata manipulation
LIGO Algorithm Library [LAL] + LAL based Applications [LALApps]
Grid LSC User Environment
C++ interface to access frame structures
LIGO Global Diagnostics System
Part of DMT offline that allow the user to down-load LIGO data from the V2 LIGO Network Data
Servers.
A collection of MATLAB® based applications for LIGO data analysis
LIGO/Virgo Alert Tools
Gravitational Wave Candidate Event Database
LIGO Archival Service
Simple setup of Python ligo namespace
GSTLAL provides a self-contained suite of GStreamer elements (and dependencies) that
expose gravitational-wave data analysis tools from the LAL library for use in GStreamer signalprocessing pipelines.
A collection of scientific visualization plugins for GStreamer using Cairo-powered graphics
Mathematical operations plugins for GStreamer
Ligo low-latency data distribution server initialization
VOEvent is the standardized language used to report observation and for
describing observations of immediate astronomical events
20
III. Conclusion
• We have briefly introduced computing resources,
environments and operation status of the KISTI GSDC
LDG T3 center.
• We hope to develop a good collaboration in the
computing and data management of KAGRA in the
future.
21
THANK YOU
감사(感謝)합니다