Transcript Remote Observing with the Keck Telescopes

Remote observing with the Keck
Telescopes from multiple sites in
California
Robert Kibrick, Brian Hayes, Steve Allen
University of California Observatories /
Lick Observatory
Al Conrad
W.M. Keck Observatory
Advanced Global Communications
Technologies for Astronomy II
7/17/2015
1
Overview of Presentation
• Background
– The Keck Telescopes and telescope scheduling
• Remote observing with the Keck Telescopes
– From remote control room at summit (30 meters)
– From Keck Headquarters in Waimea, Hawaii (32 km)
– From Santa Cruz, California via Internet2 (3200 km)
• Network Reliability Concerns
• Providing a backup data path
• Recent operational experience
• Extending the model to multiple sites
7/17/2015
2
The Keck Telescopes
7/17/2015
3
Keck Telescopes use Classical
Scheduling
• Kecks not designed for queue scheduling
• Schedules cover a semester (6 months)
• Approved proposals get 1 or more runs
–Each run is between 0.5 to 5 nights long
–Gaps between runs vary from days to months
Half of all runs are either 0.5 night
or 1 night long
7/17/2015
4
From 1993 to 1995, all Keck
observing was done at the summit
Observers
at the
summit
work from
control
rooms
located
adjacent
to the
telescope
domes
7/17/2015
5
Conducting observations involves
coordinated effort by 3 groups
• Telescope operator (observing assistant)
– Responsible for telescope safety & operation
– Keck employee; normally works at summit
• Instrument scientist
– Expert in operation of specific instruments
– Keck employee; works at summit or Waimea
• Observers
– Select objects and conduct observations
– Employed by Caltech, UC, NASA, UH, or other
7/17/2015
6
Keck 2 Control Room at the Mauna
Kea Summit
Telescope
operator,
instrument
scientist,
and
observers
work side
by side,
each at
their own
computer.
7/17/2015
7
Observing at the Mauna Kea
summit is both difficult and risky
• Oxygen is only 60% of that at sea level
• Lack of oxygen reduces alertness
• Observing efficiency significantly impaired
• Altitude sickness afflicts some observers
• Some are not even permitted on summit:
–Pregnant women
–Those with heart or lung problems
7/17/2015
8
Initiative to support remote
observing from Keck Headquarters
• 1995: Remote control rooms built at Keck HQ
• Initial tests via 1.5 Mbps (T1) link to the summit
• 1996: Videoconferencing connects both sites
• Remote observing with Keck 1 begins
• 1997: >50% of Keck 1 observing done remotely
• Link to the summit upgraded to 45 Mbps (DS3)
• 1999: remote observing >90% for Keck 1 and 2
• 2000: remote observing now the default mode
7/17/2015
9
The Remote Observing Facility at
Keck Headquarters in Waimea
• Elevation of Waimea is 800 meters
• Adequate oxygen for alertness
• Waimea is 32 km NW of Mauna Kea
• 45 Mbps fiber optic link connects 2 sites
• A remote control room for each telescope
• Videoconferencing for each telescope
• On-site dormitories for daytime sleeping
7/17/2015
10
Keck 2 Remote Control Room at
the Keck Headquarters in Waimea
Observer and
instrument
scientist in
Waimea use
video
conferencing
system to
interact with
telescope
operator at
the summit
7/17/2015
11
Keck 2 Remote Observing Room
as seen from the Keck 2 summit
Telescope
operators at the
summit converse
with astronomer at
Keck HQ in
Waimea via the
videoconferencing
system.
7/17/2015
12
Videoconferencing has proved vital
for remote observing from Waimea
• Visual cues (body language) important!
• Improved audio quality extremely valuable
• A picture is often worth a thousand words
• Troubleshooting: live oscilloscope images
• “Cheap” desktop sharing (LCD screens)
• Chose dedicated versus PC-based units:
–Original (1996) system was PictureTel 2000
–Upgrading to Polycom Viewstations
7/17/2015
13
Interaction between videoconferencing and type of monitors
• Compression techniques motion sensitive
• “Moving” scene requires more bandwidth
• CRT monitors cause “flicker” in VC image
– Beating of frequencies: camera .vs. CRT
– CRT phosphor intensity peaking, persistence
• CRT monitor “flicker” causes problems:
– Wastes bandwidth and degrades resolution
– Visually annoying / nausea inducing
• Use LCD monitors to avoid this problem
7/17/2015
14
The Keck Headquarters in Waimea
Most Keck technical
staff live and work in
Waimea. Allows
direct contact
between observers
and staff. Visiting
Scientist’s Quarters
(VSQ) located in
same complex.
7/17/2015
15
Limitations of Remote Observing
from Keck HQ in Waimea
Most Keck observers
live on the mainland.
Mainland observers
fly > 3,200 km to get
to Waimea
Collective direct
travel costs exceed
$400,000 U.S. / year
7/17/2015
16
Remote Observing from Waimea is
not cost effective for short runs
• Round trip travel time is 2 days
• Travel costs > $1,000 U.S. per observer
• About 50% of runs are for 1 night or less
• Cost / run is very high for such short runs
• Such costs limit student participation
7/17/2015
17
Motivations for Remote Observing
from the U.S. Mainland
• Travel time and costs greatly reduced
• Travel restrictions accommodated
–Sinus infections and ruptured ear drums
–Late stages of pregnancy
• Increased options for:
–Student participation in observing runs
–Large observing teams with small budgets
• Capability for remote engineering support
7/17/2015
18
Mainland remote observing goals
and implementation strategy
• Goals:
–Target mainland facility to short duration runs
–Avoid duplicating expensive Waimea resources
–Avoid overloading Waimea support staff
• Strategy:
–No mainland dormitories; observers sleep at home
–Access existing Waimea support staff remotely
–Restrict mainland facility to experienced observers
–Restrict to mature, fully-debugged instruments
7/17/2015
19
Mainland remote observing facility
is an extension of Keck HQ facility
• Only modest hardware investment needed:
–Workstations for mainland remote observers
–Network-based videoconferencing system
–Routers and firewalls
–Backup power (UPS) – especially in California!!!
–Backup network path to Mauna Kea and Waimea
• Avoids expensive duplication of resources
• Share existing resources wherever possible
–Internet-2 link to the mainland
–Keck support staff and operational software
7/17/2015
20
Keck software is accessed the same
regardless of observer’s location
• The control computers at the summit:
–Each telescope and instrument has its own computer
–All operational software runs only on these computers
–All observing data written to directly-attached disks
–Users access data disks remotely via NFS or ssh/scp
• The display workstations
–Telescopes and instruments controlled via X GUIs
–All users access these X GUIs via remote X displays
–X Client software runs on summit control computers
–Displays to X server on remote display workstation
7/17/2015
21
Overall Topology
Type your question here, and the
7/17/2015
22
View Empty
7/17/2015
23
Santa Cruz Remote Observing
Facility
7/17/2015
24
Santa Cruz Remote Observing
Video Conferencing
7/17/2015
25
The Weather in Waimea
7/17/2015
26
Why did we choose this approach?
• Operational Simplicity
• Operational control software runs only at the summit
• All users run identical software on same computer
• Simplifies management between independent sites
• Allowed us to focus on commonality
• Different sites / teams developed instrument software
• Large variety of languages and protocols were used
• BUT: all instruments used X-based GUIs
7/17/2015
27
Remote observing differences:
Waimea versus the mainland
• System Management:
–Keck summit and HQ share a common domain
–Mainland sites are autonomous
• Remote File Access:
–Observers at Keck HQ access summit data via NFS
–Observers on mainland access data via ssh/scp
• Propagation Delays:
–Summit to Waimea round trip time is about 1 ms.
–Summit to mainland round trip time is about 100 ms.
7/17/2015
29
Increased propagation delay to
mainland presents challenges
• Initial painting of windows is much slower
• But once created, window updates fast enough
• All Keck applications display to Waimea OK
• A few applications display too slowly to mainland
• System and application tuning very important
–TCP window-size parameter (Web100 Initiative)
–X server memory and backing store
–Minimize operations requiring round trip transactions
7/17/2015
30
Simulating long propagations
delays in the lab
• Instruments are designed and built on mainland
• Software is debugged on local area network
• Testing on LAN does not reveal delay problems
• Must measure delay effects before deployment
• Simulate WAN delays using NIST simulator
–Requires Linux PC with dual Ethernet interfaces
–Can select specific packets delays, losses, jitters
–http://www.antd.nist.gov/itg/nistnet
7/17/2015
31
Shared access and control of
instruments
• Most software for Keck optical instruments
provides native multi-user/multi-site control
• All users have consistent view of status and data
• Instrument control can be shared between sites
• Multipoint video conferencing key to coordination
• Some single-user applications can be shared via
X-based application sharing environments:
– XMX http://www.cs.brown.edu/software/xmx
– VNC http://www.uk.research.att.com/vnc
7/17/2015
32
Tradeoffs from this approach to
remote observing
• Disadvantages:
–X protocol does not make optimal use of bandwidth
–Long propagation delays require considerable tuning
• Advantages:
–Minimizes staffing requirements at mainland sites
–Only “vanilla” hardware and software needed there
–Simplifies sparing and swapping of equipment
–Simplifies system maintenance at mainland sites
–Simplifies authentication/access control
7/17/2015
33
Fast and reliable network needed
for mainland remote observing
•
•
•
1997: 1.5 Mbps Hawaii -> Oahu -> mainland
1998: 10 Mbps from Oahu to mainland
1999: First phase of Internet-2 upgrades:
– 45 Mbps commodity link Oahu -> mainland
– 45 Mbps Internet-2 link Oahu -> mainland
•
2000: Second phase upgrade:
– 35 Mbps Internet-2 link from Hawaii -> Oahu
– Now 35 Mbps peak from Mauna Kea to mainland
•
2002: 155 Mbs from Oahu to mainland
7/17/2015
34
End-to-end reliability is critical to
successful remote operation
• Keck Telescope time is valued at $1 per second
• Observers won’t use facility if not reliable
• Each observer gets only a few nights each year
• What happens if network link to mainland fails?
–Path from Mauna Kea to mainland is long & complex
–At least 14 hops crossing 6 different network domains
–While outages are rare, consequences are severe
–Even brief outages cause session collapse & panic
–Observing time loss can extend beyond outage
7/17/2015
35
Keck Observatory policy on
mainland remote observing
• If no backup data path is available from mainland
site, at least one member of observing team
must be in Waimea
• Backup data path must be proven to work before
mainland remote observing is permitted without
no team members in Waimea
7/17/2015
36
Mitigation plan: install end-to-end
ISDN-based fall-back path
• Install ISDN lines and routers at:
–Each mainland remote observing site
–Keck 1 and Keck 2 control rooms
• Fail-over and fall-back are rapid and automatic
• Toll charges incurred only during network outage
• Lower ISDN bandwidth reduces efficiency, but:
–Observer retains control of observations
–Sessions remain connected and restarts avoided
–Prevents observer panic
7/17/2015
37
Summary of ISDN-based fallback
path
• Install 3 ISDN lines (6 B channels) at each site
• Install Cisco 2600-series routers at each end
–Quad BRI interfaces
•
•
•
•
Inverse multiplexing
Caller ID (reject connections from unrecognized callers)
Multilink PPP with CHAP authentication
Dial-on-demand (bandwidth-on-demand)
• No manual intervention needed at either end
• Fail-over occurs automatically within 40 seconds
• Uses GRE tunnels, static routes, OSPF routing
7/17/2015
38
Running OSPF routing over a
GRE tunnel
• On each router, we configure 3 mechanisms:
–A GRE tunnel to the other endpoint
–A floating static route that routes all traffic to the other
endpoint via the ISDN dialer interface
–A private OSPF domain that runs over the tunnel
• OSPF maintains its route through the tunnel only
if the tunnel is “up”
• OSPF dynamic routes take precedence over
floating static route
7/17/2015
39
Fail-over to ISDN backup data path
• If the Internet-2 path is “up”, OSPF “hello”
packets flow across the tunnel between routers
• As long as “hello” packets flow, OSPF maintains
the dynamic route, so traffic flows through tunnel
• If Internet-2 path is “down”, OSPF “hello” packets
stop flowing, and OSPF deletes dynamic route
• With dynamic route gone, floating static route is
enabled, so traffic flows through ISDN lines
7/17/2015
40
Fall-back to the normal Internet-2
path
• OSPF keeps trying to send “hello” packets
through the tunnel, even with Internet is down
• As long as Internet-2 path remains down the
“hello” packets can’t get through
• Once the Internet-2 path is restored, “hello”
packets flow between routers
• OSPF re-instates dynamic route through tunnel
• All current traffic gets routed through the tunnel
• All ISDN calls are terminated
7/17/2015
41
Operational costs of ISDN backup
data path
• Fixed leased cost is $70 per line per month
• Three lines at each site -> $2,500 per site/year
• Both sites -> $5,000/year
• Long distance cost (incurrent only when active)
– $0.07 per B-channel per minute
– If all 3 lines in use:
• $0.42 per minute
• $25.20 per hour
7/17/2015
42
Recent operational experience
• Remote observing science from Santa Cruz:
– Low Resolution Imaging Spectrograph (LRIS)
– Echellete Spectrograph and Imager (ESI)
• Remote engineering and instrument support
– ESI
– High Resolution Echelle Spectrometer (HIRES)
• Remote Commissioning Support
– ESI
– DEIMOS (see SPIE paper 4841-155 & 4841-186)
7/17/2015
43
Unplanned use of the facility
during week of Sept. 11, 2001
• All U.S. commercial air traffic grounded
• Caltech astronomers have a 5-day LRIS run on
Keck-I Telescope starting September 13
• No flights available
• Caltech team leaves Pasadena morning of 9/13
• Drives to Santa Cruz, arriving late afternoon
• Online with LRIS well before sunset in Hawaii
7/17/2015
44
The hardest problem was the
lodging!
• LRIS operated from Santa Cruz all 5 nights
• ISDN backup path activated several times
• Observing efficiency comparable to Waimea
• Lodging was the biggest problem
• Motel check-in/check-out times incompatible
• Required booking two motels for the same night
• Motels are not a quiet place for daytime sleep
7/17/2015
45
Extending mainland remote
observing to other sites
• Other sites motivated by Santa Cruz success
• Caltech remote facility is nearly operational
– Equipment acquired
– ISDN lines and router installed
– Will be operational once routers are configured
• U.C. San Diego facility being assembled
– Equipment specified and orders in progress
• Other U.C. campuses considering plans
7/17/2015
46
Administrative challenges:
scheduling shared facilities
• Currently only one ISDN router at Mauna Kea
• Limits mainland operation to one site per night
• Interim administrative solution
• Longer term solution may require:
– Installation of additional ISDN lines at Mauna Kea
– Installation of an additional router at Mauna Kea
7/17/2015
47
Remaining challenges
• TCP/IP tuning of end-point machines
–Needed to achieve optimal performance
–Conflicts with using “off-the-shelf” workstations
–Conflict between optimal TCP/IP parameters for the
normal Internet-2 path .vs. the ISDN fall-back path
–Hoping for vendor-supplied auto-tuning
–Following research efforts of Web100 Project
• Administrative challenges
–Mainland sites are currently autonomous
–Need to develop coordination with Keck
7/17/2015
48
Summary
• Internet-2 makes mainland operation feasible
• Backup data path protects against interruptions
• Keck HQ is the central hub for remote operation
• Mainland remote observing model is affordable:
–Mainland sites operate as satellites of Keck HQ
• Leverage investment in existing facilities and staff
• Leverage investment in existing software
–Share existing resources wherever feasible
–Avoid expensive and inefficient travel for short runs
• Model is being extended to multiple sites
7/17/2015
49
Acknowledgments
• U.S. National Science Foundation
• U.S. Department of Defense
• University of Hawaii
• Gemini Telescope Consortium
• University Corp. for Advanced Internet
Development (UCAID)
• Corporation for Education Network Initiatives in
California (CENIC)
7/17/2015
50
Author Information
Robert Kibrick, UCO/Lick Observatory
University of California, Santa Cruz
California 95064, U.S.A.
E-mail: [email protected]
WWW: http://www.ucolick.org/~kibrick
Phone: +1-831-459-2262
FAX: +1-831-459-2298
7/17/2015
51