Transcript Multi-Link Iridium Satellite Data Communication System Overview, Performance and Reliability from Summer 2003 NGRIP, Greenland Field Experiments June 23-July 17, 2003 Abdul Jabbar Mohammad, Graduate Research.

Multi-Link Iridium Satellite Data
Communication System
Overview, Performance and Reliability
from Summer 2003 NGRIP, Greenland
Field Experiments
June 23-July 17, 2003
Abdul Jabbar Mohammad, Graduate Research Assistant
Dr.Victor Frost, Dan F. Servey Distinguished Professor
(September 24, 2003)
University of Kansas
Motivation

Polar Radar for Ice Sheet Measurements (PRISM)

The communication requirements of PRISM field experiments in Greenland
and Antarctica include

Data telemetry from the field to the University

Access to University and web resources from field

Public outreach to increase the interest of student community (K-12) in scientific
research and enable the science community to virtually participate in polar
expeditions

Generic data communication for Remote field research

Mainstream communication system for polar science expeditions, field camps in
Arctic/Antarctic and other research purposes

Government and security use
University of Kansas
2
Introduction – Commercial Satellite Systems

Polar regions do not have conventional communication facilities (dial-up, DSL, Cable
Modem, etc) and are not serviced by most of the major broadband satellite systems.
Inmarsat
Intelsat
Globalstar
University of Kansas
3
Introduction – Special Purpose Satellite Systems

NASA satellites like ATS3, LES9,
GOES,
TDRS
1,and
MARISAT2
provide broadband access to Polar
Regions

Geo-synchronous, they have a limited
visibility window at Poles – typically
10-13 hrs/day.

High satellite altitude and low elevation
angles (1-20) result in extremely large
field equipment.

May not be readily available
20 m diameter Marisat and GOES antenna at South Pole
University of Kansas
4
Introduction - Iridium Satellite System

The only satellite system with true pole-to-pole coverage

66 low earth orbiting (LEO) satellites with 14 spares

It has onboard satellite switching technology which allows it to service large
areas with fewer gateways

Since it was originally designed as a voice only system, it provides a low
data rate of 2.4Kbps

Not practical to be used as a main stream/ life-line communication system
University of Kansas
5
Introduction – Problem Statement

Problem Statement
A reliable, lightweight, portable and easily scalable data/Internet
access system providing true Polar coverage.

Solution
Implement a multi-link point-to-point Iridium communication system
to combine multiple satellite links to obtain a single logical channel
of aggregate bandwidth.
University of Kansas
6
Background - Iridium
Satellite Type
LEO
Satellite altitude
780 km
Minimum elevation angle
8.20
Average satellite view time
9-10 minutes
Access scheme
FDMA and TDMA
Maximum number of located users
80 users in a radius of 318 km
Theoretical throughput
2.4 – 3.45 Kbps
Type of data services
Iridium-to-Iridium, Iridium-to-PSTN
University of Kansas
7
Background - Inverse Multiplexing

Inverse Multiplexing - Data from a single application is
fragmented and or distributed over multiple low
bandwidth links.
Increases the
significantly
available
bandwidth
per

Multi-link point-to-point protocol (MLPPP), an extension
to the PPP is a packet based inverse multiplexing
solution
App 2
App 3
Multiplexing
application

Overhead of 12 bytes

Fragmentation of network layer protocol data units
(PDU’s) into smaller segments depending upon the PDU
size, link MTUs and the number of available links.
High Bandwidth Link
Low Bandwidth Links
App 1
Inv-Mux

App 1
Mux

Traditional Multiplexing - Data from a multiple
applications/users sent over a single high bandwidth link.
Inverse multiplexing
University of Kansas
8
Background - Multi-link point-to-point protocol

Layer 2 protocol that implements inverse multiplexing on a packet by packet basis

IP packets are encapsulated in to PPP frames with segment numbers – 12 byte overhead

Packet fragmentation depends on the number of available links and their capacity, packet
size and MTU size
Not Fragmented
Fragmented
IP Packet
IP Packet
SH
IP Packet
Modem i
SH
PDF
Modem 1
SH
PDF
SH
Modem 2
PDF
SH PDF
Modem 3 Modem 4
SH-Segment header; PDF- packet data fragment
University of Kansas
9
Multi-channel Iridium System – Design
The design requirements of the system are as follows.

The multi-channel implementation should maximize the throughput.

To support real-time interactions, the system should minimize the end-to-end delay.

The overall system should be reliable and have autonomous operation so as to handle
call drops and system/power failures in remote field deployment.
University of Kansas
10
Multi-channel Iridium System – Protocol Stack
Remote System
Local System
Application
Application
HTTP, FTP, SSH
HTTP, FTP, SSH
TCP
TCP
IP
IP
PPP/MLPPP
Physical Modems
point-to-point satellite links
PPP/MLPPP
Physical Modems
University of Kansas
11
Multi-channel Iridium System – Network Architecture
World Wide Web
NGRIP Camp, Greenland
100 Mbps
Ethernet
(Default gateway)
(Default gateway)
Guser 4
PPP Server
PPP Client
ITTC Default Router
eth0
User 2
100 Mbps
Ethernet
ppp0
P-T-P Satellite link
eth0
ppp0
User 1
Guser 3
User 3
Camp
WI-FI
Guser 2
G`user 1
ITTC Network, University of Kansas
University of Kansas
12
4-Channel Iridium System Implementation
I. Modem 1
I. Modem 2
I. Modem 3
I. Modem 4
Antenna
Grid
PPP client
USB-SERIAL
Remote
System
Iridium
Gateway
Remote Subsystem
PPP Server
Multi-port
PCI card
Local
System
Modem
Pool
PSTN
Local Subsystem
University of Kansas
13
4-Channel System – Implemented at KU
University of Kansas
14
4-Channel System – Implemented at KU
University of Kansas
15
4-Channel System – Software Overview

Linux based system

Open source PPP package

Custom software developed at University of Kansas

Chat scripts for Iridium modems

PPP script to tune link parameters for Iridium satellite system

Client-Server configuration

Autonomous operation-connection setup, user authentication, detecting
failures, reconnections, handling power failures/system resets, generating
status information (text)
University of Kansas
16
4-Channel System – Modem Flow Control
Check if any
modem is
alive
START
Connect
Modem 1
NO
FAIL
OK
YES
Terminate
All
OK
Monitor
Status
FAIL
FAIL
Connect
Modem 2
FAIL
FAIL
Connect
Modem 3
Connect
Modem 4
OK
OK
FAILED
OK
Monitor
Status
FAILED
OK
OK
Monitor
Status
FAILED
OK
Monitor
Status
University of Kansas
17
Field Tests and Results – Field implementation
System with
control software
USB In
USB Out
Antenna
Connectors
Dimensions: 24x19x5
University of Kansas
18
Field Tests and Results – Antenna Setup
3 FT
10 FT
University of Kansas
19
Results – Delay and Loss Measurement

Ping tests between the two machines at the end of the of satellite link

One way propagation delay = (800+8000+6000)Km / (3e6)Km/sec= 49msec

Transmission time for 64 [email protected] = 64*8/2400=213msec

Theoretically, the RTT delay =2*(49+213)= 524msec+delay at the gateway

Test results show an average RTT delay of 1.8 sec, which may be attributed to the intersatellite switching and delay at the gateway
RTT (sec)
Packet
s sent
Packets
received
%
Loss
Avg
Min
Max
mdev
50
50
0
1.835
1.347
4.127
0.798
100
100
0
1.785
1.448
4.056
0.573
100
100
0
2.067
1.313
6.255
1.272
200
200
0
1.815
1.333
6.228
0.809
University of Kansas
20
Results – Delay Measurement
Random variation of delay

High mean deviation

Delay increases linearly with
RTT Vs Time of the day
RTT in sec

packet size

Normalized delay is almost
8
7
6
5
4
3
2
1
0
Min
Avg
Max
1:00
constant for MTU sizes > 800
3:00
9:00
13:00
15:00
20:00
Time of the day in hh:mm
bytes
Normalized RTT Vs Packet Size
Changing distance between the
35
Normalized RTT (msec/byte)

user and satellite as well as
between satellites themselves
(ISLs)

Non-uniform traffic distribution,
25
32.05
26.50
20
15
13.83
12.35
10
9.24
7.51
7.52
5
0

varying delays on different routes
30
56 100 200 300
500
800
1000
7.04
1200
7.13
1500
Packet Size (Bytes)
University of Kansas
21
Results – Throughput



Method
1 Modem
2 Modems
3 Modems
4 Modems
Iperf
2.1
4.0
7.0
9.6
Iperf
1.9
3.9
7.0
9.3
Iperf
1.7
4.5
6.8
9.7
extend due to RTT variation
Ttcp
2.29
4.43
6.6
8.9
Efficiency > 90%
Ttcp
2.48
4.40
7.0
8.78
Average
2.1
4.25
6.88
9.26
Tools used – TTCP, IPERF
Throughput varies to some
Effective throughputs during large file transfers
Upload Time
(min)
Throughput
(bits/sec)
0.75
11
9091
3.2
60
7111
1.6
23
9275
2.3
45
6815
1.5
28
7143
2.5
35
9524
Throughput (K b p s)
File Size (MB)
Throughput Vs Number of Modems
10
9
8
7
6
5
4
3
2
1
0
9.26
6.88
4.25
2.1
1
2
3
Number of modems
University of Kansas
4
22
Time interval
between call drops
(minutes)
146
106
114
50
25
84
89
8
7
7
17
11
University of Kansas
21:15
20:30
19:45
19:00
18:15
17:30
16:45
16:00
15:15
14:30
13:45
13:00
12:15
11:30
10:45
10:00
9:15
8:30
7:45
7:00
6:15
5:30
4:45
4:00
3:15
2:30
1:45
1:00
0:15
23:30
22:45
22:00
21:15
Number of online
modems
21:03
20:21
19:39
18:57
18:15
17:33
16:51
16:09
15:27
14:45
14:03
13:21
12:39
11:57
11:15
10:33
9:51
9:09
8:27
7:45
7:03
6:21
5:39
4:57
4:15
3:33
2:51
2:09
1:27
0:45
0:03
23:21
22:39
21:57
21:15
Call drop event
Results – Reliability: 10th July 24 hr test
Call
dropon
events
vs.
Time
Call
drops
the first
modem
Total :
13 Call drops
Time
4
Modem up times during 24 hour test
Uptime %
3
2
1
0
80.6
91.8
94.7
96.8
Time
137
618
23
Time interval
between call drops
(minutes)
135
248
93
40
26
16
8
211
108
91
8
5
6
5
8
University of Kansas
14:57
14:21
13:45
13:09
12:33
11:57
11:21
10:45
10:09
9:33
8:57
8:21
7:45
7:09
6:33
5:57
5:21
4:45
4:09
3:33
2:57
2:21
1:45
1:09
0:33
23:57
23:21
22:45
22:09
21:33
20:57
20:21
19:45
19:09
18:33
17:57
17:21
16:45
16:09
15:33
14:57
Number of online modems
15:00
14:21
13:42
13:03
12:24
11:45
11:06
10:27
9:48
9:09
8:30
7:51
7:12
6:33
5:54
5:15
4:36
3:57
3:18
2:39
2:00
1:21
0:42
0:03
23:24
22:45
22:06
21:27
20:48
20:09
19:30
18:51
18:12
17:33
16:54
16:15
15:36
14:57
Call drop event
Results – Reliability: 12th July 24 hr test
Call Call
drops
on the
first modem
drop
events
vs. Time
Total :
16 Call drops
Time
Modem up times during second 24 hour test
4
Uptime %
3
2
1
0
80
92
95
96
Time
7
386
24
Results – TCP performance of a single link

TCP Version – TCP SACK

Throughput of a segment is defined as the size
Throughput vs. Time
-------- avg. of last 10 segments
-------- avg. of all the segments up
to that point
of the segment seen divided by the time since
the last segment was seen (in this direction).

RTT is defined as the time difference between
the instance a TCP segment is transmitted (by
the TCP layer) and the instance an
acknowledgement of that segment is received.

The average throughput of the connection is
RTT vs. Time
2.45Kbps.

The average RTT was found to be 20 seconds

Bandwidth Delay Product (BDP) = 2400*20/8 =
6000 bytes = 4 segments.

Due to low throughput, the BDP is small.
University of Kansas
25
Results – TCP performance of a 4-channel system

The average throughput obtained - 9.4 Kbps

The average RTT observed - 16.6 seconds

Factors affecting throughput and RTT

TCP Slow start

MLPPP fragmentation

Random delay variation

TCP cumulative acknowledgments
Time Sequence Graph
Throughput vs. Time
-------- avg. of last 10 segments
-------- avg. of all the segments up
to that point
RTT vs. Time
------- Received Acknowledgements
------- TCP segments transmitted
------- Received window advertisement
University of Kansas
26
Results – TCP performance of a 4-channel system
Time Sequence Graph

Outstanding Unacknowledged
data and Congestion window
Time Sequence Graph
------- Received Acknowledgements
------- TCP segments transmitted
------- Received window advertisement
Outstanding Unacknowledged Data
------- Instantaneous outstanding data samples
------- Average outstanding data
------- Weighted average of outstanding data
University of Kansas
27
Results – TCP performance degradation due to packet loss

Low packet loss, long time experiments needed to determine the performance degradation

Two packet losses were observed in the FTP video upload resulting in packet
retransmissions

Packet losses usually occur during call hand-offs

Retransmission time outs (RTO) is very large due to high RTT and high mean deviation
Time Sequence Graph
Time Sequence Graph
Retransmitted packet
Retransmitted packet
University of Kansas
28
Results – TCP performance degradation due to packet loss

Effect on the TCP performance due to packet
RTT vs. Time
loss

Decrease in the throughput following the packet
loss

RTT peaks around the packet loss

The average throughput of the connection is
observed to be 7.44 Kbps.
Outstanding Unacknowledged Data
Throughput vs. Time
University of Kansas
29
Results – TCP performance degradation due to call drop

Packet loss due to a call drop on one links of
the multilink bundle

A finite amount of time for the data link layer
realize the link has failed

Large RTO timer

The entire window of packets (12 in this case)
and acknowledgements that are in flight on
that particular link are dropped.

Throughput of the connection – 7.6 Kbps.
Time Sequence Graph
------- Received Acknowledgements
------- TCP segments transmitted
------- Received window advertisement
Outstanding Unacknowledged Data
Time Sequence Graph
------- Instantaneous outstanding data samples
------- Average outstanding data
------- Weighted average of outstanding data
University of Kansas
30
Results – Mobile tests
Test Location
University of Kansas
31
Results – Mobile Performance
START
Throughput vs. Time
STOP
Avg. Throughput = 9 Kbps
University of Kansas
32
Results – Mobile Performance
(cont.d)
START
Throughput vs. Time
STOP
Avg. Throughput = 9.34 Kbps
University of Kansas
33
Applications – Uploads and Downloads
The following files were downloaded from WEB and ITTC network. The size of these files,
their importance (on a scale of 1-10, based on user survey) are shown
Title
Downloaded/uploaded
Size
Imp
1
Spectrum Analyzer programmers
Manual
Download from Agilent.com
7.2MB
9
2
Matlab Programs
Download from ITTC
500KB
7
3
Voltage regulator data sheet
Download from Fairchild.com
226KB
9
4
GPS software
Download
800KB
9
5
Proposal submission
Upload
600KB
8
6
Access point manager software
Download from Orinoco.com
4.66MB
7
7
Drawing of machine spares to
order
Upload to University of Copenhagen
1MB
9
8
Video of core, datasheet
Upload for press release
2MB
8
9
Pictures, press release of longest
core in Greenland
Upload to Kangerlussauq for press
release
500KB
6
University of Kansas
34
Applications – Internet at camp
University of Kansas
35
Applications –WI-FI setup integrated with Iridium
University of Kansas
36
Applications - Wireless Internet

Wireless Internet access was used
by many NGRIP researchers

Email access put the researchers in
touch with their home institutions

Scientist at NGRIP were able to
send information back to their home
institutions

The system was also useful for general camp purpose: sending drawings to order
spares for a broken caterpillar, excel spreadsheet for food order, general press
releases, downloading weather reports for planning C-130 landings
University of Kansas
37
Applications – Outreach

Daily journal logs were uploaded to the University
of Kansas and posted on www.ku-prism.org

Two way video conference was held twice to test
the real time audio/video

The system not only helped PRISM group to
download critical software, but also made it
Testing Video conference between the camp and the University
possible to obtain faculty guidance and expertise
on the field

It also helped the researchers back at the
University of Kansas to virtually view the field
experiments since the video clips of experiments
being carried out were also uploaded to the
University and posted on project website
Uploading daily field logs
University of Kansas
38
Lessons Learned

Multi-link PPP is relatively efficient over Iridium system. With 4-modems efficiency was observed to
be >90%

The RTT the system is significant ~ 1.8 seconds, which is an impairment to real time interactions

There is a large (random) variation in delay of the system

The average up-time of the overall connection is good > 95%

Mobile tests showed performance very similar to that of stationary system up to speeds of 20mph

A call drop on the first modem, results in a complete loss of connection, a potential bug in the PPP
networking code; could be fixed in newer version of the PPP software

Strong interference from a nearby source on the same frequency (1.6GHz) could cause little
disturbance in the system leading to either connection failures or call drops. This was observed when
a radar sweeping between 500MHz-2GHz with an output of 20dBm was placed at distance less than
15 meters
University of Kansas
39
Conclusions

In order to provide data and Internet access to Polar Regions, we have developed a
reliable, easily scalable, lightweight, and readily available multi-channel data
communication system based on Iridium satellites that provide round the clock, pole-to-
pole coverage.

A link management software is developed that ensures fully autonomous and reliable
operation

An end-to-end network architecture providing Internet access to science expeditions in
Polar Regions is demonstrated.

This system provided for the first time, Internet access to NGRIP camp at Greenland and
obtained the call drop pattern.

The TCP performance and the reliability of the system is characterized
University of Kansas
40
Future Work

Modify the MLPPP code so that the interface is attached to the bundle and not to the
primary link

Additional research to determine the cause of delay and develop methods to
overcome it.


Evaluate the new data-after-voice (DAV) service from Iridium
Evaluate different versions of TCP to determine the enhancements that can handle
the random variation and value of RTT


Improve the user friendliness of the system

GUI for connection setup

Self-test / diagnostic tools for troubleshooting
Research into the spacing and sharing of antennas to reduce the antenna footprint
University of Kansas
41