SocialSync: Synchronization for a Smartphone Camera Network

Richard Latimer Rice University September 7, 2014

2 Intro 2

Why Smartphones?

• • • 1 billion smartphones sold in 2014 Growth in computer vision applications Advanced development platform 3 Best camera is the one with you!

What if Smartphones Could Collaborate?

4 Refocusing Lytro Kilner et al.

Free Viewpoint Television 4 3D Philips Depth Pelican

Prior Work: Camera Arrays 5 High Performance Imaging Using Large Camera Arrays [Wilburn et al.] High-Speed Videography Using a Dense Camera Array [Wilburn et al.] Traditional camera arrays are assumed to be embedded and tightly synchronized.

6 A Smartphone Camera Array Involves the OS Stack Application Application Framework Library Runtime Linux Kernel/Drivers Hardware 6

Social Capture: Problem Statement 7 If you use multiple smartphone cameras to capture video simultaneously, there is an inherit synchronization challenge that manifests (due to OS stack) when the scene is moving.

7

Scene Setup: How Bad is Misalignment?

8 0 1 0 0 1 0 0 0 0 0 Smartphone Camera Array 8 Resolution Target Scene Setup

Outcome: How Bad is Misalignment?

9 Even when photos are captured simultaneously, misalignment of frames can cause problems when target is moving.

Moving Resolution Target Captured by 4 Phones Simultaneously 9

Source of Problem 10 Camera 1: Frame 0 Frame 1 Frame 2 Camera 2: Frame 0 Frame 1 Frame 2 Frame 3 Camera 3: Frame 0 Frame 1 Frame 2 10 time

Goal of SocialSync Objective: Align frame sequences from smartphone cameras within tolerance ranges less than t milliseconds, where t < frame duration.

11 Camera 1: Frame 0 Frame 1 Frame 2 Camera 2: Frame 0 Frame 1 Frame 2 Frame 3 Camera 3: Frame 0 Frame 1 Frame 2 11 time

1. Characterization 2. SocialSync Protocol 3. Evaluation

12 12

1. Characterization

• Measurement Test Apparatus • Results

2. SocialSync Protocol 3. Evaluation

13 13

14 What We Use: Android 4.4

14

15 0 1 0 0 1 0 0 1 0 Network Setup 0 1 0 Hub and Spoke Network • Server provides global reference clock • Server broadcasts image capture request 15 0 1 0

16 System for Measuring Capture Timestamps 16

1. Characterization

• Measurement Test Apparatus • Results

2. SocialSync Protocol 3. Evaluation

17 17

Random Variables Associated with Capturing a Frame Setup Time: Delay to setup camera before capture of first frame 18 Delivery Time: After capture, there is delay to when frame reaches application and a timestamp can be recorded.

18

Camera Setup 19 19 Setup Frame 0 Frame 1 Frame 2 time

Camera 1: Is Setup Time Consistent?

20 Setup Setup Setup time 20

Camera 1: Is Setup Time Consistent?

21 Setup Setup Setup Frame 0 Frame 0 Frame 0 time 21

22 Variability in Camera Setup Makes Aligning Frames Challenging Each time camera started, the start of the image sequence will be delayed by a variable amount of time.

• Setup can vary at least 35 ms • StDev = 9.4 ms • Not deterministic 22

Frame Delay 23 Setup Frame 0 Frame 1 Frame N time 23

Measurement of Frame Delay 24

Delivery Time of Frame to Application 25 If we keep measuring capture times of frames, they are spaced evenly Delivery time of frame to app is a proxy for estimating capture time with delay: • μ = 67 ms • σ = 4.5 ms 25

1. Characterization 2. SocialSync Protocol

• Estimation • Alignment

3. Evaluation

26 26

27 Goal of Social Sync Camera 1: Camera 2: Camera 3: Setup Frame 0 Frame 1 Frame 2 Setup Frame 0 Frame 1 Frame 2 Setup Frame 0 Frame 1 Frame 2 time 27

Social Sync Protocol Estimation Step: Estimating Capture Timestamps to Measure Misalignment of Frames 28 Alignment Step: Using Repeated Attempts at Launching Image Sequence to Reduce Sub-Frame Misalignment 28

If Misalignment is Due to Camera Setup 29 Camera 1: Camera 2: Camera 3: Setup Frame 0 Frame 1 Frame 2 Setup Frame 0 Frame 1 Frame 2 Setup Frame 0 Frame 1 Frame 2 time 29

Could We Estimate Capture Time By Calibrating for Setup Time?

30 Camera 1: Camera 2: Camera 3: Setup Frame 0 Frame 1 Frame 2 Setup Frame 0 Frame 1 Frame 2 Setup Frame 0 Frame 1 Frame 2 time 30

31 Recall Wide Variability in Setup If setup time had little variance, could align capture timestamps by offsetting • Setup can vary at least 35 ms, when worst case frame misalignment was 33 ms (inverse of FPS) • Can’t calibrate for setup time to reduce misalignment!

31

Does Frame Delay Provide a Better Estimate of the Capture Timestamp?

32 32 Setup Frame 0 Frame 1 Frame N time

Measurement of Frame Delay 33

How does estimate of mean improve with number of samples?

Stats 101: Standard error of a sample mean is Based on Central Limit Theorem, 95% confidence sample mean is within range: 34

35 Mean Frame Delay Recap 35 Setup Frame 0 time

36 Mean Frame Delay Recap Mean Delay Est.: 36 Setup Frame 0 time

37 Mean Frame Delay Recap Mean Delay Est.: 37 Setup Frame 0 Frame 1 time

Mean Frame Delay Recap Mean Delay Est.: 38 38 Setup Frame 0 Frame 1 Frame N time

Samples Required for Estimate Since mean delay is , we can determine the capture image sequence position for milliseconds using: • 22 samples for 2 ms accuracy • 85 samples for 1 ms accuracy 39 39

Social Sync Protocol Estimation Step: Estimating Capture Timestamps to Measure Misalignment of Frames 40 Alignment Step: Using Repeated Attempts at Launching Image Sequence to Reduce Sub-Frame Misalignment 40

41 Synchronization Tolerance Goal: Want synchronization to a reference image sequence to be aligned within some tolerance 41 time

42 Synchronization Attempt 1 42 Setup Frame 0 time

Synchronization Attempt 1 43 43 Setup Frame 0 Frame N time

44 Synchronization Attempt k 44 Setup Frame 0 time

Synchronization Attempt k 45 45 Setup Frame 0 Frame N time

Obtaining Synchronization • Let T be the worst case misalignment between frames • Let t be the range specified by the sync tolerance, where t ≤T Let p = t/T, be the probability of synchronization error being reduced for a single phone 46 46

Single Camera Sync Probability After k Attempts P k = 1-(1-p) k Expected Number of Sync Cameras n*P k 47 47

Expected Attempts to Sync N Cameras 48 48

•

Characterization

•

SocialSync Protocol

•

Evaluation

49 49

Evaluating Synchronization 50 Smartphones in array to reduce artifacts not related to time synchronization error 4 Cameras 8 Cameras NaïveSync 23 ms 35 ms SocialSync 5 ms 6 ms 50

8 Cameras Naïve Sync SocialSync Reference Views Depth Map 51 51

4 Cameras 52 Reference NaïveSync SocialSync 52

4 Cameras 53 Reference NaïveSync SocialSync 53

Conclusion 1.

2.

3.

Characterize Android Camera Developed SocialSync Protocol Demonstrated SocialSync Future Work: - Greater control of Android camera system - In wild implementation 54 54

Acknowledgements 1.

2.

3.

4.

Ashok, Lin, Ashu and their students Co-Author Jason Holloway LF4CV Reviewers, Paper Accepted National Science Foundation 55 55