Transcript TapLogger: Inferring User Inputs On Smartphone Touchscreens

TapLogger: Inferring User Inputs On Smartphone
Touchscreens Using On-board Motion Sensors
Zhi Xu, Kun Bai, and Sencun Zhu
Introduction

Sensors equipped on a smartphone bring potential risks of leaking user’s private information
We observe the correlations between the tapped position on the touchscreen and the motion changes of smartphones;


Attack Workflow

Step 1: The user is tricked to install the TapLogger app;
Step 2: TapLogger learns the motion change patterns of
tap events when the user is interacting with it;


Step 3: TapLogger runs in the background, stealthily
monitor the motion changes, and uses the learnt tap event
pattern to infer user inputs on touchscreens.
Note that, monitoring the readings of motion sensors
requires no security permissions.

Tap Event Detection

TapLogger detects tap events by monitoring the acceleration changes (i.e. SqSum = Ax2 + Ay2 + Az2)

Unique pattern of tap events

Pattern is user specific and device specific

Experimental results of tap event detection
Tap Position Inference

TapLogger infers the position tapped by monitoring the gesture changes (i.e. the readings of Roll and Pitch)

Observed correlations

Use extracted features to distinguish tap events

The training layout and target layout
Proposed Applications With TapLogger

Number Pad Logging Attack during the call

An example of inference

Evaluation with 20 sequences of tap inputs with length of 16

Password Stealing Attack when unlocking the phone
The distribution of inferred labels after entering the passwords
“5 7 6 8” for 32 rounds


Evaluation with different passwords (30 rounds each)
Reference: This poster is based on the paper “TapLogger: Inferring User Inputs On
Smartphone Touchscreens Using On-board Motion Sensors," in Proc. o ACM
Conference on Security and Privacy in Wireless and Mobile Networks (WiSec’12)