Transcript ABRF iPRG 2012 study - Association of Biomolecular

A B
R F
Proteome Informatics
Research Group
iPRG 2012:
A Study on Detecting Modified Peptides in
a Complex Mixture
ABRF 2012, Orlando, FL
3/17-20/2012
A B
R F
Proteome Informatics
Research Group
IPRG2012 STUDY:
DESIGN
2
A B
R F
Study Goals
Proteome Informatics
Research Group
• Primary:
Evaluate the ability of participants to
identify modified peptides present in a
complex mixture
• Secondary: Find out why result sets might differ
between participants
• Tertiary:
Produce a benchmark dataset, along with
an analysis resource
3
A B
R F
Study Design
Proteome Informatics
Research Group
• Use a common, rich dataset
• Use a common sequence database
• Allow participants to use the bioinformatic tools and
methods of their choosing
• Use a common reporting template
• Report results at an estimated 1% FDR (at the
spectrum level)
• Ignore protein inference
4
A B
R F
Sample
Proteome Informatics
Research Group
• Tryptic digest of yeast (RM8323 – NIST), spiked with
69 synthetic modified peptides (tryptic peptides from
6 different proteins – sPRG)
– Phospho (STY)
– Sulfo (Y)
– Mono-, di-, trimethyl (K)
– Mono-, dimethyl (R)
– Acetyl (K)
– Nitro (Y)
5
A B
R F
Supplied Study Materials
Proteome Informatics
Research Group
• 5600 TripleTOF dataset (i.e. WIFF file)
– WIFF, mzML, dta, MGF (de-isotoped);–
conversions by MS Data Converter 1.1.0
– MGF (not de-isotoped – conversion by Mascot
Distiller 2.4)
• 1 fasta file (UniProtKB/SwissProt S. cerevisiae,
human, + 1 bovine protein + trypsin from Dec.
2011)
• 1 template (Excel)
• 1 on-line survey (Survey Monkey)
6
A B
R F
Instructions to Participants
Proteome Informatics
Research Group
1. Retrieve and analyze the data file in the format of your
choosing, with the method(s) of your choosing
2. Report the peptide to spectrum matches in the provided
template
3. Report measures of reliability for PTM site assignments
(optional)
4. Fill out the survey
5. Attach a 1-2 page description of the methodology
employed
7
A B
R F
Proteome Informatics
Research Group
iPRG 2012 STUDY:
PARTICIPATION
8
A B
R F
Soliciting Participants and Logistics
Proteome Informatics
Research Group
Study advertised on the ABRF website and listserv and by direct invitation from iPRG members
1. Email participation
request to
‘[email protected]’
Participant
2. Send official study letter
with instructions
iPRG members
Questions / Answers
3. All further communication (e.g.,
questions, submission) through
‘[email protected]’
“Anonymizer”
9
A B
R F
Participants (i) – overall numbers
Proteome Informatics
Research Group
• 24 submissions
– One participant submitted two result sets
• 9 initialed iPRG member submissions (with
appended ‘i’)
• 2 vendor submissions (identifiable by
appended ‘v’)
10
A B
R F
About the Participant
Proteome Informatics
Research Group
ABRF Member
I routinely
analyze these
sorts of data
Nonmember
I have worked
with several data
sets
I have worked
with a few data
sets
Complete Novice
Bioinformatician/S
oftware Developer
Mass
Spectrometrist
Lab Scientist
Director/Manager
Post-Grad
11
A B
R F
About the Participant’s Lab
Proteome Informatics
Research Group
Academic
Manufacturer/Ve
ndor
Biotech/Pharma/I
ndustry
Government
North America
Europe
Other
Asia
Australia/NZ
Core Only
Africa
Software
development only
(no research
facility)
Conduct both core
functions and noncore lab research
12
A B
R F
Participation in sPRG Study
Proteome Informatics
Research Group
YES
NO
•Only one participant indicated he used sPRG information to
aid his analysis.
•This person was one of the least successful in identifying
the spiked-in peptides!
13
A B
R F
Search Engine Used
Proteome Informatics
Research Group
10
9
8
7
6
5
4
3
2
1
0
14
A B
R F
Site Localization Software
Proteome Informatics
Research Group
8
7
6
5
4
3
2
1
0
•4 participants did not list using software for site localization.
15
A B
R F
Summary of Submitted Results
Proteome Informatics
Research Group
7000
# spectra Id Yes
# unique Peptides UC ID Yes
6000
5000
4000
3000
Only reported
modified peptides
2000
1000
11821
45511
14152
47603
52781
14151
74564
23117
34284i
92653
87048i
40104i
23068
77777i
42424i
97053i
94158i
87133i
58409
11211
93128i
33564
58288v
71755v
0
16
A B
R F
Summary of IDs and Localizations
Proteome Informatics
Research Group
7000
# No Mods
# Common Mods (^q,^c,m,n,q)
# Nterm Mods
# AA Mutation Mods
# Interesting Mods
6000
5000
# Spectra
Peptide Identification
in all Spectra
4000
3000
2000
11821
45511
14152
47603
52781
14151
74564
23117
# Interesting Mod Loc Certainty N
# Interesting Mod Loc Certainty Y
Site Localization in Spectra
With Interesting Modifications
500
400
300
200
100
11821
45511
14152
47603
52781
14151
74564
23117
34284i
92653
87048i
40104i
23068
77777i
42424i
97053i
94158i
87133i
58409
11211
93128i
33564
58288v
0
71755v
# Spectra
34284i
92653
87048i
40104i
23068
77777i
42424i
97053i
94158i
87133i
58409
11211
93128i
33564
600
58288v
0
700
71755v
1000
17
A B
R F
Overlap of spectrum identifications
Proteome Informatics
Research Group
12000
10000
Cummulative # Spectra
8000
7840 agreed on by 3 or more participants
6000
4000
2000
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# Participants Agreeing
16
17
18
19
20
21
22
23
24
18
A B
R F
Room for improvement in thresholding?
Proteome Informatics
Research Group
7500
7000
6500
6000
5500
5000
4500
4000
3500
3000
2500
2000
1500
1000
500
0
11821
45511
14152
47603
52781
14151
74564
23117
34284i
92653
87048i
40104i
23068
77777i
42424i
97053i
94158i
87133i
58409
11211
93128i
33564
58288v
71755v
# Spectra
#ND No Id, Diff from Consensus
#Y<3 P Id Yes
#YD Yes Id, Diff from Consensus
#NS No Id, Same as Consensus
#YS Yes Id, Same as Consensus
71755v 58288v 33564 93128i 11211 58409 87133i 94158i 97053i 42424i 77777i 23068 40104i 87048i 92653 34284i 23117 74564 14151 52781 47603 14152 45511 11821
Peaklist
mgf
Spectral Pre-Processing
Peptide Identification
By
Discovery of Unexpected
Mods
By
Site Localization
mzML
mzML
Pk
PkDB
Pk
PkDB
Pk
mgf
mgf
mzML
mzML
mgf_nd
mgf
PPi
Pk
PkDB
M
PPi
M
O
ST
XT
Ot
Pk
PkDB
Pk
PkDB
Pk
M
PPi
Pk
MDe
PPi
PPi
Results Filtering
By
Pk
PkDB
Pk
XL
IH
Ot
NTT
2
5-10
years
1
5-10
years
1
5-10
years
2
>10
years
Experience
mgf
mgf
mgf_nd
mzML
mzML
Ot
P/PP
TPP
XT
mgf
SM
pF
SM
pF
M
SM
pF
M
SM
pF
M
ST
PPr
pF
IH
MO
M
O
ST
P/PP
TPP
PPr
pF
IH
M
O
P/PP
TPP
PPr
XL
pF
M
MM
IH
IP
IH
XL
IH
XL
pF
M
1
5-10
years
1
>10
years
1
5-10
years
2
5-10
years
1
5-10
years
?
1
5-10
years
2
>10
years
2
3-4 years
1-2 years
2
>10
years
?
M
O
XT
mgf
pF
1
MG
mgf_nd
PPr
< 1 year 3-4 years
O
ST
XT
mgf_nd
mzML
WIFF
MDi
PW
M
MM
IH
mzML
Pk
PkDB
Pk
PkDB
Pk
PkDB
Pk
PkDB
mgf
M
PPi
mgf
mgf
M
O
PPi
M
Pk
PkDB
M
2
?
>10
years
1-2 years
mgf_nd
M
O
AS
Sc
1
>10
years
mgf_nd
2
1-2 years
mgf
mgf_nd
MDi
PD
Sq
PD
Sq
M
M
IH
An
MDe
Ot
PD
Sq
XL
Sc
M
PR
2
>10
years
2
>10
years
?
1-2 years
2
>10
years
An
Andromeda/MaxQuant
MG
MS-GFDB
pF
pFind
Sc
Scaffold
AS
A-Score
MM
MyriMatch
Pk
PEAKS
SM
Spectrum Mill
By
Byonics
MO
MODa
PkDB
PEAKSDB
Sq
Sequest
IH
In-house software
O
OMSSA
PPi
Protein Pilot
ST
SpectraST
IP
IDPicker
Ot
Other
PPr
Protein Prospector
TPP
TransProteomic Pipeline
M
Mascot
P/PP
Pep/Prot Prophet
PR
PhosphoRS
XL
Excel
MDe
Mascot Delta Score
PD
ProteomeDiscoverer
PW
ProteoWizard
XT
X!Tandem
MDi
Mascot Distiller
19
A B
R F
ESR and FDR
Proteome Informatics
Research Group
Extraordinary Skill Rate or High False Discovery Rate?
ESR + FDR = 100* (Y<3P+YD)/total ids Y
24 participants
3 for consensus
22
20
Y<3 P percent
18
YD percent
16
14
10
8
6
4
2
11821
45511
14152
47603
52781
14151
74564
23117
34284i
92653
87048i
40104i
23068
77777i
42424i
97053i
94158i
87133i
58409
11211
93128i
33564
58288v
0
71755v
%
12
20
A B
R F
Characteristics of consensus spectra
Proteome Informatics
Research Group
7840 spectra >=3 participants agreeing on sequence
1
10
100
1000
447
Nterm-Acetyl
Nterm-Carbamyl
3
Nterm-Other
3
70
PyroGlu Q
11
PyroGlu E
6
PyroCarbamidomethylCys
94
m: Oxidation
310
n: Deamidation
183
q: Deamidation
6
c
94
d
107
e
5
w: Oxidation
p
3
165
k
r
45
294
s
t
137
y
132
No Variable Mods
10000
6117
Consensus requires agreement on
Sequence, but not modification localization
21
A B
R F
Peak lists
Proteome Informatics
Research Group
• Two types of peak lists were supplied
– Deisotoped and non deisotoped
• Can only tell fragment charge state from nondeisotoped
• Requires search engine to be able to de-isotope
spectrum
22
A B
R F
Peaklists
Proteome Informatics
Research Group
•Number of spectra with undefined precursor charge state
Deisotoped
1031 (304 in consensus results)
Non-deisotoped 6094 (1140 in consensus results)
•For 1013 out of 7840 consensus spectra the precursor m/z differ by
greater than 0.02 Da between deisotoped and non-deisotoped peak list.
•For 238 consensus spectra the peak lists had different specified charge
state
–193 consensus results only possible with deisotoped peak list
–45 consensus results only possible with non-deisotoped peak list
–For 19 consensus results multiple people who searched the nd peak
list agreed on a confident different answer
–For 4 consensus results multiple people who searched the
deisotoped peak list agreed on a confident different answer
23
A B
R F
Mixed Spectra
Proteome Informatics
Research Group
465.19 2+
464.59 3+
465.19 2+
Deisotoped
peaklist
464.59 3+
Non-deisotoped
peaklist
24
A B
R F
Synthetic Peptide ID by Peptide
Proteome Informatics
Research Group
NGDTASPkEYTAGR
LkAQLGPDESK
LkAEGSEIR
FPkAEFAEVSK
EkLLDFIK
NGDTASPkEYTAGR
LkAQLGPDESK
LkAEGSEIR
FPkAEFAEVSK
EkLLDFIK
GTrDYSPR
GILrQITVNDLPVGR
LDELrDEGK
ESTLHLVLrLR
AEGSEIrLAK
NGDTASPkEYTAGR
LkAQLGPDESK
LkAEGSEIR
EkLLDFIK
FPkAEFAEVSK
GTrDYSPR
GILrQITVNDLPVGR
LDELrDEGK
AEGSEIrLAK
ADEGISFrGLFIIDDK
LkLVSELWDAGIK
LkAEGSEIR
GLFIIDDkGILR
FkDLGEENFK
DEGkASSAK
Trimethyl
Methyl (K)
Methyl (R)
Dimethyl (K)
Dimethyl (R)
Acetyl
0
5
10
15
# participants
20
25
TVIDyNGER
TIAQDyGVLK
SVSDyEGK
DISLSDyK
ALAPEyAK
yKPEsDELtAEK
WVtFIsLLFLFssAYSR
VPQVstPtLVEVsR
VPQVstPtLVEVSR
VDAtEEsDLAQQyGVR
tyEtTLEK
tLsDyNIQK
TLSDyNIQK
tLsDYNIQK
tLSDYNIQK
tItLEVEPsDtIENVK
TITLEVEPsDtIENVK
THILLFLPKsVSDYEGK
tHILLFLPKsVsDyEGK
SVsDYEGK
NVAVDELsR
LVQAFQFtDK
LVNEVtEFAK
IFsIVEQR
EstLHLVLR
EsTLHLVLR
EStLHLVLR
DQGGELLsLR
DIsLsDyK
DIsLSDyK
DISLSDyK
DISLsDyK
AEFAEVsK
ADEGIsFR
TLSDyNIQK
TIAQDyGVLK
SVSDyEGK
DISLSDyK
ALAPEyAK
Sulfo
Phospho
Nitro
0
5
10
15
# participants
20
25
25
30
A B
R F
Synthetic Peptide ID by Participant
1
Acetyl (K)
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
Dimethyl (K)
1
Methyl (R)
Nitro (Y)
Phospho
(STY)
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
11821
45511
14152
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
47603
1
1
1
1
52781
14151
74564
23117
34284i
92653
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
Sulfo (Y)
1
1
1
1
1
1
1
1
1
1
1
1
Trimethyl (K)
87048i
40104i
23068
77777i
42424i
97053i
1
1
1
1
1
94158i
1
1
1
1
1
87133i
58409
11211
1
1
1
1
1
Methyl (K)
1
1
1
1
1
1
1
1
1
1
Dimethyl (R)
93128i
33564
58288v
71755v
Proteome Informatics
Research Group
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
26
A B
R F
Correct Localization of Modified Synthetic Peptides
Proteome Informatics
Research Group
70 synthetic modified peptides were spiked into sample.
7 of these were confidently found by no participant
Correct localization & name
of modification reported
70
# Spiked Unique Peptides Correct Mod Loc, Name; Mod Loc Certainty N
# Spiked Unique Peptides Correct Mod Loc, Name; Mod Loc Certainty Y
50
40
30
20
10
11821
45511
14152
47603
52781
14151
74564
23117
34284i
92653
87048i
40104i
23068
77777i
42424i
97053i
94158i
87133i
58409
11211
93128i
33564
58288v
0
71755v
# Unique Peptides Correct Mod Localization
60
27
A B
R F
FLR of Modified Synthetic Peptides
Proteome Informatics
Research Group
500
# Spiked PSMs Mod Loc Certainty N
# Spiked PSMs Mod Loc Certainty Y, Ignored
# Spiked PSMs Wrong Mod Loc; Mod Loc Certainty Y
# Spiked PSMs Correct Mod Loc; Mod Loc Certainty Y
450
# Spiked Peptide PSMs
Ignored PSMs contain mods of
residues other than s,t,y,k,r . Sample
handling mods (n,q,d,e, etc).
FLR = 100% * # PSMs wrong localization of s,t,y,k,r
# PSMs wrong + right localization of s,t,y,k,r
400
350
300
250
200
150
100
50
93128i
11211
58409
87133i
94158i
97053i
42424i
77777i
23068
40104i
87048i
92653
34284i
23117
74564
14151
52781
47603
93128i
11211
58409
87133i
94158i
97053i
42424i
77777i
23068
40104i
87048i
92653
34284i
23117
74564
14151
52781
47603
<1
0.5 10% <30%
11821
33564
33564
5%
45511
58288v
58288v
5%
14152
71755v
71755v
0
16
14
12
10
8
6
4
2
1%
1-2%
1%
0.01 <5%
11821
45511
0
14152
Spiked Peptide PSM FLR (%)
18
<1%
28
Incorrect Localization by Peptide
EKLLDFIK
AEGSEIRLAK
VDATEESDLAQQYGVR
TITLEVEPSDTIENVK
ESTLHLVLR
AEFAEVSK
LKLVSELWDAGIK
DQGGELLSLR
TYETTLEK
NGDTASPKEYTAGR
LKAEGSEIR
TVIDYNGER
ADEGISFR
YKPESDELTAEK
GTRDYSPR
VPQVSTPTLVEVSR
ADEGISFRGLFIIDDK
ALAPEYAK
TIAQDYGVLK
THILLFLPKSVSDYEGK
WVTFISLLFLFSSAYSR
IFSIVEQR
TLSDYNIQK
GILRQITVNDLPVGR
NVAVDELSR
LDELRDEGK
ESTLHLVLRLR
DEGKASSAK
SVSDYEGK
LVQAFQFTDK
LVNEVTEFAK
GLFIIDDKGILR
FPKAEFAEVSK
LKAQLGPDESK
DISLSDYK
FKDLGEENFK
11821
45511
14152
47603
52781
14151
74564
23117
34284i
92653
87048i
40104i
23068
77777i
42424i
97053i
94158i
87133i
58409
11211
93128i
Number of PSM’s with Incorrect Site Localization – Mod Loc Confidence Y
• Present as sulfo-Tyr
• Present as phospho S-10 often mislocalized as S-12 or Y-14
• Present as mono, di, tri methyl K often mislocalized at R
33564
71755v
Proteome Informatics
Research Group
58288v
A B
R F
1
1
1
4
1
4
3
4
4
4
1
4
4
1
4
1
1
3
3
1
1
5
1
3
1
1
6
2
8
3
1
1
1
1
1
2
1
4
1
1
1
1
1
1
1
1
1
1
1
1
1
2
3
1
3
6
29
A B
R F
Phospho vs Sulfo
Proteome Informatics
Research Group
DISLSDY(Phospho)K
Observe modified fragment
ions.
DISLSDY(Sulfo)K
Observe ‘unmodified’
fragment ions.
Spectrum looks essentially
identical to unmodified
peptide spectrum
30
A B
R F
Conclusions
Proteome Informatics
Research Group
• Reasonable number of participants from around the globe,
mainly experienced users but a few first-timers
• Large spread in number of spectra identified
• False negatives (NS) are generally much higher than false
positives, so there is generally room for improvement
• Peak list was a significant factor on performance
• Varied performance in detecting PTMs
• Most participants struggled with sulfation
• Multiply phosphorylated harder to find than singly
• Most common errors in site assignment were:
• Reporting sulfo(Y) as phospho(ST)
• Mis-assignment of site/s in multiply phosphorylated peptides 31
A B
R F
What did the participants think?
Proteome Informatics
Research Group
“The spiked proteins made it possible to game the study - look
for the uncommon modifications only on the spikes. Of course
we didn't do this. Overall I'd say this was a flawed but very
interesting ABRF study.”
22 out of 24 participants found the study useful
“Too many modifications at the same time. Manual validation
is necessary and the right time necessary for this study is too
demanding for this challenge.”
32
A B
R F
Participant’s Confidence in Analyzing PTM Data
Proteome Informatics
Research Group
Before
After
Very Confident
Confident
Very Confident
Not Confident
Confident
No Experience
Not Confident
33
A B
R F
How difficult do you think this study was?
Proteome Informatics
Research Group
Too difficult
Challenging
Just right
What was your total analysis
time for the entire project?
12
10
8
6
4
2
0
0-8
8-16
16-24 24-32 32-40 40-48
Time (hrs)
>48
34
A B
R F
Proteome Informatics
Research Group
Based on this study, would you consider
participating in future ABRF studies?
Yes
No
35
A B
R F
Thank you! Questions?
Proteome Informatics
Research Group
THANK YOU TO ALL
STUDY PARTICIPANTS!
iPRG
Nuno Bandeira
Robert Chalkley(chair)
Matt Chambers
Karl Clauser
John Cottrell
Eric Deutsch
Eugene Kapp
Henry Lam
Hayes McDonald
Tom Neubert (EB liaison)
Ruixiang Sun
Dataset Creation
Chris Colangelo
Anonymizer:
Jeremy Carver, UCSD
36