Transcript pptx

Programming By
Voice
Andrew Begel
Advisor: Prof. Susan L. Graham
University of California, Berkeley
VL/HCC 2004
September 28, 2004
Programming by Voice
for int i equals zero i less than ten i plus plus
September 28, 2004
VL/HCC 2004 Graduate Student Consortium
1
Programming by Voice
for int i equals zero i less than ten i plus plus
for (int i = 0; i < 10; i++ ) {
▌
}
September 28, 2004
VL/HCC 2004 Graduate Student Consortium
1
Ambiguities
Spelling of ID?
KW or #?
KW or ID?
4 int eye equals 0 aye less then ten i plus plus
for (int i = 0; i < 10; i++ ) {
▌
}
September 28, 2004
VL/HCC 2004 Graduate Student Consortium
2
Sometimes it’s hard!
for times ate equals file two load equals one
September 28, 2004
VL/HCC 2004 Graduate Student Consortium
3
Many Valid Interpretations!
for times ate equals file two load equals one
for (times; ate == file; to().load = 1) {
▌
}
September 28, 2004
VL/HCC 2004 Graduate Student Consortium
3
Many Valid Interpretations!
for times ate equals file two load equals one
for (times; ate == file; to().load = 1) {
▌
}
4 * 8 = file; toload = won ▌
September 28, 2004
VL/HCC 2004 Graduate Student Consortium
3
Many Valid Interpretations!
for times ate equals file two load equals one
for (times; ate == file; to().load = 1) {
▌
}
4 * 8 = file; toload = won ▌
fore.times(8).equalsFile(2, load == 1) ▌
September 28, 2004
VL/HCC 2004 Graduate Student Consortium
3
Disambiguating “filetoload”
file to load
September 28, 2004
file 2 load
file toload
VL/HCC 2004 Graduate Student Consortium
filetoload
4
Disambiguating “filetoload”
file to load
file 2 load
file toload
filetoload
1. file to load
9. file 2 load
12. file toload
15. filetoload()
2. file(to, load)
10. file(2, load)
13. file(toload)
16. filetoload
3. file(to.load)
11. (file, 2, load) 14. file.toload
4. file(to(load))
5. file.to(load)
6. file.to load
7. file to.load
8. file.to.load
September 28, 2004
VL/HCC 2004 Graduate Student Consortium
4
Disambiguating “filetoload”
file to load
file 2 load
file toload
filetoload
1. file to load
9. file 2 load
12. file toload
15. filetoload()
2. file(to, load)
10. file(2, load)
13. file(toload)
16. filetoload
3. file(to.load)
11. (file, 2, load) 14. file.toload
4. file(to(load))
1. No immediately adjacent identifiers
5. file.to(load)
6. file.to load
7. file to.load
8. file.to.load
September 28, 2004
VL/HCC 2004 Graduate Student Consortium
4
Disambiguating “filetoload”
file to load
file 2 load
file toload
filetoload
15. filetoload()
2. file(to, load)
10. file(2, load)
13. file(toload)
16. filetoload
3. file(to.load)
11. (file, 2, load) 14. file.toload
4. file(to(load))
1. No immediately adjacent identifiers
5. file.to(load)
2. No variables named “file”
8. file.to.load
September 28, 2004
VL/HCC 2004 Graduate Student Consortium
4
Disambiguating “filetoload”
file to load
file 2 load
file toload
filetoload
15. filetoload()
2. file(to, load)
10. file(2, load)
13. file(toload)
16. filetoload
3. file(to.load)
4. file(to(load))
1. No immediately adjacent identifiers
2. No variables named “file”
3. No methods named “file”
September 28, 2004
VL/HCC 2004 Graduate Student Consortium
4
Disambiguating “filetoload”
file to load
file 2 load
file toload
filetoload
15. filetoload()
16. filetoload
1. No immediately adjacent identifiers
2. No variables named “file”
3. No methods named “file”
4. No method named “filetoload”
September 28, 2004
VL/HCC 2004 Graduate Student Consortium
4
Disambiguating “filetoload”
file to load
file 2 load
file toload
filetoload
 filetoload
1. No immediately adjacent identifiers
2. No variables named “file”
3. No methods named “file”
4. No method named “filetoload”
September 28, 2004
VL/HCC 2004 Graduate Student Consortium
4
The Vision
• Naturally Verbalized Programs
– Spoken Java Language
• Navigation and Editing Command Language
• Analyses that Resolve Ambiguities
• Prototype: SPED: SPeech EDitor
September 28, 2004
VL/HCC 2004 Graduate Student Consortium
5
Implementation
• Speech Recognition: IBM ViaVoice
• Eclipse IDE
• Harmonia program analysis toolkit
– Generalized LR parsing with input stream
ambiguities
– Persistent, incremental semantics
September 28, 2004
VL/HCC 2004 Graduate Student Consortium
6
Evaluation
•
Hypothesis
– Programmers can learn to use SPED efficiently for
many programming tasks
•
User Study
1. Train users on Spoken Java and command language
2. Edit an existing program
3. Create some new code
•
Metrics:
– Speed, vocabulary mistakes, grammatical mistakes,
system understanding errors, subjective impressions
September 28, 2004
VL/HCC 2004 Graduate Student Consortium
7
Contributions
• Enabling programming by voice
– New methods for handling input ambiguities
– Exploiting syntax and semantics of programming
domain
– Analyses for mixed command and programming
languages
– Interface to commercial speech recognition tools
Andrew Begel: [email protected]
September 28, 2004
VL/HCC 2004 Graduate Student Consortium
8