Transcript Hey Cortana, set a reminder to …. Hey Cortana, send email to Jonas Brandell …. Hey Cortana, call Jonas Brandell Hey Cortana, text Jonas Brandell …

Hey Cortana, set a
reminder to ….
Hey Cortana, send
email to Jonas
Brandell ….
Hey Cortana, call
Jonas Brandell
Hey Cortana, text
Jonas Brandell …
Hey Cortana, what’s
up with Jonas Brandell
on Facebook
Hey Cortana, create a
station for Hozier on
Pandora
Hey Cortana, send a
message to Jonas
Brandell on Viber
Hey Cortana, Tweet
flying into San
Francisco tonight
Application
Call
Text
Remind
Email
Voice Command
Voice Command
Definition
Microsoft Speech
Recognition Service
X
• Viber
• Facebook
• Adventure Works
• By default uses the app’s display name
• Provide a alternate such as short form when the app
name is long or hard to pronounce
• Explicitly specify the app name where feels most natural
Viber
Facebook
Adventure Works
• Increase the recognition accuracy by providing a
phonetically-rich phrase
Application
Call
Text
X
default.html
<meta name="msapplication-cortanavcd"
content="https://contoso.com/vcd.xml"/>
if (typeof Windows != 'undefined')
{
Remind
<script src="voice.js"></script>
var activation = Windows.ApplicationModel.Activation;
<Application
Email
Id="App" Windows.UI.WebUI.WebUIApplication.addEventListener("activated", function (args)
{
StartPage="https://flightarcade.azurewebsites.net">
Voice Command
Package.appxmanifest
Contoso.com
if (args.kind === activation.ActivationKind.voiceCommand)
/default.html
{
<uap:VisualElements
var speechRecognitionResult = args.result;
...
var properties = speechRecognitionResult.SemanticInterpretation.Properties;
</uap:VisualElements>
//...
Vcd.xml
}
<uap:ApplicationContentUriRules>
});Type="include" WindowsRuntimeAccess="all“ Match="https://flightarcade.azurewebsites.net"/>
<uap:Rule
</uap:ApplicationContentUriRules>
</Application>
Microsoft Speech
Recognition Service
Your App
A
Here are your upcoming
trips
VCD
Web Service
San Francisco
May 29th
Yosemite
August 15th
Go to Adventure Works
Microsoft Speech
Recognition Service
<VoiceCommands
xmlns="http://schemas.microsoft.com/voicecommands/1.2">
<CommandSet xml:lang="en-us" Name=“en-us-CommandSet">
<AppName> Adventure Works </AppName>
<Example> Show my trip to San Francisco </Example>
<Command Name=“showTripToDestination">
<Example>Show my trip to San Francisco</Example>
<ListenFor >show [my] trip to {destination}</ListenFor>
<Feedback>Looking for trip to {destination}</Feedback>
<VoiceCommandService Target=“AdventureWorksService"/>
</Command>
public async void Run(IBackgroundTaskInstance taskInstance)
<Extensions>
<Extension
Category="windows.appService"
EntryPoint="AdventureWorks.AdventureWorksService">
<AppService Name=“AdventureWorksService"/>
</Extension>
</Extensions>
See Session 765 For Details on AppServices
using Windows.ApplicationModel.AppService;
using Windows.ApplicationModel.Background;
using Windows.ApplicationModel.VoiceCommands;
•
•
•
•
•
•
•
VoiceCommandDefinitionManager
VoiceCommandDefinition
VoiceCommandServiceConnection
VoiceCommand
VoiceCommandUserMessage
VoiceCommandContentTile
VoiceCommandResponse
• RequestConfirmationAsync(…)
• RequestDisambiguationAsync(…)
• RequestAppLaunchAsync(…)
• ReportProgressAsync(…)
• ReportFailureAsync(…)
• ReportSuccessAsync(…)
• VoiceCommandCompleted
Which one do
you wanna
cancel?
App Screen
A
Which trip would you like to
cancel?
Las Vegas
May 14th 2015
San Diego
May 28th 2015
Text-To-Speech
(TTS)
App Icon
App GUI Text
Content Tile
Content Area
Phoenix
August 24th 2015
Go to Adventure Works
Link to your
app
Which one do
you wanna
cancel?
A
Which trip would you like to
cancel?
Las Vegas
May 14th 2015
San Diego
May 28th 2015
Phoenix
August 24th 2015
Go to Adventure Works
Which one do
you wanna
cancel?
A
Which trip would you like to
cancel?
Las Vegas
May 14th 2015
VoiceCommandResponse
{
VoiceCommandUserMessage
Message;
San Diego
May 28th 2015
VoiceCommandUserMessage
Phoenix
RepeatMessage;
August 24th 2015
List<VoiceCommandContentTile> VoiceCommandContentTiles;
Go to Adventure Works
string
}
AppLaunchArgument;
A
Which one do you want to
add to your trip?
Title Name
Text Line One
Text Line Two
Text Line Three
VoiceCommandContentTile
{
string
Title;
string
TextLine1;
string
TextLine2;
string
TextLine3;
Object
AppContext;
string
AppLaunchArgument;
Windows.Storage.StorageFile Image;
VoiceCommandContentTileType ContentTileType;
}
Icon, Title, Line1, Line2, Line3
Icon, Title
A
A
Which one do you want to
add to your trip?
Which one do you want to
add to your trip?
Title Name One
Title Name One
Long text lorem ipsum dolor
sit amet, Consec tetuer amias
Sed do eiusmod
Title Name Two
Title Name Three
Title Name Four
Go to Adventure Works
Title Name Two
Long text lorem ipsum dolor
sit amet, Consec tetuer amias
Sed do eiusmod
Title Name Three
Long text lorem ipsum dolor
sit amet, Consec tetuer amias
Sed do eiusmod
Title Name Four
Listening…
Listening…
Listening…
68x68 Icon, Title, Line1, Line2
68x92xIcon, Title, Line1,
Line2, Line3
A
A
A
Which one do you want to
add to your trip?
240x140 Icon, Title, Line1,
Line2, Line3
Which one do you want to
add to your trip?
Title Name One
Title Name One
Long text lorem ipsum
Long text two lines lorem ipsum
consec tetuer amias
Title Name Two
Which one do you want to
add to your trip?
Title Name Two
Long text lorem ipsum
Title Name Three
Long text two lines lorem ipsum
consec tetuer amias
Long text lorem ipsum
Title Name Three
Title Name Four
Title Name One
One line text
Long text two lines lorem ipsum consec tetuer
amias odoran
Long text two lines lorem ipsum
consec tetuer amias
Long text lorem ipsum
Title Name Four
Title Name Five
Long text two lines lorem ipsum
consec tetuer amias
Long text lorem ipsum
Go to Adventure Works
Title Name Two
One line text
Listening…
Listening…
Listening…
Which ‘Vegas’ trip do
you wanna cancel?
Vegas Tech Conference
or Party in Vegas.
Do you wanna cancel
‘Vegas Tech
Conference’?
A
Which one do you want to
cancel?
Cancelling this trip…
A
A
Cancel this trip?
Vegas Tech Conference
Cancelling this trip…
I’ve cancelled it.
[success-earcon]
A
Cancelled this trip.
Vegas Tech Conference
Vegas Tech Conference
Vegas Tech Conference
May 14th 2015
May 14th 2015
May 14th 2015
May 14th 2015
Party in Phoenix
Yes
Go to Adventure Works
No
Cancel
August 18th 2015
Go to Adventure Works
Cancel my Adventure
Works trip
Listening…
Listening…
Vegas Tech
Conference
Yes
Which trip do you wanna cancel?
A
Which one do you want to
cancel?
Vegas Tech Conference
May 14th
Vegas Tech
Conference
Party in Phoenix
August 18th
The first one
The one on
August 18th
Listening…
Do you wanna cancel ‘Vegas Tech
Conference’?
A
Cancel this trip?
Vegas Tech Conference
May 14th 2015
Yes
Yes
No
No
Listening…
Cancelling this trip…
A
Cancelling this trip…
Vegas Tech Conference
May 15th 2015
Cancel
Go to Adventure Works
Looking for your trip to
San Francisco…
A
I’ll get Adventure Works on
that.
A
Working on that…
Vegas Tech Conference
May 14th 2015
Adventure Works
Looking for trip to San Francisco
Cancel
Cancel
Go to Adventure Works
Search “Cancel my Adventure Works
trip to Vegas”
Listening…
Adventure Works,
cancel my trip!
Invoke IBackgroundTask::Run()
RequestDisambiguationAsync
Las Vegas
Las Vegas
Cancun
//build/
VoiceCommandDisambiguationResult
RequestConfirmationAsync
yes no
yes
VoiceCommandConfirmationResult
Adventure Works,
cancel my trip!
Event: VoiceCommandCompleted
Invoke IBackgroundTask::Run()
RequestDisambiguationAsync
Las Vegas
Las Vegas
Cancun
//build/
VoiceCommandDisambiguationResult
Event: VoiceCommandCompleted
RequestConfirmationAsync
yes no
yes
UI Interaction
VoiceCommandConfirmationResult
App Launch
ReportProgressAsync
Foreground
Background
Adventure Works Sample Code
Design Guidelines for extending Cortana