Transcript Comparison of Narrowband and Wideband Speech codecs in noisy environnement

Workshop on Wideband Speech Quality in Terminals and
Networks: Assessment and Prediction
8th and 9th June 2004 - Mainz, Germany
Comparison of
Narrowband and Wideband
Speech codecs
in noisy environnement
Noel Chateau [email protected], Laetitia Gros [email protected],
Catherine Quinquis [email protected], Jean Yves Monfort [email protected]
The present document contains information that remains the property of France Telecom. The recipient’s
acceptance of this document implies his or her acknowledgement of the confidential nature of its contents
and his or her obligation not to reproduce, transmit to a third party, disclose or use for commercial purposes
any of its contents whatsoever without France Telecom’s prior written agreement.
confidential
D1 - 23/05/2016
Content
1.
2.
3.
4.
5.
3Gpp/VoIP context
Methodology
Description of Experiment
Results
Conclusion
confidential
France Telecom R&D
Distribution of this document is subject to France Telecom’s authorization
D2 - 23/05/2016
3GPP/VoIP Context
s
s
s
s
Rel 99 : March 2000
Q only narrow band speech and Circuit switched
communications
Rel4: March 2001
QPacket switched for download
QSelection of wide band speech codec
Rel5: March 2002
QPacket switched conversationnal communications
Qnarrow band and wide band codecs
Rel6: September 2004
QCharacterisation of default codecs for Packet switched
conversationnal communications
confidential
France Telecom R&D
Distribution of this document is subject to France Telecom’s authorization
D3 - 23/05/2016
3GPP/ Characterisation of default
codecs for Packet switched
conversationnal communications
s
s
s
1st phase
QAMR codec in quiet and noisy environment
QAMRWB codec in quiet and noisy environment
2nd phase
QAMR and AMRWB versus legacy codecs in quiet
environment
Further experiment
QAMR and AMRWB versus legacy codecs in noisy
environment : car noise
confidential
France Telecom R&D
Distribution of this document is subject to France Telecom’s authorization
D4 - 23/05/2016
Methodology
s
VoIP in UMTS
Q delay is one major issue
– Interleaving due to radio environment
– Buffering in decoder side to cope with jitter
– IP transport
– Packetisation in IP as well as on radio leg
QNot only quality of codecs but acceptability of service
s
Conversation test
QMore accurate to assess acceptability of service
confidential
France Telecom R&D
Distribution of this document is subject to France Telecom’s authorization
D5 - 23/05/2016
Experiment: Description
s
s
s
Noisy environment : car noise @ 60dB Pa
Q Car noise is in one of the room, the other room is quiet
Codecs under test
QAMR (6.7 and 12.2 modes)
QAMRWB (12.65 and 15.85 modes)
QG.723.1@ 6.3 kb/s
QG.729@ 8 kb/s
QG.711@ 64 kb/s
QG.722 @ 64 kb/s
Environmental conditions
QIP packet loss : 0% or 3%
confidential
France Telecom R&D
Distribution of this document is subject to France Telecom’s authorization
D6 - 23/05/2016
Experiment: Description
PC 1 :
VOIP Terminal
Simulator
PC 3 :
Network Simulator
Network
Board A
Hub 1
PC 2 :
VOIP Terminal
Simulator
Network
Board B
Hub 2
confidential
France Telecom R&D
Distribution of this document is subject to France Telecom’s authorization
D7 - 23/05/2016
Results on voice quality criterion
s
4,5
Planned comparisons
4,0
th
QInteraction band th x noise for PLP=0%: NS
QInteraction Band x PLP for Noise: NS
Tuckey tests show a superiority of some
codecs on others for noisy and/or lossy
conditions, but not systematic. Without
noise and packet losses: no significant
differences.
Packet loss = 3%
G.711 + plc
G.722 64 kbs + plc
G.729 8 kbs
G.723.1 6.4 kbs
AMR WB 15.85 kbs
F(1,30) = 13.61 - p<0.001
AMR WB 12.65 kbs
F(1,30) = 20.23 - p<0.001
AMR NB 12.2 kbs
F
AMR NB 6.7 kbs
Packet loss = 0%
G.711 + plc
G.722 64 kbs + plc
G.729 8 kbs
AMR WB 15.85 kbs
codec *
AMR WB 12.65 kbs
France Telecom R&D
packet loss percentage (PLP) *
2,5
AMR NB 12.2 kbs
noise *
3,0
AMR NB 6.7 kbs
The superiority of WB Codecs is higher for
noisy and packet-loss conditions, but
there is no systematic benefit of WB in
noise.
Factor
3,5
G.723.1 6.4 kbs
s
5,0
Qsignificant main effects: noise, PLP & codec
Qsignificant interactions: codec x PLP
MOS
s
Voice Quality
ANOVA
ANOVA
s
Car noise
Without noise
F(7,210) = 14.26 - p<0.001
noise x PLP
F(1,30) = 0.68 - p=0.42
noise x codec
F(7,210) = 1.25 - p=0.28
codec x PLP*
F(7,210) = 2.3 - p<0.05
noise x codec x PLP
F(7,210) = 0.76 - p=0.62
confidential
Distribution of this document is subject to France Telecom’s authorization
D8 - 23/05/2016
Results on understanding criterion
s
Planned comparisons
4,0
th
QInteraction band th x noise for PLP = 0%: Signif.
QInteraction Band x PLP for Noise: NS
Tuckey tests show an almost systematic
superiority of WB and G.711 codecs on NB
codecs (except AMR NB 12.2 kbs) for
noisy
and/or lossy conditions. Without noise and
packet losses: no significant differences.
F(1,30) = 0.53 - p=0.47
noise x codec
F(7,210) = 1.86 - p=0.08
codec x PLP*
F(7,210) = 2.09 - p<0.05
noise x codec x PLP
F(7,210) = 1.88 - p=0.07
Packet loss = 3%
G.711 + plc
noise x PLP
G.722 64 kbs + plc
F(7,210) = 20.3 - p<0.001
G.729 8 kbs
codec *
G.723.1 6.4 kbs
F(1,30) = 11.07 - p<0.01
AMR WB 15.85 kbs
packet loss percentage (PLP) *
AMR WB 12.65 kbs
F(1,30) = 68.11 - p<0.001
AMR NB 12.2 kbs
F
AMR NB 6.7 kbs
Packet loss = 0%
G.711 + plc
G.722 64 kbs + plc
G.729 8 kbs
G.723.1 6.4 kbs
AMR WB 15.85 kbs
noise *
2,5
AMR WB 12.65 kbs
Factor
3,0
AMR NB 12.2 kbs
Considered globally, WB and G.711 codecs
better resist to noise for PLP=0%.
3,5
AMR NB 6.7 kbs
s
4,5
MOS
s
5,0
Qsignificant main effects: noise, PLP & codec
Qsignificant interactions: codec x PLP
ANOVA
s
Understanding of the partner
ANOVA
Car noise
Without noise
confidential
France Telecom R&D
Distribution of this document is subject to France Telecom’s authorization
D9 - 23/05/2016
Results on interaction criterion
s
s
Interactivity
ANOVA
5,0
Qsignificant main effects: noise, PLP & codec
Qsignificant interactions: codec x noise
4,5
Planned comparisons
4,0
th
2,5
ANOVA
noise x PLP
Packet loss = 3%
G.711 + plc
F(7,210) = 6.97 - p<0.001
G.722 64 kbs + plc
codec *
G.723.1 6.4 kbs
F(1,30) = 12.66 - p<0.001
AMR WB 15.85 kbs
packet loss percentage (PLP) *
AMR WB 12.65 kbs
F(1,30) = 24.58 - p<0.001
AMR NB 12.2 kbs
F
AMR NB 6.7 kbs
G.722 64 kbs + plc
G.723.1 6.4 kbs
AMR WB 15.85 kbs
AMR WB 12.65 kbs
AMR NB 12.2 kbs
noise *
AMR NB 6.7 kbs
Factor
Packet loss = 0%
G.729 8 kbs
s
As for the understandy criterion,
considered globally, WB codecs better
resist to noise for PLP=0%.
3,0
G.711 + plc
Tuckey tests show only a few significant
differences for noisy conditions only.
3,5
G.729 8 kbs
s
MOS
QInteraction band th x noise for PLP = 0%: Signif.
QInteraction Band x PLP for Noise: NS
Car noise
Without noise
F(1,30) = 0.3 - p=0.59
confidential
France Telecom R&D
noise x codec
F(7,210) = 2.67 - p<0.05
codec x PLP*
F(7,210) = 0.63 - p=0.73
noise x codec x PLP
F(7,210) = 0.96 - p=0.46
Distribution of this document is subject to France Telecom’s authorization
D10 - 23/05/2016
Results on default perception criterion
s
s
5,0
Qsignificant main effects: noise, PLP & codec
Qsignificant interactions: codec x PLP
4,5
4,0
Planned comparisons
th
QInteraction band th x noise for PLP = 0%: NS
QInteraction Band x PLP for Noise: NS
Tuckey tests show no significant
differences between codecs for any
condition.
3,5
MOS
s
Default perception
ANOVA
3,0
2,5
2,0
Packet loss = 3%
G.711 + plc
G.722 64 kbs + plc
G.729 8 kbs
G.723.1 6.4 kbs
AMR WB 15.85 kbs
F(1,30) = 11.95 - p<0.01
AMR WB 12.65 kbs
F(1,30) = 29.5 - p<0.001
AMR NB 12.2 kbs
F
AMR NB 6.7 kbs
Packet loss = 0%
G.711 + plc
G.722 64 kbs + plc
G.723.1 6.4 kbs
AMR WB 15.85 kbs
codec *
AMR WB 12.65 kbs
packet loss percentage (PLP) *
AMR NB 12.2 kbs
noise *
AMR NB 6.7 kbs
Factor
G.729 8 kbs
The default perception criterion does not
seem relevant to detect a possible
advantage of WB codecs in noise.
ANOVA
s
Car noise
Without noise
F(7,210) = 6.48 - p<0.001
noise x PLP
F(1,30) = 0.21 - p=0.65
noise x codec
F(7,210) = 1.67 - p=0.12
codec x PLP*
F(7,210) = 2.47 - p<0.05
noise x codec x PLP
F(7,210) = 0.82 - p=0.57
confidential
France Telecom R&D
Distribution of this document is subject to France Telecom’s authorization
D11 - 23/05/2016
Results on global quality criterion
s
4,0
Planned comparisons
th
QInteraction band th x noise for PLP = 0%: NS
QInteraction Band x PLP for Noise: NS
3,5
Tuckey tests show an almost systematic
superiority of WB and G.711 codecs on NB
codecs (except AMR NB 12.2 kbs) for noisy
and/or lossy conditions. Without noise and
packet losses: no significant differences.
2,5
2,0
F(1,30) = 0.29 - p=0.59
noise x codec
F(7,210) = 1.28 - p=0.26
codec x PLP*
F(7,210) = 1.68 - p=0.11
Packet loss = 3%
G.711 + plc
noise x PLP
G.722 64 kbs + plc
F(7,210) = 13.54 - p<0.001
G.729 8 kbs
codec *
G.723.1 6.4 kbs
F(1,30) = 14.51 - p<0.001
AMR WB 15.85 kbs
packet loss percentage (PLP) *
AMR WB 12.65 kbs
F(1,30) = 57.37 - p<0.001
AMR NB 12.2 kbs
F
AMR NB 6.7 kbs
Packet loss = 0%
G.711 + plc
G.722 64 kbs + plc
G.729 8 kbs
G.723.1 6.4 kbs
AMR WB 15.85 kbs
AMR WB 12.65 kbs
AMR NB 12.2 kbs
As for the voice-quality criterion, the
superiority of WB Codecs is higher for noisy
and packet-loss conditions, but there is no
systematic benefit
Factor
of WB in noise.
noise *
3,0
AMR NB 6.7 kbs
s
4,5
MOS
s
5,0
Qsignificant main effects: noise, PLP & codec
Qsignificant interactions: none
ANOVA
s
Global quality of the communication
ANOVA
Car noise
Without noise
confidential
France Telecom R&D
noise x codec x PLP
F(7,210) = 0.4 - p=0.9
Distribution of this document is subject to France Telecom’s authorization
D12 - 23/05/2016
Summary of subjective tests results
s
ANOVA is not useful for determining if WB codecs are significantly better than NB
codecs in noisy environment but show that noise was the strongest experimental factor
that affected the subjective data.
s
Planned comparisons and Tuckey tests show that:
Qthe MOS differences between WB and NB are significantly larger in noisy conditions than in
silent conditions for no packet losses, for "Understanding" and "Interactivity" criteria.
Q the MOS differences between WB and NB are often but not systematically larger in noisy
and/or 3% packet-loss conditions than in the silent and 0 % packet-loss condition, for the
"Voice quality" and "Global quality" criteria.
Qthe differences between WB and NB is independant on the presence of noise and packet loss
for the "Default perception" criterion.
s
It can be concluded that without packet loss, in noise, the advantage of WB compared
to NB is that it enhances the comprehension and the interaction with the other partner,
but not voice and global quality, since these criteria are highly affected by the presence
of noise.
confidential
France Telecom R&D
Distribution of this document is subject to France Telecom’s authorization
D13 - 23/05/2016