Tems Moving From Pesq to Polqa

MOVING FROM PESQ TO POLQA The Next-Generation Mobile Voice Quality Testing Standard

Dr. Irina Cotanis, on Moving from PESQ to POLQA

1

MOVING FROM PESQ TO POLQA

 Why is a move from PESQ to POLQA recommended? (And, why will the move eventually be necessary?)  What are the limitations of the POLQA technology?  What are the main differences between POLQA and PESQ?  What are the implications of moving to the new POLQA technology?

Dr. Irina Cotanis, on Moving from PESQ to POLQA

2

WHY MOVE FROM PESQ TO POLQA? Reason 1. POLQA fixes known and recognized PESQ weaknesses.  WB measurements in general  Inaccuracy with CDMA codecs (e.g., EVRC) and, therefore, CDMA networks t k iin generall  Sensitivity to certain GSM/WCDMA network conditions ((DTX/Voice Activityy Detection in AMR codec))  VoIP limitations (variable delay up to 1sec)  Strong linear distortions (e.g., phone’s frequency shaping) as well as gain variations  Limitations with speech processing (enhancement) devices (noise reduction reduction, automatic gain control) Dr. Irina Cotanis, on Moving from PESQ to POLQA

3

WHY MOVE FROM PESQ TO POLQA? Reason 2. POLQA has new operational features.  Works in all three available bandwidths (NB, WB, SWB) and under different background noise conditions (street, car, restaurant, babble, impulse, etc.)  W Works k with ith allll new commercial i l and d standardized t d di d codecs d used d iin diff differentt technologies (GSM/WCDMA, LTE, VoIP, VoIP over IMS) such as AMR-WB, EVRC-WB, iLBC, AMB+, AAC, Skype / SLIK, G.711, G.729, etc.  Allows measurements at various combinations of interfaces: electrical-electrical, acoustical-acoustical, electrical-acoustical, and acoustical electrical — providing the opportunity to test terminals and acoustical-electrical hands-free applications

Dr. Irina Cotanis, on Moving from PESQ to POLQA

4

WHY MOVE FROM PESQ TO POLQA? Reason 3. POLQA copes with new types of degradations.  New codec conditions  Single speech codecs and speech codecs working in tandem, packet loss and d concealment l t strategies t t i (PS), (PS) fframe errors and d bit errors ((wireless), i l ) effects of coding on pre-noised speech p degradations g  Network specific  Interruptions (e.g., unconcealed packet loss or handover in GSM), front-end clipping (temporal clipping), amplitude clipping (overload, saturation), t ti ) time ti warping/time i /ti scaling li (VoIP (V IP over IMS)

Continued on next page Dr. Irina Cotanis, on Moving from PESQ to POLQA

5

WHY MOVE FROM PESQ TO POLQA? Reason 3. POLQA copes with new types of degradations. (continued)  Voice Enhancement Devices (speech processing systems) implemented both in the CN as well as in the terminals  Eff Effects t off noise i reduction d ti systems t and d echo h cancellers ll on clean l speech h as well as on pre-noised speech  Terminals  Influence of time variant linear distortions (spectral shaping), non-linear distortions produced by microphone/transducer at acoustical interfaces  Reverberations caused by hands-free test setups in defined acoustical environments

Dr. Irina Cotanis, on Moving from PESQ to POLQA

6

WHY MOVE FROM PESQ TO POLQA? Reason 4. POLQA has been proven to over-perform PESQ.  POLQA NB mode has shown increased performance of up to 25% as compared with PESQ NB  Similar values haven been observed for POLQA SWB mode as compared with PESQ WB

Dr. Irina Cotanis, on Moving from PESQ to POLQA

7

WHAT ARE POLQA’S LIMITATIONS? STRICT LIMITATIONS. POLQA should be not used at all for:  Other dimensions of speech quality such as conversational aspects and talking quality  Speech quality per call. POLQA is not intended to score longer sequences of speech. It is focused on prediction of quality for shorter speech utterances of 6 to 12 seconds  Noisy listening environments. POLQA does not predict perceived speech quality in these environments; it is designed in accordance with P.800, ACR testing.  Music (including multimedia)  Evaluation of performance or ranking of voice enhancement devices (e.g., noise suppressors)  Other technologies or components such as speech storage formats or non-telephony applications such as public safety networks or professional mobile radio connections Continued on next page Dr. Irina Cotanis, on Moving from PESQ to POLQA

8

WHAT ARE POLQA’S LIMITATIONS? (continued) POSSIBLE LIMITATIONS (scenarios on which POLQQ has not been tested/evaluated). In the following use cases, POLQA should be used carefully. Additional subjective testing might be needed for increased confidence.  Other languages (e.g., (e g Arabic Arabic, or others not used for POLQA development)  Longer speech samples  Multi-dimensional audible speech p q quality y diagnosis g NOTE. Work on Perceptual Approaches for Multi-Dimensional Analysis (PAMD) to predict perceptual dimensions of g linked to the overall speech p q quality y in NB, WB, and SWB telecommunication scenarios is ongoing g g degradations within ITU. PAMD aims at providing more detailed information about individual quality dimensions as additional information to the POLQA overall MOS. It should be noted that this is different than diagnosing network problems based on speech quality parameters as provided by POLQA.

Dr. Irina Cotanis, on Moving from PESQ to POLQA

9

WHAT ARE THE MAIN DIFFERENCES BETWEEN POLQA & PESQ?  Reference speech material: 48kHz vs. 8kHz sampling frequency  MOS-LQO output: POLQA score directly represents estimated MOS values on a subjective scale of 1 to 5 MOS, unlike PESQ, which used the P.862.1 mapping function for conversion to the MOS domain  Dual operational modes:  Narrowband NB (IRS filtering up to 3.4kHz; 8kHz sampling frequency of the reference sample) For backwards compatibility to PESQ — recommended to be run only for this scope

 Super wideband SWB (Flat filtering up to 14kHz, 48kHz sampling frequency of the reference sample) Continued on next page Dr. Irina Cotanis, on Moving from PESQ to POLQA

10

WHAT ARE THE MAIN DIFFERENCES BETWEEN POLQA & PESQ? (continued)  Unique triple bandwidth output scale:  SWB mode: NB, WB, and SWB tested samples presented on a single SWB scale (bandwidth being part of the transmission chain’s degradation)  This has the advantage of direct comparison of different bandwidth scores. NOTE: It is generally expected that the SWB unique scale (running a SWB reference sample) has the following transparency: •

NB:

Maximum MOS value 4.25

•

WB:

Maximum MOS value 4.5

•

SWB: Maximum MOS value 4.75

 New extensively tested and validated languages: Chinese, Japanese, Czech  Accuracy: POLQA proven to over over-perform perform PESQ (both NB and WB) Dr. Irina Cotanis, on Moving from PESQ to POLQA

11

WHAT ARE THE IMPLICATIONS OF MOVING TO POLQA?  Possibility to achieve slightly lower spatial-temporal granularity of the speech quality measurements  Longer processing time than PESQ due to algorithm complexity  Larger L memory requirements i t d due tto 48kHz 48kH sampling li ffrequency  Proper understanding of the SWB unique scale, which provides three bandwidth scores on the same scale  Backwards compatibility to PESQ  Valid only for PESQ NB.  Provides rankings rather than exact absolute values  Comparison should be made only between PESQ NB and POLQA NB

Dr. Irina Cotanis, on Moving from PESQ to POLQA

12

CONCLUSIONS

Dr. Irina Cotanis, on Moving from PESQ to POLQA

13

CONCLUSIONS The move from PESQ to POLQA is recommended (and will eventually be required) due to:  POLQA fixes well-known weaknesses of PESQ  POLQA copes with ith today’s t d ’ speech h codecs, d voice i enhancement h td devices, i and network conditions Q offers significantly g y higher g accuracy y  POLQA POLQA technology has still some limitations that must be well understood and controlled

Continued on next page Dr. Irina Cotanis, on Moving from PESQ to POLQA

14

CONCLUSIONS (continued) Main differences between POLQA and PESQ:  Reference speech sampling frequency  Operational modes / bandwidths  Unique triple bandwidth MOS LQO output scale  Accuracy Implications of moving to POLQA: p y to PESQ  Backwards compatibility  Possibly lower space–time granularity than PESQ  Understanding the unique SWB output scale as well as its difference from the NB scale Dr. Irina Cotanis, on Moving from PESQ to POLQA

15

THANK YOU!

Dr. Irina Cotanis, on Moving from PESQ to POLQA

16