Assessing Customer Experiences with a universal mobile broadband benchmarking methodology
Assessing Customer Experiences with a universal mobile broadband benchmarking methodology
MedUX’s mobile QoE benchmarking methodology provides a customizable and reliable framework for assessing customer experience, utilizing a unique scoring methodology that combines technical and user-centric parameters and follows international standards, enabling MNOs and NRAs to make data-driven decisions and promoting fair competition.
Regulatory authorities and telecommunications operators alike, face the challenge of ensuring top-notch Quality of Experience (QoE) while elevating investment in network deployments and new technologies.
Our mobile QoE benchmarking is relevant for various stakeholders, including mobile network operators (MNOs), device manufacturers, equipment/service providers, content providers, and end-users. It allows them to assess the performance of their services, devices, and networks, and to identify areas for improvement. By providing objective and transparent data, our benchmarking helps to promote fair competition and to improve the overall quality of mobile services.
Customer experience is a key metric for MNOs. It has become an essential part of performance and operational reporting, management reporting, customer satisfaction programs, and their positioning in the market for internal review or external promotion.
Customer experience is turning more important than ever for national regulatory agencies (NRAs). It has become an essential part of consumer protection, as the traditional approach to Quality of Service (QoS) is evolving to embrace the user’s perception of a particular service and to assess the impact of network and service degradations in customer satisfaction.
Our Mobile benchmarking methodology and framework focuses on evaluating the QoE that mobile end-users have when using telecommunication services, also called customer-perceived network quality or network experience. This process measures technical QoS parameters such as throughput, DNS resolution time, latency, jitter, packet loss, as well as user-centric QoE parameters when browsing the web, streaming media, gaming, or using social networks or cloud storage.
We combine different performance indicators into a synthetic indicator or single score, the MedUX QoE Scorecard. The score is used for competitive intelligence, network optimization, public visibility, and quality certifications.
Setting-up the benchmarking framework
Our mobile QoE benchmarking framework is designed to be flexible and customizable to meet the specific needs of each client.
To ensure that our benchmarking results are reliable and repeatable, we follow a standardized methodology and use a comprehensive framework that covers all the key aspects of mobile QoE. This includes defining the use cases, selecting the appropriate devices and collection methods, defining the metrics and measuring the technical parameters, and analyzing the results to draw conclusions and recommendations.
The benchmarking process involves setting up a framework for collecting and analyzing data.
- Select target use cases: We identify the needs and set the goals depending on client type, e.g. marketing and positioning, network intelligence, technology or vendor comparative analysis, or consumer empowerment.
- Select appropriate collection methods: We put together the most appropriate ecosystem to perform the measurements based on the use case. We can use a combination of proprietary software and hardware tools, which allow us to measure the technical QoS parameters, to simulate real user behavior and to gather insights into the true QoE. We combine drive and walk tests based on our proprietary network testing and data collection robots, with other data collection tools from our multi-platform monitoring ecosystem, such as crowdsourcing, SDKs and Web Agents.
- Define key performance indicators (KPIs): We determine the metrics and testing schedule that best measure the QoE for each use case and business goal. It includes the selection of testing scenarios, meaning the configuration of our software and hardware tools to cover the technologies and services subject to the benchmarking assessment.
- Conduct testing and analysis: We simulate user interactions and collect KPIs using automated tools within a target geographical and temporal reach. We then analyze the results and apply our scoring methodology providing a fair, transparent, and independent evaluation of all networks. We call it the MedUX QoE Scorecard and MedOS (MedUX Opinion Scores).
By conducting these tests and analyzing the results, MNOs, and NRAs can identify areas for improvement and ensure better quality services to the end-users.
Combining international standards and best practices into a unique scoring methodology
The MedUX QoE Scorecard is a benchmarking tool for network quality and performance assessment, providing a QoE score for mobile broadband providers. We follow testing procedures from an end-user perspective, measuring from real commercial equipment in a real environment, with real services, and to real end-points. The score is a combination of different KPIs for each service category, weighted based on international best practices and industry contributions.
Our testing capabilities cover the complete ecosystem affecting the end-user experience: environmental particularities, terminal capabilities, network equipment, services configuration, infrastructure architecture, servers’ location, etc. In short, we assess from the source to the destination to gain visibility into end-to-end network and service performance.
Our active speed tests commonly measure the true end-to-end throughput experience by using endpoints in multiple Content Delivery Networks (CDNs). We allow operators to deploy dedicated endpoint servers for dedicated projects but not for independent benchmarking initiatives.
The methodology behind the MedUX QoE Scorecard is based on international best practices, standards, and recommendations from standardization bodies such as ETSI TR 103 559 V1.1.1 and. This ensures that the benchmarking is carried out in a consistent and reliable manner and that the results are comparable across different operators and countries.
By following the guidelines set out in the ETSI TR 103 559 V1.1.1, we are able to provide operators with a high-quality, accurate assessment of their network performance, which they can use to make data-driven decisions to improve their QoE.
Apart from having a consistent and standard methodology to enable global comparisons, we also are open to adjusting the testing methodology and scoring models to the developments of the market, the technology, and the particular use case. Our QoE Scorecard also considers changing consumer behaviors and preferences, growing customer expectations, and rising data speeds by revisiting the requirements and thresholds every year to ensure technology and market relevance.
In summary, our scoring methodology follows these basic principles:
- International standards and recommendations
- Comparability
- End-user perspective
- End-to-end performance
- Real services and true QoE KPIs
- Technology and market relevance
Leveraging True QoE KPIs from real-field data
Our QoE benchmarking campaigns contain the set of QoE KPIs we believe best represent customers’ experience of using real services. We constantly invest in research and development to find new tests and metrics.
Our current list of service experience categories, that in turn have an independent MedOS differentiating between 5G-only and blended results e.g. 3G/4G/5G, include:
- Overall QoE Score – A combination of availability, accessibility, speed, streaming, and services and OTT experience results.
- Availability (coverage) – how available the service, particularly in 5G it measures only the average proportion of time a 5G device spends with an active 5G connection
- Accessibility (responsiveness) – how long it takes to access the network, particularly in 5G it measures how long takes to access the 5G network across all testing categories while taking latency and network responsiveness results into account
- Value for Speed (throughput)- how much data can be transferred, particularly in 5G measures how is the network capacity when stressing the line (multi-thread and multi-server stress test with our patented Cloud Speed Test),
- DL Speed
- UL Speed
- Video Streaming experience, it measures the average video experience for each operator while taking picture quality, video loading time, and stall rate into account.
- Data and OTT experience, a combination of the following
- Gaming, it evaluates the average gaming experience while taking mobile network conditions into account, including latency, packet loss, and jitter, for some of the most famous games.
- Web browsing, it evaluates the average web browsing experience while taking loading time, DNS resolution, and time to the first byte into account for some of the most famous websites.
- Social networks and messaging, it evaluates the average experience when accessing social networks such as Facebook or Twitter and using messaging platforms such as Whatsapp
- HTTP Data Transfer, it evaluates the average experience while downloading and uploading files (single thread) that represent typical interactions with webpages and apps.
- Reliability – how consistent a network performs, particularly in 5G it measures how consistent the 5G network performs across all testing categories considering the success rate (%) of all tests and services above
For each of the service categories we obtain several KPIs and thousands or millions of complex data points depending on the benchmarking campaign and then we transpose them into a synthetic indicator using a proprietary algorithm based on international best practices, standards, and recommendations.
Stay tuned to our soon-to-be-launched Quality seals and awards leveraging our QoE scorecard and MedOS, including a mark for each operator as well as a best-in-class award.
Exploiting our unique synthetic QoE indicator for actionable insights
The MedUX QoE Scorecard covers a wide range of service categories, including web browsing, streaming video, online gaming, and cloud storage. This allows mobile network operators to understand the quality of experience that their subscribers are having, and to compare their performance to that of their competitors.
The MedUX Scorecard is designed to be accessible to different stakeholders within a mobile network operator, from executive management to more technical departments like engineering and optimization. This is because the service category scoring model and drill-down capabilities allow for granular analysis of the QoE of each service category, the so called MedOS (MedUX opinion scores). This means that executives can get a high-level overview of the overall performance of the network, while engineers can dig deep into the data to identify and troubleshoot specific issues.
Our QoE Scorecard, and particularly the MedOS for each service category that we test, offer a simple way to translate several KPIs into a clear and easy-to-understand synthetic performance indicator.
With this approach, providers and operators can better understand their customers’ QoE and take proactive measures to improve it. The MedUX Scorecard provides a reliable benchmark for mobile broadband performance, allowing operators to compare their performance against industry averages and identify areas for improvement. By optimizing their networks and services, providers can offer a better experience to their customers and gain a competitive advantage in the marketplace.
Moreover, the QoE Scorecard can also provide insights into the impact of new technologies, such as 5G, on network performance and user experience. By evaluating the impact of new technologies and services on QoE, mobile operators can make informed decisions about network investment and service offerings.
Besides, the Scorecard can be used to assess the mobile experience in different geographical areas, allowing operators to identify areas for improvement and prioritize network investments. This is particularly important in ensuring equitable access to mobile services and bridging the digital divide.
In addition to providing valuable insights to mobile network operators, the MedUX QoE Scorecard is also used by industry regulators, governments, and consumer organizations to monitor the quality of mobile broadband services and to advocate for improvements in service quality. Overall, the MedUX QoE Scorecard is a powerful tool for promoting greater transparency and accountability in the mobile telecommunications industry, and for helping to ensure that consumers can enjoy the best possible mobile experience.
In summary, our mobile QoE benchmarking methodology and framework are essential tools for measuring and improving the quality of mobile services. All in all, they help ensure that mobile end-users have a seamless and satisfying experience. They provide valuable insights into:
- the overall performance of mobile networks and devices
- your performance against competitors and international peers
- your performance evolution and trends of your performance
- performance depending on environmental, resourcing or configuration particularities, including spectrum, setup and vendors
Illustrating scoring principles for a particular use case: 5G Quality of Experience benchmark in Europe
MedUX will present and discuss the results from the first 5G Quality of Experience benchmark in Europe in the MWC. Based on our testing framework and methodology:
- Select target use cases: Marketing and positioning, and network intelligence use cases.
- Select appropriate collection methods: MedUX Mobile robots with 5G capabilities (in unforced mode)
- Define key performance indicators (KPIs): Data testing only, with web browsing, social networks (Facebook and Twitter), gaming (LOL, Valve, Blizzard), streaming, speed test, file download and upload, responsiveness and accessibility (ping), DNS resolution
- Conduct testing and analysis: 3-4 days continuous testing schedule in city center of main European cities
To assess network availability, reliability, accessibility, and value for speed, across the typical activities from end-users on their daily lives, like web browsing, using social networks, gaming, making calls, and sending texts, we have followed these guidelines based on our consistent and standardized methodology:
Above indicators are also calculated across an operator’s 3G, 4G and 5G networks with a 5G device and subscription to obtain a blended result across technologies for 5G enabled devices. In this case, it does factor the results in 3G, 4G and 5G networks, and also the availability of each network technology.
About us
MedUX is the leading Quality of Experience (QoE) company, providing comprehensive and innovative solutions for measuring the performance of fixed, mobile, and TV telecommunications networks for telecom operators, governments, and digital enterprises. With a focus on delivering grand scale, end-to-end network and service visibility, MedUX leverages real-time customer perspective data and advanced analytics to ensure quality and regulatory compliance, while also offering valuable insights for optimizing networks and improving customer experiences. With a presence in over 25 countries across Europe, America, Africa, and the Middle East, and monitoring over 60 operators worldwide, MedUX’s patented technology and expertise make it a trusted partner for improving the digital experiences of customers everywhere.
Tagged with: Customer Experience • Customer Satisfaction • CX • Network Monitoring • NetworkPerformance • OTT • QoE • QoS • Quality of Experience • telecommunications
related POSTS:
MWC23 confirmed the QoE Revolution is a real movement!
MWC23 confirmed the #QoERevolution is a real movement! MedUX is grateful to all attendees that participated in the presentations of
All 5G networks are not created equal: Unveiling the TRUE QoE of 5G in Europe
All 5G networks are not created equal: Unveiling the TRUE QoE of 5G in Europe MedUX will present the results
Empowering Customer Experiences with MedUX at MWC23
Universal QoE Solutions Built for the Digital Era Join in the Quality of Experience (QoE) Revolution at MWC23 with MedUX-
C/ Fuencarral, 123. 28010. Madrid (Spain)
(+34) 916 683 733
hello@medux.com
Any need related to Quality and Customer Experience? We are here to help you!