Assessing User QoE with MedUX's Fixed Broadband Benchmarking Methodology

MedUX’s fixed broadband Quality of Experience (QoE) benchmarking methodology offers a customizable and dependable framework for evaluating customer experience. This methodology employs a distinctive scoring system that integrates both technical and user-focused parameters and adheres to international standards. This enables ISPs and NRAs to make informed decisions based on data, fostering fair competition.

Regulatory authorities and internet service providers face the ongoing challenge of ensuring exceptional QoE while increasing investments in network infrastructure and emerging technologies.

Our fixed broadband QoE benchmarking is valuable for a range of stakeholders, including internet service providers (ISPs), equipment/service vendors, content providers, and end-users. It provides means to:

• Assess the performance of their services, devices, and networks.
• Identify areas needing improvement.
• Promote fair competition through objective and transparent data.
• Enhance the overall quality of fixed broadband services.

For ISPs, customer experience is a critical metric. It has become integral to performance and operational and management reporting, customer satisfaction initiatives, and their market positioning, both internally and for external promotion.

For national regulatory agencies (NRAs), customer experience is increasingly vital. It has become a key aspect of consumer protection, evolving from traditional Quality of Service (QoS) approaches to include the user’s perception of services and to assess how network and service issues affect customer satisfaction.

Our fixed broadband benchmarking methodology and framework concentrate on evaluating the QoE that fixed broadband users experience, also referred to as customer-perceived network quality or network experience. This assessment includes:

• Technical QoS metrics: throughput, DNS resolution time, latency, jitter, and packet loss.
• User-centric QoE metrics: web browsing, media streaming, gaming, and using social networks or cloud storage.

We aggregate various performance indicators into a composite score, the MedUX QoE Scorecard. This score is used for competitive analysis, network optimization, public visibility, and quality certifications.

Establishing the Benchmarking Framework

Our Fixed Broadband QoE benchmarking system is designed to be adaptable and customizable to address the specific requirements of each client.

To ensure that our benchmarking outcomes are dependable and consistent, we adhere to a standardized methodology and employ a comprehensive framework that encompasses all essential aspects of fixed broadband QoE. This process includes defining the use cases, selecting suitable devices and data collection methods, establishing the metrics, measuring the technical parameters, and analyzing the results to form conclusions and recommendations.

The benchmarking process involves creating a structure for gathering and analyzing data:

1. Identify Target Use Cases: We determine the needs and set the objectives based on client type, such as marketing and positioning, network intelligence, technology or vendor comparison, or consumer empowerment.
2. Choose Appropriate Data Collection Methods: We assemble the most suitable ecosystem to perform measurements based on the use case. This may involve a mix of proprietary software and hardware tools to measure technical QoS parameters, simulate real user behavior, and gather insights into true QoE. We combine measurements from customer premises with our proprietary network testing and data collection robots, along with other tools from our multi-platform monitoring ecosystem, such as crowdsourcing, SDKs, and Web Agents.
3. Define Key Performance Indicators (KPIs): We identify the metrics and testing schedules that best measure QoE for each use case and business objective. This includes selecting testing scenarios and configuring our software and hardware tools to cover the technologies and services subject to the benchmarking assessment.
4. Execute Testing and Analysis: We simulate user interactions and collect KPIs using automated tools within a targeted geographical and temporal scope. We then analyze the results and apply our scoring methodology, providing a fair, transparent, and independent evaluation of all networks, summarized in the MedUX QoE Scorecard and MedOS (MedUX Opinion Scores).

By conducting these tests and analyzing the outcomes, ISPs and NRAs can pinpoint areas for improvement and ensure better quality services for end-users.

Integrating International Standards and Best Practices into a Unique Scoring Methodology

The MedUX QoE Scorecard serves as a benchmarking tool for assessing network quality and performance, providing a QoE score for fixed broadband providers. We follow testing procedures from an end-user perspective, using real commercial equipment in actual environments, with real services, and real end-points. The score aggregates various KPIs for each service category, weighted based on international best practices and industry contributions.

Our testing capabilities cover the entire ecosystem affecting the end-user experience, including environmental factors, terminal capabilities, network equipment, service configurations, infrastructure architecture, and server locations. We evaluate from source to destination to gain visibility into end-to-end network and service performance.

Our active speed tests typically measure true end-to-end throughput using endpoints in multiple Content Delivery Networks (CDNs). We enable operators to deploy dedicated endpoint servers for specific projects but not for independent benchmarking initiatives.

The methodology underpinning the MedUX QoE Scorecard is based on international best practices, standards, and recommendations from standardization bodies such as ETSI TR 103 559 V1.1.1. This ensures consistent and reliable benchmarking, making results comparable across different operators and countries.

By adhering to guidelines set forth in ETSI TR 103 559 V1.1.1, we provide operators with high-quality, accurate assessments of their network performance, enabling them to make data-driven decisions to enhance QoE.
In addition to maintaining a consistent and standard methodology for global comparisons, we are open to adjusting the testing methods and scoring models to reflect market developments, technological advancements, and specific use cases. Our QoE Scorecard also takes into account changing consumer behaviors and preferences, growing customer expectations, and rising data speeds, by annually revisiting the requirements and thresholds to ensure relevance to technology and market trends.

In summary, our scoring methodology adheres to these core principles:

• International standards and recommendations
• Comparability
• End-user perspective
• End-to-end performance
• Real services and true QoE KPIs
• Technology and market relevance

Leveraging True QoE KPIs from Real-Field Data

Our QoE benchmarking campaigns encompass a comprehensive set of KPIs that we believe best represent the customer experience with real services. We continuously invest in research and development to discover new tests and metrics.

Our current service experience categories include:

• Overall QoE Score: Combines accessibility, speed, streaming, and service and OTT experience results:

· Accessibility (Responsiveness): Assesses network access time. Particularly in fixed networks, it measures the time required to access the network across all testing categories, considering latency and network responsiveness.

· Matching the promise (Throughput): Evaluates data transfer capacity. In fixed networks, it measures network capacity under stress (multi-thread and multi-server stress test with our patented Cloud Speed Test), including:

• Contracted Speed Compliance (DL Speed):

· Video Streaming Experience: Measures the average video experience for each operator, taking into account picture quality, video loading time, and stall rate.

· Data and OTT Experience: A combination of the following:

• Gaming: Evaluates the average gaming experience considering mobile network conditions such as latency, packet loss, and jitter for popular games.
• Web Browsing: Assesses the average web browsing experience, including loading time, DNS resolution, and time to the first byte for popular websites.
• Social Networks: Evaluates the average experience when accessing social networks like Facebook or Twitter.
• HTTP Data Transfer: Assesses the average experience of downloading and uploading files (single thread) that represent typical interactions with web pages and apps.
• Reliability: Evaluates network consistency. Particularly in fixed networks, it measures the consistency of the network performance across all testing categories, considering the success rate (%) of all tests and services.  A test is considered as not successful if it fails or if does not meet minimum service thresholds.

For each service category, we collect several KPIs and process thousands or millions of complex data points, depending on the benchmarking campaign. These data points are then transformed into a synthetic indicator using a proprietary algorithm based on international best practices, standards, and recommendations.

Stay tuned for our upcoming Quality seals and awards, which will leverage our QoE scorecard and MedOS, including a mark for each operator and a best-in-class award.

Leveraging Our Unique Synthetic QoE Indicator for Actionable Insights

The MedUX QoE Scorecard spans various service categories such as web browsing, video streaming, online gaming, and cloud storage. This comprehensive coverage enables fixed broadband operators to gauge the quality of experience (QoE) their subscribers encounter and compare their performance with that of their competitors.

The MedUX Scorecard is designed to be user-friendly for various stakeholders within a broadband operator's organization, from executive management to technical departments like engineering and optimization. The scoring model for each service category, along with its drill-down capabilities, facilitates detailed analysis of the QoE for each service category, known as MedOS (MedUX Opinion Scores). This structure allows executives to gain a high-level overview of the network's overall performance, while engineers can delve into the data to pinpoint and resolve specific issues.

Our QoE Scorecard, particularly the MedOS for each service category, simplifies the process of translating multiple KPIs into a clear and easily comprehensible synthetic performance indicator. This approach enables providers and operators to better understand their customers' QoE and take proactive measures to enhance it. The MedUX Scorecard serves as a reliable benchmark for fixed broadband performance, helping operators compare their performance against industry standards and pinpoint areas needing improvement. By optimizing their networks and services, providers can offer a superior experience to their customers, thereby gaining a competitive edge.

Furthermore, the QoE Scorecard provides insights into the impact of new technologies on network performance and user experience. By assessing the effects of new technologies and services on QoE, operators can make informed decisions about network investments and service offerings. The Scorecard also enables the evaluation of customer experience across different geographic areas, helping operators identify regions needing improvement and prioritize network investments. This is crucial for ensuring equitable access to fixed broadband services and bridging the digital divide.

In addition to benefiting fixed broadband operators, the MedUX QoE Scorecard is also utilized by industry regulators, governments, and consumer organizations to monitor the quality of fixed broadband services and advocate for service quality improvements. Overall, the MedUX QoE Scorecard is a powerful tool for promoting transparency and accountability in the broadband industry, helping to ensure that consumers enjoy the best possible service.

In conclusion, our fixed broadband QoE benchmarking methodology and framework are essential for measuring and enhancing the quality of fixed broadband services. They provide invaluable insights into:

• The overall performance of fixed broadband networks and devices
• Your performance compared to competitors and international peers
• Trends and evolution of your performance
• Performance variations based on environmental, resourcing, or configuration factors, including spectrum, setup, and vendors
• By ensuring a seamless and satisfying experience for broadband end-users, our benchmarking tools contribute to the continuous improvement and optimization of broadband services.

Illustrating scoring principles for a particular use case: Fixed Broadband Quality of Experience benchmark in European countries

MedUX will present the results from its Fixed Broadband Quality of Experience benchmark in two European countries. Based on our testing framework and methodology:

• Select target use cases: Marketing and positioning, and network intelligence use cases.
• Select appropriate collection methods: MedUX Home robots  testing units, agnostic and all-router compatible, are a Plug and Play solution based on non-intrusive, reliable tools. Our MedUX Home probes perform active measurements without integration into the operators’ networks, in the home network environment, including the performance of fixed-broadband (Ethernet) and Wi-Fi (2.4GHz and 5GHz) interfaces.
• Define key performance indicators (KPIs): Data testing only, with web browsing, social networks (Facebook and Twitter), gaming (Epic, Riot Games, Valve, Blizzard), streaming, speed test, file download and upload, responsiveness and accessibility (ping)
• Conduct testing and analysis: 3 months of continuous testing schedule in selected households based on our statistical basis. 

Our statistical model is focused on providing statistical significance at national level for benchmarking purposes. MedUX uses in this case an approach similar to the TV audience measurement (e.g. Nielsen), with a statistical approach defined by International Telecommunications Union (ITU) and European Telecommunications Standards Institute (ETSI). With this benchmarking approach, the defining parameters of the model are:

1. The number of observation points per dimension for fixed networks measurements
2. The number of tests per relevant geographical area
3. The period of time

MedUX introduces this concept to measure and characterize a service with some common specific attributes (service provider, technology, or geographical area, among others).

To assess network reliability, accessibility, and value for speed, across the typical activities from end-users on their daily lives, like web browsing, using social networks, gaming, video streaming, uploading and downloading files, we have followed these guidelines based on our consistent and standardized methodology:

The service category scoring model and its drill-down capabilities makes it exploitable by different stakeholders, from executive management (CxOs) to more technical departments (engineering and optimization). Thus, providers or operators can improve their networks and services offering a better experience to their customers.

The overall score is calculated in a range of 0-5 points. Each service category is an aggregation of at least one MedUX Opinion Score. Each MOS is characterized by a Qualifier KPI and one or several Differentiator KPIs which turn technical measurements to a QoE metric of each Service.

In order to achieve an overall score, the individual test results have to be given a weight. This weight is the importance with which the result enters into the overall valuation of the service. The weighting of the results is done on each level of aggregation, from the service categories to the Qualifiers and Differentiators KPIs.

GET A DEMO

Don’t forget to follow us on social media and subscribe to our newsletter to receive the latest updates and news.

Follow us on:

   

 About us

MedUX is the leading Quality of Experience (QoE) company, providing comprehensive and innovative solutions for measuring the performance of fixed, mobile, and TV telecommunications networks for telecom operators, governments, and digital enterprises. With a focus on delivering grand scale, end-to-end network and service visibility, MedUX leverages real-time customer perspective data and advanced analytics to ensure quality and regulatory compliance, while also offering valuable insights for optimizing networks and improving customer experiences. With a presence in over 25 countries across Europe, America, Africa, and the Middle East, and monitoring over 60 operators worldwide, MedUX’s patented technology and expertise make it a trusted partner for improving the digital experiences of customers everywhere.