Assessing Customer Experience with MedUX’s Advanced Mobile Crowdsourcing Benchmarking Methodology

Current methodology · Version 2026.1

MedUX’s mobile QoE crowdsourcing methodology provides a scalable and consistent framework for assessing customer experience using measurements from real devices and real network conditions.

Building on MedUX’s long-standing experience in mobile benchmarking, the methodology combines technical network indicators with service-level Quality of Experience metrics, applying a common scoring framework, geographical population standardisation and statistical inference to produce comparable operator assessments.

This enables Mobile Network Operators, National Regulatory Authorities and other telecom stakeholders to understand how mobile networks perform across everyday digital services, identify areas for improvement and support transparent market comparisons.

Crowdsourcing adds a complementary dimension to controlled probes and field-testing campaigns: it extends observation to a broad variety of devices, locations, technologies and usage contexts that cannot be fully reproduced through scheduled measurement campaigns alone. The current methodology strengthens this approach by making explicit how geography is standardised and how statistical leadership is determined.

Setting up the Crowdsourcing Benchmarking Framework

Customer experience has become a key performance dimension for Mobile Network Operators, influencing operational reporting, customer satisfaction, churn analysis and market positioning.

At the same time, regulators increasingly complement traditional Quality of Service metrics with Quality of Experience indicators that provide a more service-centric view of network performance.

MedUX’s methodology captures both traditional QoS parameters, such as throughput, latency, jitter, DNS resolution time, packet loss and radio indicators, and QoE measurements associated with services including web browsing, streaming, gaming, social media and cloud applications.

These measurements feed the MedUX QoE Scorecard, which converts multiple technical indicators into a structured view of mobile network experience.

The benchmarking process typically involves:

1. Selecting the target use case, such as market benchmarking, competitive intelligence, regulatory monitoring, network optimisation, technology assessment or consumer transparency.
2. Choosing the appropriate data-collection methods, including MedUX Crowdsourcing and SDK-based measurements, which may be complemented by controlled field testing or dedicated probes when required by the project.
3. Defining the relevant KPIs and services, covering both technical QoS measurements and service-oriented QoE indicators.
4. Applying validation, scoring, geographical standardisation and statistical analysis to obtain comparable market results.

The methodology and Scorecard configuration used for a benchmark are applied consistently to all eligible operators within the same publication.

MedUX QoE Crowdsourcing Methodology

MedUX captures mobile network experience from real end-user devices through a lightweight measurement technology integrated into mobile applications.

The framework combines active and passive observations, validates the resulting dataset and converts technical measurements into service-level experience indicators before producing national or market-level benchmark results.

The current methodology is built around five main pillars:

Real-world data collection, data validation, QoE scoring, geographical standardisation and statistical inference.

Real-world data collection

Data is collected through a combination of active and passive measurements, providing a multidimensional view of mobile network performance.

Active measurements reproduce common digital activities under standardised measurement logic. Depending on the benchmark configuration, these can include web browsing, speed and file transfers, video streaming, online gaming, cloud services and other digital applications.

Tests are designed to minimise interference with normal device usage while maintaining comparability between networks.

Passive observations complement active testing by capturing relevant network and device conditions, including radio environment, access technology and other contextual information available to the measurement framework.

Together, these observations allow MedUX to analyse mobile experience across different technologies, devices, locations and usage conditions.

Data validation and processing

All measurements pass through MedUX’s data-quality and processing pipeline before being included in benchmark calculations.

The validation process identifies measurements that do not meet the applicable technical or data-quality conditions and applies the controls required by each measurement type.

Depending on the dataset, these controls can include detection of anomalous or corrupted samples, duplicate measurements, invalid measurement states, unsupported configurations and observations that do not meet the benchmark scope.

Country benchmarks also define the network context being evaluated. For example, a mobile-network benchmark may exclude sessions running over Wi-Fi or roaming when the objective is specifically to assess experience delivered by the user’s own mobile network.

Once validated, measurements are processed into technical KPIs and service-level QoE indicators through the MedUX scoring framework.

From technical KPIs to Quality of Experience

Network performance cannot be fully understood through one indicator such as download speed.

Different digital services depend on different combinations of performance factors. Gaming is particularly sensitive to latency, jitter and packet loss. Streaming depends on successful playback, startup behaviour and delivered video quality. Web browsing depends on responsiveness and page-loading behaviour.

MedUX therefore transforms individual measurements into service-oriented experience scores.

Within each service, the Scorecard combines two types of indicators:

Qualifiers evaluate whether the basic conditions for a satisfactory session have been met.

Differentiators assess the quality delivered once those basic conditions are achieved.

Reliability is assessed and published separately and is not included in the weighting above.

The same Scorecard configuration is applied consistently to every eligible operator within the same benchmark.

These weights reflect the current MedUX Mobile Crowdsourcing Benchmarking Methodology 2026.1. If the Scorecard configuration changes materialise in future editions, MedUX will identify the applicable methodology version and assess whether historical comparison remains valid.

This hierarchy prioritises successful and consistent service delivery before rewarding additional performance. Very high values in one technical KPI therefore cannot automatically compensate for unsuccessful or inconsistent sessions.

Technical KPIs such as speed, latency or jitter can still be published alongside the QoE Score as valuable diagnostic information. However, the MedUX QoE Score is not calculated by simply ranking raw KPIs.

Geographical standardisation: from measurement volume to a national benchmark

Crowdsourcing naturally generates different volumes of measurements across a country.

Highly populated areas, specific applications or particular device populations may produce substantially more observations than other locations. A simple aggregation of every measurement would therefore allow heavily sampled locations to disproportionately influence a national result.

MedUX addresses this through population-standardised geographical aggregation.

Where measurement support is sufficient, observations are estimated at H3 resolution 7, corresponding to cells of approximately 5 km² on average.

Where a cell does not contain sufficient measurement support, the methodology does not force an estimate at that resolution. Instead, it progressively falls back to a broader geographical level — first H3 resolution 6 and, where necessary, a wider administrative geography. The official population and administrative references applied depend on the market: in Europe, for example, Eurostat Census 2021 population data and GISCO NUTS2 regional assignment.
 

The resulting geographical estimates are aggregated using an external official population reference, so locations contribute according to the target population structure rather than simply according to how many crowdsourcing tests happened to be generated there.

This approach reduces distortion caused by geographical over- or under-sampling while preserving as much local resolution as the available evidence supports.

Does population standardisation make the sample representative of the population?

No. This distinction is important.

Crowdsourcing measurements come from participating real devices and do not constitute a probability sample of randomly selected individuals from the national population.

Population standardisation addresses the geographical composition of the observed dataset: it prevents locations with more measurements from receiving disproportionate influence simply because they generated more data.

It does not transform voluntary device observations into a random population survey.

For this reason, MedUX Country Crowdsourcing Reports describe their results as population-standardised network-experience estimates rather than claiming probabilistic representativeness of all consumers.

This replaces the generic representativeness and fixed margin-of-error assumptions used in the previous version of MedUX’s crowdsourcing methodology. The previous public methodology described national results as statistically representative and referred generally to a 95% confidence level and ±5% margin of error; the current framework instead derives uncertainty from the actual benchmark estimator and sample structure.

Statistical inference: measurement volume is not the same as independent evidence

Crowdsourcing datasets can contain many measurements generated by the same user or device.

These observations are not fully independent: they may share device characteristics, regular locations, network conditions and usage patterns.

The current MedUX methodology therefore accounts for dependency between repeated observations when estimating statistical uncertainty.

The user or device is treated as the clustering or resampling unit, rather than assuming that every individual measurement is an independent statistical observation. This principle is explicitly incorporated into the current Country Crowdsourcing methodology.

This means that millions of tests can provide extensive measurement depth without being incorrectly interpreted as millions of independent users.

How MedUX determines statistical leadership

An operator having the highest numerical score does not automatically mean that the available evidence supports a differentiated leadership claim.

For categories in which statistical leadership is evaluated, MedUX analyses the difference between operators.

A statistically differentiated leader is identified only when the 95% confidence interval of its advantage over every relevant competing operator excludes zero.

If this condition is not met, MedUX does not force a statistically conclusive order between those operators.

This distinction separates a simple numerical ranking from a benchmark conclusion supported by statistical evidence.

The interval used for this decision refers to the difference between operators, not merely to the uncertainty interval around an individual operator score.

Operator eligibility

The presence of an operator in the underlying crowdsourcing dataset does not automatically guarantee inclusion in a Country Report.

Before publication, MedUX evaluates whether each operator provides sufficient evidence for competitive benchmarking during the reporting period.

Eligibility is based on predefined requirements covering areas such as data quality, geographical support and statistical sufficiency.

This assessment is specific to the crowdsourcing dataset and does not depend on participation in a separate drive-test or controlled measurement campaign.

The purpose is to ensure that competitive conclusions are based on evidence that supports a comparable level of analysis across the operators included.

Combining International Standards and Best Practices into a Unique Crowdsourcing-Based Scoring Methodology

The MedUX QoE Scorecard provides a common framework for translating large volumes of network and service measurements into understandable experience indicators.

The methodology prioritises the end-user perspective by analysing real commercial devices, real network conditions and real digital services.

Its principles include:

• end-user and service-centric measurement;
• end-to-end performance;
• real services and QoE KPIs;
• statistical robustness;
• comparable treatment of evaluated operators;
• technology and market relevance;
• transparent scope and methodological limitations.

MedUX’s benchmarking methodology takes into account good practices described in ETSI TR 103 559, together with other relevant industry recommendations and MedUX’s own network-testing experience.

The MedUX scoring model, geographical estimator, statistical framework and public QoE scale remain proprietary components of the methodology.

External standards should therefore be understood as methodological references and good-practice inputs, rather than as certification of the complete MedUX benchmarking methodology.

Leveraging True QoE KPIs from Crowdsourced Real-Field Data

MedUX crowdsourcing benchmarks focus on indicators that help explain how digital services behave from the end-user perspective.

Depending on the publication configuration, measurements can cover technologies from legacy mobile access through 4G and 5G and include the following experience dimensions:

Overall QoE Score

A synthetic view of the experience dimensions included in the applicable MedUX Scorecard.

Reliability

Evaluates the consistency with which network and service tasks are completed successfully.

Accessibility

Assesses the ability to initiate and access services with adequate responsiveness.

Value for Speed

Evaluates the experience delivered in relation to observed throughput performance.

Streaming

Assesses video experience using indicators related to session completion and delivered playback quality.

Data & OTT

Aggregates experience across services such as web browsing, social media, gaming and cloud applications.

Individual service results can also be analysed separately using the technical indicators most relevant to each use case. For example:

• Gaming: latency, jitter and packet loss.
• Web browsing: loading and responsiveness indicators.
• Streaming: successful playback, startup performance and delivered quality.
• Cloud services: successful completion and transfer performance.

The exact set of services and KPIs may evolve as technologies and digital usage patterns change.

Exploiting the MedUX QoE Score for Actionable Insights

The MedUX QoE Scorecard translates a large number of technical measurements into clear service and experience indicators that can be used across different areas of an organisation.

Country Reports publish the MedUX QoE Score on a 0–5 scale, calculated from the underlying Scorecard configuration. The scale is proprietary to MedUX and should not be confused with Mean Opinion Score (MOS) as defined by ITU-T.

The Scorecard makes it possible to move from an executive view of market performance to more detailed analysis by service, KPI, geography, technology or device segment where the available evidence allows.

For operators, this supports competitive intelligence, network optimisation and customer-experience management.

For regulators, it provides a framework for understanding market performance from the perspective of services experienced on real devices.

For the wider telecom ecosystem, it contributes to more transparent comparisons based on network and service evidence rather than on a single technical KPI.

Illustrating Scoring Principles across MedUX’s Global Benchmarking Publication Series

MedUX publishes Country Crowdsourcing Reports and related Mobile Network Experience analyses based on recurring market measurements.

The publication process follows the same fundamental sequence:

collect → validate → score → standardise geography → estimate uncertainty → publish conclusions supported by the evidence.

Country Reports typically evaluate the main eligible mobile operators within the reporting period and publish:

• the overall MedUX QoE Score;
• Reliability;
• Accessibility;
• Value for Speed;
• Data & OTT;
• Streaming;
• selected service-level results;
• relevant technical KPIs and technology indicators.

The MedUX QoE Score is published on a 0–5 scale.

A numerical first place becomes a statistically differentiated leadership result only where the statistical criterion described above is satisfied.

Where the evidence does not support an ordering, the report explicitly states that the difference is not statistically conclusive rather than assigning an unsupported ranking.

Methodology configurations can evolve over time as the Scorecard, technologies and market conditions change. When a change prevents homogeneous historical comparison, MedUX does not automatically present trends between incompatible editions.

Privacy and Data Security

User privacy is a fundamental consideration in MedUX crowdsourcing measurement and processing.

Measurements are collected under the applicable consent framework and are designed to analyse network and service performance rather than user content.

Published Country Report results are aggregated according to MedUX privacy and minimum-sample requirements.

MedUX applies technical and organisational safeguards throughout the measurement and data-processing lifecycle and operates in accordance with applicable data-protection requirements.

Frequently Asked Questions about MedUX Mobile Crowdsourcing Benchmarking

How does MedUX mobile crowdsourcing work?

MedUX collects active and passive network and service measurements from real devices, validates the observations, converts relevant KPIs into service-level QoE scores, geographically standardises results against an external population reference and applies statistical inference before publishing competitive conclusions.

Is MedUX crowdsourcing a representative survey of mobile users?

No. Crowdsourcing is not a probability sample of randomly selected individuals. Population standardisation reduces geographical composition bias but does not convert the dataset into a random population survey.

Why does MedUX geographically standardise crowdsourcing results?

Because different locations naturally produce different volumes of measurements. Population standardisation prevents heavily sampled areas from determining a national result simply because they generated more observations.

How does MedUX determine the mobile network leader?

A statistically differentiated leader is identified only when the 95% confidence interval of its advantage over every relevant competing operator excludes zero.

Why are measurements clustered by user or device?

Repeated tests from the same user or device are not fully independent. Treating the user/device as the statistical resampling unit provides a more realistic estimate of uncertainty.

Is the MedUX QoE Score the same as MOS?

No. The MedUX QoE Score is MedUX’s proprietary 0–5 benchmarking scale. It should not be confused with Mean Opinion Score as defined by ITU-T.

Does the operator with the highest download speed automatically have the highest QoE?

No. Different services depend on different performance dimensions, including successful completion, responsiveness, latency, jitter, stability and throughput. Speed is an important technical indicator, but it does not by itself determine overall mobile Quality of Experience.

Turning Mobile Network Measurements into Better Decisions

Crowdsourcing provides an opportunity to observe mobile network experience at a scale and diversity that complements traditional controlled measurement approaches.

The value of that data, however, depends on how it is processed and interpreted.

MedUX’s current methodology combines real-device measurement, service-oriented QoE scoring, geographical population standardisation and statistical inference to produce market comparisons whose scope and limitations are explicit.

This framework underpins MedUX Country Crowdsourcing Reports and supports operators, regulators and other stakeholders seeking an independent, data-driven view of mobile network experience.

Explore the latest Mobile Network Experience benchmarks in the MedUX Connectivity Observatory