DT, 1&1, Vodafone and O2: Internet outage in Germany
SERVICE OUTAGE ANALYSIS
DT, 1&1, Vodafone and O2: Internet outage in Germany
- Operators impacted: Deutsche Telekom (DT), Vodafone, 1&1 and O2
- Date: February 12, 2020
- Duration: approximately 7 hours, mostly between 2 AM and 9 AM (CET)
- Access technologies affected: ADSL and VDSL
- Potential impact: 25 million DSL lines
- Most impacted regions: Nordrhein-Westfalen, Rheinland-Pfalz, Hessen, Mecklenburg-Vorpommern, Berlin, Brandenburg and Munich.
On February 12, many Internet users in Germany woke up frustrated as a significant outage disrupted Internet service across several regions of the country and prevented them from getting online. Some national newspapers reported on the Internet issue, and complaints to the four operators filled the Internet. According to the German newspaper Morgenpost, 78% of Telekom and O2 customers reported Internet problems, as did 82% of 1&1 and 65% of Vodafone users.
German provider Deutsche Telekom (DT) worked hard to resolve the issue after the Internet outage hit many clients, starting in the early morning hours. The company stated that a software update at a technical facility caused the outage. Other ISPs were impacted mainly due to their use of DT’s xDSL wholesale services. Vodafone’s and Unity’s cable networks, as well as other fiber-based or independent networks did not suffer an internet outage.
Check out our impact assessment and the value of MedUX solutions for monitoring, analysing and troubleshooting network issues in real-time by delivering end-to-end performance and Customer Experience insights based on objective network measurements and not exclusively on users’ reports and complaints. MedUX ecosystem helps our clients reduce the time to insight by obtaining meaningful information in real-time about impacted customer services. Furthermore, it reduces the time to resolution by collecting detailed statistics about network performance and consequently responding quickly to customer issues and improving the Customer Experience.
How is German Broadband market?
According to BNetzA -The Federal Network Agency (Bundesnetzagentur in German) is the German regulatory office for telecommunications, electricity, gas, post and railway markets- statistics, the number of broadband connections in Germany increased by 0.4 million in the first half of 2019, bringing the total to around 34.6 million by mid-2019.
All non-DSL technologies accounted for approximately 9.4 million connections. Most of these were based on HFC networks (around 8.1 million), while approximately 1.2 million were based on fibre-to-the-building (FTTB) or fibre-to-the-home (FTTH).
DT’s competitors market some DSL connections to customers on the basis of specific wholesale products and alternative carriers. Out of approximately 25 million operational DSL connections at the end of June 2019, around 7 million lines were offered by DT’s competitors based on bitstream and resale wholesale reference offers.
Figure 1: DSL connections in Germany
VDSL accounted for a share of around 56% of all DSL connections by mid-2019. Around 6.3 million VDSL connections were provided by DT’s competitors and around 7.7 million direct VDSL connections by DT. The rise in the spread of VDSL is due mainly to vectoring technology, which currently enables transmission rates of up to 250 Mbps. The increasing significance of VDSL was also reflected at the wholesale level.
The above information indicates the potential size of this DSL service degradation or outage and the degree of dependency on DT for DSL service provision for Vodafone, 1&1 and O2, among others.
Network outage overview
MedUX observed a service outage during the morning hours, mostly between 2 and 9 AM, on February 12. This service degradation varied across technologies, operators and regions. The lack of Internet connectivity significantly affected Customer Experience. Service availability was at the lowest level between 3 and 4 AM and gradually recovered afterwards.
All operators were affected in some way, but DT and O2 apparently had less severe degradation in VDSL services in relative terms, and 1&1 ADSL services suffered no relevant degradation according to our estimates.
Figure 2: Network stability by operator, technology access and time of day
Network service outage in all xDSL operators affected the network and Customer Experience in Germany, especially in the regions of Nordrhein-Westfalen, Rheinland-Pfalz, Hessen, Mecklenburg-Vorpommern, Berlin, Brandenburg and Munich, as shown in Figure 2.
Figure 3: Estimated xDSL outage at peak time
For the most affected regions, service disruption reached its peak between 3 and 4 AM, with network accessibility ratios around 80% on average. Naturally, the majority of the users were not heavily affected, as the network accessibility recovered to over 90% after 8 AM.
Internet service monitoring and troubleshooting
MedUX monitors Over-The-Top (OTT) applications, including YouTube, Web Browsing and Dropbox. In addition, the MedUX solutions’ analytical capabilities enable clients to locate and resolve incidents affecting service quality and Customer Experience.
As can be seen in Figure 3 below, this outage affected most basic Internet services. However, the extent of the outage depended mainly on the operator and service region. All basic activities were affected across the board at a similar level because of the lack of Internet connectivity.
Figure 4: Internet service performance
For the most affected regions, service disruption reached its peak between 3 and 4 AM, with network accessibility ratios around 80% on average (see Figure 4 below). According to our data, service availability was even below 60% in certain regions during peak outage time, which is considerably higher than national average values.
MedUX insights pointed to a lack of connectivity as the root cause for this service outage. This finding was later confirmed by public statements from the ISPs’ customer care teams.
Figure 5: Network accessibility in heavily affected regions (shown above)
By collecting millions of data analytics, MedUX technology helps ISPs better understand their customers and monitor service quality. MedUX gathers 24/7 information and statistics about perceived end-user’s experience, which is eventually affected by the frequency, duration and severity of network events. In this sense, user reports and complaints alone or simple high-level monitorization is not enough to resolve and prevent customer experience issues.
Our insights help our clients reduce the time to insight by obtaining meaningful information on end-to-end network performance and on impacted customer services. Furthermore, our insights help our clients reduce the time to resolution by collecting detailed End-to-End performance statistics in real-time and consequently responding to customer issues in a timely manner.
Our controlled and dedicated technology offers extended root cause analysis and powerful investigation capabilities for network events, linked to the service/application layer as well as to other network layers. During our intensive QoE test protocol, we collect valuable information about performance limiting factors that helps detect, isolate and determine the root causes. It includes basic path quality information such as throughput, latencies and packet loss, as well as other service level information associated to web browsing, streaming and cloud storage, among other. Examples of these performance limitations available to most monitored service or applications relate to connection time outs, DNS resolution, destination host connectivity, network connectivity or server errors.
MedUX analyses network performance based on real-time information and results obtained from 5.000 MedUX HOME devices deployed in eight countries in Europe, from which over 800 measurement points evaluate German networks. These devices allowed MedUX to measure, in real time, the broadband experience in the customers’ premises.DOWNLOAD PDF REPORT
Technical problems can happen, but ISPs should be prepared to detect, solve and anticipate them.
Fastly brings down Amazon, Twitter, Twitch and many more: MedUX impact assessment Fastly CDNs went down between 12 PM and
YouTube goes down: MedUX impact assessment YouTube went down between midnight and 2AM UTC on Thursday, November 12th, 2020, which
Any need related to Quality and Customer Experience? We are here to help you!