We are publishing our first granular forecast view of broadband adoption. This offers a tool as opposed to an answer. It is intended to be used as a support for other analysis.
It presents a benchmark of adoption across the technology and ISP landscape of the UK at postcode level with quarterly intervals out to 2030 with a broad split by wholesaler.
For analysts it can provide a useful foundation for multi-level models of the UK market. For strategists it offers a best fit set of projections to allow a baseline/benchmark of the progression in the UK market and local variations. For marketers it can serve as a targeting tool and to shape timelines and a basic schedule of product focus and expectations.
Two key underlying components serve as the training datasets that drive the forecast models.
Subscriber numbers with granular timelines on ISP and technology adoption. At a national level. It is not possible to get reliable data at sub-footprint level from an operator or an ISP. Often it is difficult to get national level numbers for the larger organisations. Sky and Vodafone are particular examples of this. They report at European level usually.
Granular (postcode level) UK maps of deployments and availability of ISPs and technology available in an area again over time. These produce intersections of the various combination of ISP and technology in the UK. Effectively what you would see if you removed the physical geography dimension.
There are two sets. One for residential and one for retail business lines. A detailed methodology is available below.
Some notes and outcomes
We model the expansion of networks and footprints using an upgraded version of our availability forecast model. More detail is available in the methodology below.
The nationally projected forecasts of total take-up and market share is generated from a set of linear optimisation models which distribute total expected subscribers across postcodes according to the Intersection (combination of operators and technologies) the postcode falls under.
We do not allow for the consolidation of any of the named operators/networks. We assume that they are all going to exist in 2030. In the ‘other’ forecast pot the story will be different but we treat this pot as one operator and the expansion is based on the timelines to date.
This is the first version of the take-up forecasts and we will be adding more ISPs and refining the outputs at least twice a year from now on.
The model allows analysis at a granular level across 1.7M postcodes. You can also aggregate the data. This is the national level market share by quarter for residential subscribers.
Note that you need to exclude postcode with zero households if you’re doing your own analysis. The SQL query for this chart is at the end of this post.
You can also aggregate to a national level and separate out the technology and platform.
This model was built before the Sky announcement on reselling CityFibre. So while it’s possible to compare Openreach and CityFibre wholesale numbers we are, as ever, running to catch up with developments in the market.
More granular views are also straightforward to generate. This is BTs market share at Output Area level in 2030 in England and Wales.
As it is published at postcode level it is possible to aggregate to any geography above.
UK fixed broadband analysis platform
The take-up forecasts add to the suite of data, models and analysis that offer a framework for examining fixed broadband in the UK.
Detailed current and historic availability datasets. Who is where, when and with what?
Producing the intersections that are a core element in our postcode level take-up models
Country level as well as operator and ISP and technology specific data
Regulation and government are an important UK specific data and reporting
Tariffs, demographics, cost related analysis and other datasets and models can be directly connected and related.
The methodology below outlines our approach and an overview of the contents. A sample is available on request.
Two points to note:
The data has residential and business lines and market shares. One thing to be aware of is the postcodes without households should be excluded from residential analysis at risk of skewing the outcomes.
This model uses an updated version of our operator footprint forecast. As a consequence this will report slightly different outcomes depending on which quarter outputs are being compared. We expect, as of Q42024, to align the footprint forecast models for Q12025 outputs.
Take Up Forecast Methodology
1. High level overview
The primary objective of this project is to forecast how Internet Service Providers (ISPs) will allocate their lines across the UK at the postcode level for future quarters. In simpler terms, it aims to estimate the market share of each ISP in every postcode for specific upcoming periods. Moreover, the project goes beyond just giving the overall market share of ISPs in a postcode; It also attempts to provide insights on the proportion of technologies used by ISPs and the different types of lines within that postcode.
This model offers a benchmark and does not include local level (postcode) real world input on ISP subscriber numbers. It is representative of the outcome if most of the variables are equal. For example it does not take account of particular successes or failures in particular areas due to hitting a marketing sweet spot or well established, foundational client bases that are for whatever reason resistant to churn.
If, as an ISP or operator, your own data shows you above or below our estimates it is a sign that you are over/under performing against expectations and attention should be paid to how to replicate or address those differences.
The latest model outperforms its predecessor by employing improved methods like linear optimization and better utilization of input data to distribute ISP lines more effectively. It overcomes previous limitations, establishing a new standard for robust and efficient allocation of lines.
Project scope
a) ISP and technologies
As of now, this project includes 8 specific ISPs along with their respective technologies, with the rest combined into one ‘Other’ operator:
BT (FTTP/FTTC/ADSL)
Sky (FTTP/FTTC/ADSL)
TalkTalk – Openreach (FTTP/FTTC/ADSL)
TalkTalk – CityFibre (FTTP)
Gigaclear (FTTP)
Hyperoptic (FTTP)
Kcom Lightstream (FTTP)
Virgin (Cable)
Other
b) Type of line
There are 2 types of line included in this project, which are:
Residential
Business
c) Postcode
We will examine all 1.7 million postcodes, which are the total number of postcodes of the UK. In this project, we use the Point Topic UPC demographic and geographic dataset with annual updates to allow the correct balance between regularity and consistency.
d) Time Scope and Frequency
This project delivers the line distribution results for each future quarter, up to and including the 2030Q4 period.
This project runs on a quarterly basis, typically within 1.5 to 2 months after the end of each quarter, depending on the reporting cycles of the major operators.
Predict Methodology
Note: From this point, ISP also means the ISP with its technologies, like BT_FTTC or SKY_ADSL. For example: if BT exists in a postcode, that means at least one technology (amongst 3) of BT is available at that postcode.
To forecast the future line distribution at the quarterly level, the project applies the same distribution algorithm, adjusting the input data for each specific future quarter. Accordingly, this documentation will focus on the core of the project—the distribution algorithm.
This project uses 2 main data sources as the input for the predicting algorithm, which are:
Technology footprint of ISPs at postcode level
Number of business/residential lines of ISPs across the UK
To decrease the amount of computation and inference time, instead of distributing directly at postcode level, we cluster postcodes that have the same presence of ISP combinations into groups called intersections. In other words, an intersection is a group of postcodes having the same ISP’s existence in it. After calculating an estimation of line distribution results at intersection level, we will distribute down the number of lines into postcode level.
Therefore, the main computational flow of this project is:
Estimate the market share of ISPs and their technologies in each intersection
Distribute down the market share from intersection level into postcode level
In these 2 above steps, the first step is the most important and also the hardest part. The second project is just simply distributing the result of the first step into a more granular geographical level by assuming that the market share of an ISP in a postcode is similar to the market share of that ISP in the intersection which the postcode belongs to.
Distribute at intersection level
In one intersection, what we have is the number of lines of the intersection and which combination of ISPs exist in that intersection. Our objective is to find the number of lines that each ISP accounts for in that intersection (or the market share).
For example, suppose we need to distribute lines for an intersection that has 2 ISPs A and B in it. What we need to find is the number of lines of each ISP in this intersection, and we call these 2 variables that need to be found are: line_A and line_B.
We define this as an linear optimization problem, where the objective function that we want to minimize is:
In above formula:
• total line A, B: Total number of lines of ISP A, B across UK
• fill factor: factor to describe how hard to fill for an ISP for the algorithm (or the solver)
• n: exponent used to transform the fill factor to into exponential, in order to emphasize the effect of fill factor
We will examine each factor right below.
A. Total line
As we defined above, this factor is the total number of lines of ISP in the UK. This factor helps to ensure the results that the solver provides to us are reasonable in terms of the scale of the ISP. Regarding the scale of ISP, we mean that if the size (sum of lines) of ISP A is much larger than that of ISP B at a national level, then it’s likely that ISP A owns more lines when existing in the same intersection with ISP B.
For example: considering (total line A > total line B) , then if these 2 ISPs exist in the same intersection, ISP A is likely to have more lines than ISP B. And by defining this factor in the objective function, the solver attempts to make line_A larger than line_B in order to minimize the objective function.
B. Fill factor
A drawback arises when ISP A has a significantly larger line capacity (the total lines it can be potentially provided) compared to ISP B. The line capacity of an ISP is the sum of lines across all intersections where the ISP operates. Essentially, it represents the maximum number of lines an ISP could have in an ideal scenario (where the ISP captures 99.9999% of lines in all its intersections). Consequently, the algorithm encounters challenges when distributing lines for ISPs with lower line capacity, as it lacks the flexibility seen with ISPs having higher capacity.
When distributing lines for two ISPs, the algorithm tends to favor providing more lines to the ISP with more difficulty in achieving its total number of lines. To convey this information to the algorithm, we introduce a factor known as the fill factor, defined as:
The closer the fill factor of an ISP is to 1, the more challenging it is for the algorithm to fill lines for that ISP.
If this factor is less than 1, it indicates that it is impossible to fill lines for that ISP, even in the hypothetical scenario where the ISP owns all the lines in all the intersections it operates in.
If this factor is still ambiguous, we can come to a closer to real life example:
Consider two companies, A and B, each with its revenue target. Company A operates in an area where the market size is 10 times larger than A's target revenue, while company B's market size is only 3 times its target revenue. Consequently, it is much more challenging for company B to achieve its revenue target compared to company A.
In the context of our problem, the market size corresponds to the ISP's line capacity (the numerator of the fill factor formula), and the revenue target represents the total lines of the ISP (the denominator of the fill factor formula).
C. Exponent n
When certain ISPs have fill factors that are very close to 1, meaning it's extremely challenging for the algorithm to fill lines for them, we aim to amplify the impact of the fill factor. Currently, we achieve this by exponentiating the fill factors. For instance, if the fill factor of A is 20 and that of B is 1.5, considering the squared fill factors, the algorithm's inclination to make the lines of B much larger than A is approximately 400/2.25 ~ 177, which is significantly higher than the non-exponentialized ratio of 20/1.5 ~ 13.
The formula we use to make the fill factors exponential is:
In this formula:
• To guarantee logical consistency, we subtract 1 from the fill factor before applying the exponential emphasis. This ensures that when a fill factor reaches 1, the emphasized fill factor becomes 0, which is the lowest possible positive value.
• In practical terms, this means that an ISP with a fill factor of 1 will effectively claim all available lines at any intersection where it's present. However, this adjustment primarily serves logical purposes and doesn't significantly alter the overall results compared to simply using:
Therefore, it can be disregarded if desired.
A challenge arises in determining the appropriate exponent value. To address this, we examine the performance of various exponents across the three most recent quarters. The exponent that yields the best average performance across these quarters, as measured by the following criteria, is selected:
• ISP line error: This metric gauges the overall discrepancy in lines between the algorithm's filling results and the actual input data, specifically focusing on differences across individual ISPs.
• Intersection line error: Similarly, this metric assesses the extent of line discrepancies between the filling results and input data, but it concentrates on variations across different intersections.
• Count below 1: This metric tracks the frequency of filling results that fall below 1
• Count equal 1: This metric monitors the number of filling results that precisely equal 1
It's crucial to minimize the occurrence of filling results that are equal to or less than 1. This is because such values can signal a tendency of the algorithm to prioritize its own convenience over accuracy when allocating lines. By carefully monitoring these metrics, we can ensure that the algorithm remains unbiased and produces filling results that closely align with the actual input data.
D. Overall objective function
From the beginning, we aim to estimate values for line_A and line_B to minimize the objective function. By combining these 2 factors (total lines & fill factor), even if total line A is significantly larger than total line B, if the fill factor of A is greater than B, the algorithm adjusts the distribution so that line_A remains larger than line_B, but not excessively so as it would be without considering the fill factor.
By introducing these 2 above factors, the algorithm's robustness is enhanced significantly, as it now takes into account not only the total lines of ISPs but also considers the popularity or fill capacity of each ISP.
E. Postcode level
Once we've determined the number of lines each ISP has in each intersection, we can directly calculate their market share within those intersections. This market share then seamlessly translates to the corresponding postcodes. In other words, an ISP's market share within a postcode mirrors its market share in the intersection that the postcode falls under.
To illustrate this, consider the following example: If ISP A holds a 50% market share in intersection B, it logically follows that ISP A also claims 50% of the lines within each postcode situated within intersection B.
Because we already have the number of lines of each postcode, we can effortlessly ascertain the number of lines that each ISP possesses within each postcode.
F. Forecast future distribution
To predict the line distribution results for each future quarter, the project uses forecasted inputs specific to that quarter. These inputs are generated by various forecasting models, such as the footprint forecast at postcode level, lines forecast, and premises forecast. The project integrates the outputs from these models and transforms them into the two inputs required by the line distribution algorithm.
Conclusion
In conclusion, this project delivers a comprehensive approach to forecasting the distribution of ISP lines across the UK at the postcode level for future quarters. By estimating the market share of each ISP in upcoming periods, it offers valuable insights into the competitive dynamics of the UK broadband market. Additionally, by analyzing the expected technologies and line types used by ISPs within postcodes, the project provides a detailed understanding of the evolving local technological landscape. This future-focused analysis helps stakeholders and clients make data-driven decisions to optimize broadband access, enhance market competition, and improve overall consumer experience.
Field list
Both tables, business and residential, have the same structure and track the same set of ISPs and wholesale platforms.
POSTCODE | KCOM_LIGHTSTREAM_FTTP |
INTERSECTION_ID | VIRGIN_CABLE |
PREMISES | GIGACLEAR_FTTP |
HOUSEHOLDS | HYPEROPTIC_FTTP |
BUS_SITES_TOTAL | OTHER |
LINES | BT_MARKET_SHARE |
BT_FTTC | SKY_MARKET_SHARE |
BT_FTTP | TALKTALK_MARKET_SHARE |
BT_ADSL | KCOM_LIGHTSTREAM_MARKET_SHARE |
SKY_FTTC | VIRGIN_MARKET_SHARE |
SKY_FTTP | GIGACLEAR_MARKET_SHARE |
SKY_ADSL | HYPEROPTIC_MARKET_SHARE |
TALKTALK_OPR_FTTC | OTHER_MARKET_SHARE |
TALKTALK_OPR_FTTP | QUARTER |
TALKTALK_OPR_ADSL | REPORTED_AT |
TALKTALK_CF_FTTP |
Query samples
Market share by quarter, residential, national level
SELECT
T.QUARTER,
sum(T.households),
avg(T.BT_MARKET_SHARE) AS avg_bt_market_share,
avg(T.SKY_MARKET_SHARE) AS avg_sky_market_share,
avg(T.TALKTALK_MARKET_SHARE) AS avg_talktalk_market_share,
avg(T.KCOM_LIGHTSTREAM_MARKET_SHARE) AS avg_kcom_market_share,
avg(T.VIRGIN_MARKET_SHARE) AS avg_virgin_market_share,
avg(T.GIGACLEAR_MARKET_SHARE) AS avg_gigaclear_market_share,
avg(T.HYPEROPTIC_MARKET_SHARE) AS avg_hyperoptic_market_share,
avg(T.OTHER_MARKET_SHARE) AS avg_other_market_share
FROM
TAKE_UP_FORECAST.REPORT.RPT_POSTCODE_LINES_DISTRIBUTION_RESIDENTIAL T
WHERE
T.households>0
GROUP BY
QUARTER
ORDER BY
QUARTER;
Subscriber totals by ISP, technology and platform
SELECT
QUARTER,
SUM(BT_FTTC) AS TOTAL_BT_FTTC,
SUM(BT_FTTP) AS TOTAL_BT_FTTP,
SUM(BT_ADSL) AS TOTAL_BT_ADSL,
SUM(SKY_FTTC) AS TOTAL_SKY_FTTC,
SUM(SKY_FTTP) AS TOTAL_SKY_FTTP,
SUM(SKY_ADSL) AS TOTAL_SKY_ADSL,
SUM(TALKTALK_OPR_FTTC) AS TOTAL_TALKTALK_OPR_FTTC,
SUM(TALKTALK_OPR_FTTP) AS TOTAL_TALKTALK_OPR_FTTP,
SUM(TALKTALK_OPR_ADSL) AS TOTAL_TALKTALK_OPR_ADSL,
SUM(TALKTALK_CF_FTTP) AS TOTAL_TALKTALK_CF_FTTP,
SUM(KCOM_LIGHTSTREAM_FTTP) AS TOTAL_KCOM_LIGHTSTREAM_FTTP,
SUM(VIRGIN_CABLE) AS TOTAL_VIRGIN_CABLE,
SUM(GIGACLEAR_FTTP) AS TOTAL_GIGACLEAR_FTTP,
SUM(HYPEROPTIC_FTTP) AS TOTAL_HYPEROPTIC_FTTP,
SUM(OTHER) AS TOTAL_OTHER
FROM TAKE_UP_FORECAST.REPORT.RPT_POSTCODE_LINES_DISTRIBUTION_RESIDENTIAL
GROUP BY QUARTER
ORDER BY QUARTER;
Comentários