Building a Framework for Standardized Wi-Fi Performance Testing

When service providers are selecting Wi-Fi equipment for residential broadband deployments, you might think that evaluating device performance would be an obvious first step. It sounds straightforward enough: apply an objective, vendor-agnostic testing plan and make apples-to-apples comparisons. But this is one of those technology areas that sounds simple in theory but is actually quite complex. In fact, it’s challenged the industry for years.

There are, of course, tools that vendors and customers use to measure aspects of wireless performance. But for most of the history of Wi-Fi, standardized performance testing largely didn’t exist. Standards organizations offered highly effective testing frameworks to validate conformance with wireless standards and regulatory requirements governing RF airspace. Indeed, the massive success of Wi-Fi stems in no small part from testing regimes like the Wi-Fi Alliance, which gives customers confidence that any Wi-Fi-certified device will interoperate with any other in multi-vendor environments. When it comes to evaluating device performance, however, standards bodies have remained largely silent.

Now, that’s beginning to change. We recently published a white paper, The Importance of Performance Testing for Residential Wi-Fi Deployment detailing these efforts. The paper describes the challenges that have impeded standardized Wi-Fi performance testing and the evolving industry efforts to overcome them. The good news: while new technical hurdles continue to emerge, we finally have a solid foundation for standardized performance testing for diverse Wi-Fi use cases.

The Wi-Fi Performance Testing Challenge

Knowing that a certified device will interoperate in multi-vendor environments is obviously important. But for service providers planning large-scale residential Wi-Fi deployments, validating conformance and interoperability is just the start. They need to understand how different devices actually perform in subscriber households. After all, for the vast majority of residential customers (92% in the U.S. alone), Wi-Fi is a primary means of consuming broadband. The quality of experience that subscribers attribute to their provider increasingly depends on the performance of their Wi-Fi router or gateway—performance that service providers must be able to measure and understand.

Unfortunately, evaluating the performance of Wi-Fi equipment has historically been a difficult, expensive task, for multiple reasons:

• It’s extremely difficult to achieve consistent, repeatable results. Wireless devices are highly sensitive to their physical environment and other devices sharing their airspace. Since every space is different, and RF environments continually change, it’s very hard to test for realistic conditions in a consistent way.

• Wireless testing requires significant time, effort, and expertise. Historically, the only reliable way to compare performance was to physically walk through a venue with sophisticated testing equipment. That’s just not an efficient option for providers serving thousands or millions of residential customers.

• There is no universal understanding of what “good performance” means. It’s not just operating environments that affect performance. What constitutes “good” for a given device varies widely depending on a complex mix of factors, including its supported features, throughput, range, price, and especially, intended application. For example, a product that delivers excellent price/features/performance for a small home environment might not fare as well in a large, crowded, interference-prone office environment, and vice versa.

Inside Wi-Fi Performance Testing

Given the many variables involved in gauging performance, vendors have been reluctant to adopt any framework that might give the impression that one product is objectively “better” than another. Instead, they’ve advocated a “fit-for-purpose” approach that would allow customers to compare different devices’ suitability to specific environments and applications. The growing acceptance of this approach, along with ongoing advances in testing technologies, has laid the groundwork for the standardized Wi-Fi performance testing that service providers have been clamoring for.

Today, multiple industry groups have issued standardized Wi-Fi performance testing plans, each geared towards different purposes. These include:

• Broadband Forum (BBF): BBF TR-398 was the first residential Wi-Fi test plan to include performance test cases and methodology, as well as pass/fail criteria for each test. True to the fit-for-purpose model, the BBF’s Wi-Fi In-Premises Performance Testing plan focuses exclusively on residential Wi-Fi access points (APs), and exclusively in-premises performance. The test plan is geared towards the relatively simple single-AP environments that dominate home deployments. It covers RF capability, coverage, stability, and performance, both at baseline and with multiple connected clients.

• Wi-Fi Alliance’s Wi-Fi Customer Experience Group: The Wi-Fi Device Metrics Test Plan is similar to BBF TR-398 but with a different focus. Here, the goal is not to provide pass/fail results for a given device, but to enable consistent statistical analysis so that service providers can compare devices for a specific purpose. Test cases include rate vs. range, AP latency, channel switching, roaming, and other scenarios.

• European Telecommunications Standards Institute (ETSI): The ETSI Broadband Radio Access Network (BRAN) Multiple Access Points Performance Testing Plan (specification TS 103 754) is designed exclusively for multi-AP environments, such as Wi-Fi mesh or extender scenarios. It covers roaming time and throughput, one- and two-hop throughput, band steering, and network configuration and self-healing.

Together, the new testing standards give service providers powerful new tools to plan and support residential Wi-Fi deployments. For the first time, they can evaluate Wi-Fi performance for specific use cases in meaningful ways, using consistent, in-depth testing plans from industry groups they trust.

Of course, this doesn’t mean that the performance testing challenge is “solved.” Wi-Fi is a constantly evolving technology, and new features are developed all the time. For example, performance testing specifications don’t yet address new Wi-Fi 6 and 7 features such as advanced power-saving mechanisms, multi-link operation, and others that add even more complexity. Just as important, Wi-Fi performance is becoming mission-critical for more and more applications. For augmented reality, smart home connectivity, and many others, the ability to maintain consistent latency, throughput, coverage, and other performance metrics will be essential.

Learn More

Want to know more about how the Wi-Fi testing landscape is changing, and how industry leaders are attacking the performance testing challenges of the future? Download the white paper.