Egress switch broadband data from the Pryor Public Library in Oklahoma
Since our last post, the daily life and work of most everyone on the planet has been altered. Our project’s response to the COVID-19 global pandemic was to pause on all external activities with partner libraries. We suspended shipment of measurement computers to new libraries, and have shifted to documentation and data analysis, along with ongoing development of the measurement system’s data visualization service.
Though our program staff meet regularly and coordinate our work online, we’re waiting on advice from expert public health professionals on when it may be safe to return to working together in person before resuming work with partners at libraries around the country. We definitely don’t think that any libraries should be open right now! Our program work has also of course been delayed by additional responsibilities of helping to moving all of our institutions’ work, classes, and communications online, all while caring for those around us and staying safe ourselves.
Broadband Data Analysis
The good news is that our team is continuing to work on the measurement system’s data visualization service, and we are beginning to analyze data from our first year libraries. In late 2019 we did an initial export of the data collected by our devices in year 1 libraries to begin some initial data analysis. As our first year libraries were upgraded early this year, our archive of year 1 data was updated as the year 1 devices were returned. In January and February, we began exploring this initial dataset looking at a six month period between April and September 2019. We’d like to thank Dr. Brian Whitacre at Oklahoma State University for developing the charts for our initial data analysis.
Our analysis is in a very early stage at this point, but one area worth mentioning that we are interested in exploring is the comparison between different tests that measure a connection’s performance, but use different platform and test methods.
M-Lab has an FAQ article describing the differences between the Network Diagnostic Tool (NDT) and other popular speed test platforms, and also published a blog post addressing the differences in what is meant by “Internet Speed” and claimed accuracy or inaccuracy of NDT or other speed tests. Long story, short- there isn’t agreement about what “Internet Speed” means, much less one way to measure it. To quote the M-Lab blog post:
“Though the US Federal Communications Commission has set an official definition “broadband” (i.e., 25 Mbps download & 3 Mbps upload speed), to our knowledge no agency has set standards for how those benchmarks ought to be verified or measured.”
Because of the variety in testing platforms and test methodologies, in this research we wanted to run multiple types of tests in order to study the differences. The devices in our partner libraries are running NDT and speedtest-cli, a community developed program that tests to the speedtest.net platform. OOKLA also provides a command line client test, but we used speedtest-cli instead due to limitations within the terms of use for the official client. Though the developer of speedtest-cli acknowledges some inconsistencies between their client and OOKLA’s official client, in our initial analysis we’re seeing relative parity between the measured speeds of NDT and speedtest-cli, as seen in the charts below.
Library Broadband Data from Oklahoma and Oregon
Here we’re looking at data from the Pryor Public Library in Pryor, Oklahoma and the St. John’s branch library in Portland, Oregon, part of the Multnomah County library system. In both cases we’re looking at the download speed in megabits per second, aggregated by the hour of the day, for tests conducted between April and September, 2019. Each dot represents a single test and is placed on the x-axis of the chart at the hour of the day it was conducted and on the y-axis at the speed measured.
First we’ll see two charts below showing measurement data conducted from the egress switch at the Pryor Public Library, a small independent rural library in Oklahoma. Pryor Library purchases their internet service through a regional research and education network, OneNet.
Pryor Public Library’s advertised download speed is 100 Mbit/s, and the clusters of tests along the top of the chart as well as the trend line across the top (showing median speed across the day) confirm that the library is receiving download service at the advertised rate. Further, both NDT and speedtest-cli data confirm this with similar measurements of performance, with slightly more spread in measurement speeds from the NDT test.
Our second example takes measurements from the St. John’s branch library in the Multnomah County library system in Portland, Oregon, a large urban library system. Multnomah County library system is itself a network operator, meaning that instead of buying internet service from available providers that would serve businesses or homes, a regional education network like Pryor, or a city or state run network, Multnomah County has the scale to purchase connectivity directly from transit providers. Their advertised speeds are 400Mbps (up and down), and our test data confirm that they are receiving it across the hours of the day.
It’s encouraging to see relative parity in the measurements of both NDT and speedtest-cli in this initial analysis. In our second year data we expect to see even more interesting comparisons between these datasets, and also data from additional test versions. Beginning with this year’s rollout of new measurement devices, we’ve started running both single TCP stream and multiple TCP stream speedtest-cli tests, as well as running both the original NDT test and M-Lab’s newest protocol for NDT, ndt7.
We’ll be continuing to analyze the data and report our findings, but we hope this post provides an initial look at what we’re measuring and provides some provenance for the data itself. We’re also exited about exploring the measurements of multiple performance measurement tests, and are hopeful this research will improve public understanding of internet measurement in general.
