We’re now in the full swing of site visits to our year 1 libraries, visiting Pasco County, FL; Pryor, OK; and Twin Falls, ID over the last three weeks. At each of the ten year 1 libraries, we’re interviewing staff and patrons, getting to know the context and character of our partner libraries’ communities, and gaining a good sense of what it’s like to work at and be a patron of each library. Another important goal related to the installation of the measurement devices is to learn about the library networks’ nuances at each location, refining our device configurations and setup instructions as needed. Our end goal is to uncover most potential barriers to the measurement devices working properly in our year 2 libraries’ networks, where we won’t have the luxury of being there in person.
This iterative work on the measurement system goes back to the pilot research in Alexandria, VA that preceded this program, where findings included a list of technical challenges that would need to be addressed in order to scale up similar initiatives. Among those needs was a service or system to install, manage, remotely administer, and update many small computers placed within managed networks. In Alexandria, installing the operating system and measurement software on each device was manual, and we worked with school IT staff to provide basic remote access. IT staff also helped setup firewall exceptions to allow the Network Diagnostic Tool (NDT) to successfully run, since the Internet ports it currently uses are often blocked in managed networks. While this was fine for a small scale pilot project, it would be untenable in a larger scale research study like this one.
Device Setup and Management
Fortunately we were able to address many of these challenges prior to this program. To provide scalable device setup and management of our measurement devices, we selected Balena Cloud, an “Internet of Things” (IoT) virtualization and device management software as a service (SaaS) provider with a commitment to open source tools.
Balena allows us to setup and pre-configure a wide array of single board computers, register them in a project connected to our code, provide remote access to them, and enables the ability to push updates to an entire fleet of devices. Balena has also recently released an open source server edition, Open Balena, that should provide a means for self-hosting the tools used in their supported Balena Cloud service. These options fit well with other open source services used by the library community such as LibLime’s support for the open source Integrated Library System, Koha. For this research, we’ve subscribed to the Balena Cloud service, but will also test and document use of Open Balena.
We wanted our measurement system to be easy to set up and manage, and using Balena has made that possible. But also knew that we would need to coordinate with both our year 1 library staff and administrators, as well as with their IT support staff and possibly with their system vendors for a couple of reasons. Depending on the library’s network management practices, firewall exceptions to allow our measurement devices to communicate with Balena services would need to be added. Also, the ports currently used by the NDT test are sometimes blocked by firewalls. However, we expect that by the time we’re working with year 2 libraries, the M-Lab development team will have completed a new release of NDT server and reference client code, which will remove the port requirements for the NDT test. NDT7 will also use the latest TCP compression algorithm, BBR, which stands for Bandwidth, Bottleneck, and Round Trip Time. BBR will enable NDT7 to estimate the end-to-end performance of a connection in approximately 2 seconds and use far less data than the current version, which will enable more accurate measurement of all connections, but in particular satellite and mobile phone Internet service.
The Role of M-Lab Tests and Servers
M-Lab servers through which tests are conducted are hosted within Internet exchange points, or IXPs, and have built in redundancy and testing infrastructure. Each M-Lab server “pod” consists of four servers and a high capacity switch, connected to a single transit provider with an uplink speed between 1 Gbps and 10Gbps.
Currently, M-Lab is in the process of upgrading all US servers to 10Gbps links. In large metro areas, M-Lab hosts between 4 and 8 server pods, each connected to different providers. Three of the servers in each pod are used for production traffic, and the fourth is used for testing. As a part of a large project to update the software on the M-Lab platform, the M-Lab team has deployed the NDT7 server to its testing servers, and an early release of the NDT7 client, libndt, is available to test against these M-Lab beta test servers. During this program, we are initially using the current version of NDT, but will also use the libndt client and M-Lab’s test NDT7 servers, particularly in our libraries that may have very restrictive firewalls or have satellite connections to the Internet.
Developing the Training Manual
To scale up to 50+ libraries in our second year we need to have a very clear setup guide to send to partner libraries along with pre-configured measurement devices, which includes both the requirements for IT staff to connect each device, as well as a list of potential nuances that might be present in specific library settings.
Prior to each year 1 library visit, when emailing to coordinate times, travel, schedules, etc. we’re sending a document that describes what will be installed at their library, and asks librarian and/or IT staff to provide key information needed to make the installation go as smoothly as possible. Some of the questions from this document include:
- If your library’s public WiFi network requires a patron to complete a consent for, acceptable use agreement page, etc. can this be disabled for our measurement devices?
- What is the name (SSID) of your public WiFi network?
- If a password for WiFi is required, what is the password?
- If you need our measurement devices to be configured with static IP addresses, please provide:
- The three IP addresses our devices should use (2 wired, one WiFi)
- The subnet size of each IP address
- The IP address of the gateway
- Does your library wish to add more measurement devices, funded from your budgets? If yes, how many devices are you considering adding?
As we visit each of our year 1 libraries, we are refining this document based on our experiences with the installation.This document will help produce the complete setup guide that will be sent to our year 2 participating libraries.
Lessons Learned
We’ve learned a number of important things in our first four site visits related both to the pre-visit preparation library and their IT staff have done, and nuances related to how libraries IT staff or support implement our system requirements. Some of those nuances include:
- Firewall exceptions/rules are implemented in different ways by library IT staff or service providers, often based on the infrastructure they use, or the vendor system being used.
- Implications for our documentation:
- Supporting static IP addresses configured on each device may be necessary
- Providing the MAC address of each device’s network interfaces will be needed in some libraries
- Adding exceptions for our WiFi devices to avoid captive portals and for tests to run correctly could prove difficult to manage, depending on the WiFi system, vendor product, or vendor support.
- Implications for our documentation:
- Determining an easy way for library staff and IT to test that firewall rules/exceptions provide the right access for our system may be needed.
- WiFi configurations on our devices need an explicit configuration to prioritize the WiFi interface instead of the ethernet interface.
These experiences have emphasized for our team the critical importance of the close relationships we are forming with our year 1 libraries and the need for site visits to uncover as many potential scaling issues as possible. We’re incredibly grateful for the support and contributions of our partner libraries throughout this process.
