REC Overview and Root Cause Analysis of the 2023 LPFM Window LMS Vulnerability Incident

Sat, 12/09/2023 - 11:43

Table of Contents

Introduction

During an FCC filing window for new broadcast construction permits, such as the 2021 Noncommercial Educational (NCE) FM filing window and the current 2023 Third Generation Low Power FM (LPFM) filing window, the FCC suppresses the applications in the License Management System (LMS) from being publicly viewed until after the close of the window.  This is in order to assure the fairness of the window as all applications filed in the window are considered as simultaneously filed.

A recent discussion on a Facebook group related to Low Power FM (LPFM) had expressed concern over what was being called by some as a “data breach”.  Specifically, it was reported that it was possible to view LPFM applications that were filed during the window period prior to the close of the window period. 

The issue was reported on Thursday, December 7 from another member of the LPFM advocacy community to FCC staff.  The report was apparently taken seriously by FCC staff, which prompted FCC staff to announce an unscheduled outage of the LMS system on the same day from 6PM to 8PM Eastern Standard Time. 

REC reached out to the community member who discovered the issue to get more detailed information.  Based on the information provided, REC has analyzed the situation and we present our findings here.  First, we must explain how LMS works in connection with the key elements of the incident and how the assignment of Facility IDs played a role.

How LMS handles applications

Normally, when a (non-window) broadcast application is filed in LMS, the information about that application is made instantly available to the public. This availability is presented through the LMS Application Search function available on the LMS Public Search page. 

Prior to the FCC using LMS for broadcast (radio and television) applications, the FCC utilized a 1998 vintage system called the Centralized Database System (CDBS).  CDBS would accept applications but not publish them the same day.  Instead, at midnight Eastern Time, CDBS would run a batch program that would publish the applications and application details received from the previous day.

When filing windows were handled by CDBS, the system was programmed to suppress the window related applications from the daily batch run.  Applications were made public at a time when staff ran a program to release them to the public after the close of the window.

Understanding Facility IDs

For broadcast services, the FCC uses a facility ID number.  The facility ID is a simple integer (all numeric) that distinctly identifies a broadcast facility.  When the FCC converted from CDBS to LMS, the same facility IDs were used.  The facility ID allows for the static identification of the broadcast facility even if the broadcast station’s call sign changes.  The facility ID is also utilized outside of the FCC, including as a part of the technical standard used for HD Radio. 

When the FCC started converting services to LMS, it would start assigning new Facility IDs with 6-digit numbers starting with “7”.  These Facility ID numbers are sequentially assigned. There are many different types of requests that are sent through LMS that will trigger a new Facility ID number.  These include:

  • Original construction permit applications for new FM booster stations,
  • Original construction permit applications for new Alaska Class D stations,
  • Original construction permit applications associated with a filing window (such as the current LPFM window).
  • Original construction permit applications for commercial services associated with auctions.
  • Applications associated with rulemaking requests to amend the FM or TV Table of Allotments (to add new commercial allocations).
  • Applications associated with experimental Special Temporary Authority operations (such as the Live Sports Radio LPTV requests).
  • New and changed broadcast facilities outside of the United States reported to the FCC’s International Affairs Office (formerly International Bureau) to assure protections by US stations (this is not entered through a publicly available application).
  • Every ownership report (biennial and non-biennial) filed in LMS.  Facility IDs assigned for ownership reports are used internally within the FCC database and are not normally exposed to the public like with other non-window uses of new Facility IDs.

When these types of applications are started in LMS, LMS will assign a Facility ID number.  As mentioned, each Facility ID number is sequentially assigned from the same pool regardless of the service type shown above.  Therefore, it is possible that sequential Facility ID numbers will be for different services. 

The FCC turned on the current form for LPFM construction permits in mid-June, 2023.  From this time on, applicants could start entering their data into LMS, but they would not be able to file the application until the window dates.  This is a process we call “staging” the application.  Every eligible application was assigned a sequential Facility ID at the time that the form was originally requested.  If the application was deleted before filing, the Facility ID would not be reused. 

We must also point out that the FCC held their biennial filing window for commercial Ownership Reports in the months of October and November.  The ownership report filings consumed thousands of sequential Facility ID numbers. 

Between the time that the FCC has made the LPFM Construction Permit form available and Saturday, December 9, LMS has assigned over 10,000 new facility ID numbers.  In contrast, REC Networks estimates that on the first three days of the LPFM filing window, approximately 617 LPFM applications have been filed.

About the incident

While we will not describe exactly how it was done, we will state that it involves the Facility ID and LMS functionality that is used, but not directly entered by the general public.  Using this vulnerability, someone knowing the Facility ID of one of the new LPFM facilities would have been able to view a screen which would have linked them to the application, even though the application is supposed to be suppressed until after the close of the filing window.  This would have given someone a full view of the application. 

On Thursday, December 7, after receiving notification from the member of the LPFM community regarding this vulnerability, the FCC swiftly placed an announcement in LMS stating that the system would be unavailable from 6PM to 8PM that evening. 

The patch that the FCC put in place not only shored up the vulnerability, but had also blocked certain information about existing LPFM facilities from being viewed in the LMS Facility Search functionality.  It did not impact the LMS Application Search functionality.  This patch did resolve the vulnerability.

REC’s findings and conclusions about the incident

Following the installation of the patch by FCC staff and after receiving a full briefing of the incident from the person who discovered the issue, REC did a small amount of post-fix testing and has reached the following determinations:

  • The vulnerability only exposed applications that were actually filed.  Meaning that applicant has completed all sections of the application, placed an electronic signature on the certification page and the application was assigned a sequential file number (the 10-digit number, beginning with 0000).
  • Applications that were still saved, but not yet filed, were not at risk of exposure.
  • To view an exposed application, knowledge of the specific LPFM Facility IDs would have had to have been known.  Out of the last 10,000 sequentially-assigned Facility IDs assigned by LMS since mid-June, a large number of them were provided to ownership reports.  Normally, the FCC would make these ownership report Facility ID numbers readily available in the FCC’s daily raw data “dump” file, which is used by REC and others to provide broadcast data.  However, because of the filing window, the FCC stopped updating the Facility table in the daily dump starting in June.  Those ownership report facility IDs can be found in a different table, but they are not in the Facility table.
  • For someone to view the exposed data, they would have needed an intermediate knowledge of performing HTTP GET transactions on a web server and knowing the appropriate parameters to use.  Thus, meaning that the vulnerability could not be achieved by simply going to your LMS bookmark, clicking a link to a search form and filing out the form and then clicking on a link.  It would have required bypassing the normal process by manipulating the URL.
  • Since we have determined that only filed applications (with file numbers) were vulnerable, no application was exposed for more than 42 hours.  The application would have been vulnerable from the time when the applicant or their consultant entered the applicant’s electronic signature information, clicked [Certify] and LMS returned an acknowledgement of the filing with a File Number.  All filed applications were no longer vulnerable after Thursday, December 7, at approximately 6PM when the FCC stopped LMS services in order to install the emergency patch.

The vulnerability was concerning to LPFM interests as if someone had the knowledge, and the time to send multiple HTTP GET requests to the LMS server, either manually or automated, they could get the application information of those who had filed in the window thus giving a bad actor the intelligence needed to strategically place applications that could “torpedo” another application.  In the same way, it could also be used to avoid mutual exclusivity.  Either way, there is an expectation of applicant privacy during the window, and this vulnerability breached that expectation. 

REC is going to estimate that the incident had made approximately 600 filed LPFM applications vulnerable to unauthorized viewing.

Since the incident involved the use of HTTP GET transactions, it is very possible that FCC staff can examine their web server logs to determine if any transactions, other than those used to troubleshoot the issue were made and how many applications may have been exposed to parties other than those involved in the discovery of the incident, if any at all. 

REC’s root cause analysis and opinion

Overall, despite some functionality quirks, LMS is by far, a more superior filing system than its CDBS predecessor.  The user interface for filing applications which departed from the CDBS method, which presented the application in a way that emulates its paper form predecessor is much more user friendly.  To use an analogy, using LMS was like using a tax preparation site, such as H&R Block for filling out an IRS Form 1040 versus filling out an actual IRS Form 1040, similar to the CDBS experience. 

Since most broadcast applications are filed on a “first come, first served” basis, the ability to instantly see application data once it is filed does have its advantages and is the main driver for REC systems, such as eLMS, which is used to drive websites such as FCC.today and FCCdata.org.  With that instant availability of applications also comes the risk that applications can be “torpedoed” by a competing application filed the same day.  Existing regulations, which go back to the paper form days treat most applications filed the same day as “simultaneously filed” and therefore mutually exclusive.  This can cause applications filed on the same day to offset each other, making neither application grantable.

LMS was first introduced for television in the mid-2010s.  It was not used for FM broadcast engineering until 2019.  The concept of using LMS for filing windows appears to have been a mere afterthought.  To add the functionality to do filing windows in the system, the IT developers and their business partners should have identified every aspect of the system that needed modification in order to accommodate a fair filing window and include those within the IT requirements for implementing the software release that would implement filing window functionality in LMS.  Based on the nature of the incident, not every vulnerability was identified up front.  Only those related to suppressing the applications from displaying on the Application Search were properly implemented.

This latest incident further underscores REC’s position, which has been communicated many times in the past to FCC staff that any major system release should be preceded by a level of external User Acceptance Testing (UAT) where stakeholders in the business (such as REC) be given the ability to voluntarily test functionality in a development environment and report issues prior to the general release of a software update.  If REC was allowed to perform UAT prior to the release, it would be very likely that we would have discovered this vulnerability prior to the overall release.

Absent any external UAT, the vulnerability demonstrates that there was limited internal UAT and other testing that took place within the FCC prior to the release of the software version.  Normally, internal testing involves the writing of internal test cases and then attempting to implement each test to see if the development version passes or fails.  Apparently any pre-release testing performed internally did not anticipate the issue at hand, perhaps under the assumption that end-users would only use the system as instructed. 

Based on the nature of this incident and the events leading up to the discovery of the vulnerability, it is our opinion that this vulnerability also existed during the 2021 NCE Filing Window, but was not discovered until the 2023 LPFM Window, the second radio filing window involving suppressed simultaneously filed applications since the conversion from CDBS to LMS.

It is REC’s conclusion that the vulnerability could have been prevented if the FCC, through the Media Bureau, which was the “business” customer for the development work and the Office of Managing Director, which oversees the “IT” side of the relationship would have reached out to well-known external stakeholders (such as broadcast engineers, consultants, SBE Certified Broadcast Technologists and attorneys) who were willing to devote time to assure that any proposed software update or new system introduction was user friendly, functioned properly and free of potential vulnerabilities.

We hope this incident is a wake-up call to the FCC to improve their relationships with the public where it comes to system releases and for that matter, open data, another aspect where a solid relationship between the provider and the user is lackluster. 

Next steps

The FCC has already sealed up the vulnerability.  Therefore, any applications filed after Thursday when LMS came back up after the patch will not be vulnerable.   Applications filed from midnight on Wednesday until 6PM on Thursday, which were once vulnerable, are now no longer vulnerable.

Members of the LPFM community have formally petitioned the FCC for an extension of the LPFM filing window to allow for applicants to be notified of the issue and if they need to make decisions to do so while “major” changes can still be made (before the window closes).  REC Networks is a signatory to that request.

Questions and Answers - General

Q: What exactly happened?

A: There was an exploit in the LMS software where upon entering a request into a web browser using a non-traditional method would have given exposure to LPFM applications filed during the window period with the intention that they would not be exposed to the public until after the window closed. 

Q: How many applications were potentially impacted?

A: We are estimating about 600.

Q: How long were the applications vulnerable?

A: For the LPFM filing window, the vulnerability would have happened from the time when the application was certified and a file number was issued until Thursday, December 7, at 6PM Eastern Time.  This would mean that an LPFM application that was filed at exactly midnight on on Wednesday when the filing window open would have been vulnerable for 42 hours. 

In addition, it is our belief that all applications that were filed in the 2021 NCE Filing Window were vulnerable from the time when the application was certified until the close of that filing window.

Q: Could this have been prevented?

A: Yes, through a more thorough testing regimen that involves stakeholders, both internal and external to the federal agency and a better security review.

Q: Like with other data incidents that we hear about in the news, was any personal information, such as passwords, credit card numbers, etc. exposed as a result of this vulnerability?

A: No.  The vulnerability only caused a “sneak peek” to pending applications prior to the time when they were going to be made available to the public. 

Q: Was this incident as a result of a “hack” or other malicious activity from a domestic or foreign player?

A: No.  It was just a software requirement that was overlooked in development.  The thought about the method used was probably not on the FCC’s radar either on the business end (Media Bureau) or the IT end (OMD).   REC is considering this a vulnerability.

Questions and Answers – REC Networks related

Q: Did REC receive any information on the specific LPFM applications filed in the window prior to Thursday at 6PM?

A: No. Since the applications were properly suppressed from the LMS Application Search, these applications were never seen by REC’s eLMS background services and were never picked up for publication in REC systems such as FCC.today and FCCdata.org.  Other than the work we are doing for clients, which is in its own secure environment, there are no traces of the vulnerable applications within the REC Data Store.

Q: Did REC either cause or exploit the vulnerability?

A: No.  We were informed of it after it was discovered and while REC does have systems that directly interact with LMS, we do not have any “normal business” routines that interact with the functionality where the vulnerability existed.

Q: What involvement did REC have in the discovery of the vulnerability?

A: We had no involvement in the discovery.  We were advised of it after it was discovered and have prepared this root cause document based on the information provided and a small amount of “post mortem” testing. 

Q: Did any part of REC’s incident investigation testing expose filed applications to REC?

A: No.  We did not start the post-mortem testing related to the incident until Friday evening and Saturday morning, after the FCC installed their patch on Thursday evening.  Our testing was limited to how LMS handles other (non-LPFM) application types where a new Facility ID is created.

Q: I am a client of REC’s 2023 Window Filing Services, was my information exposed?

A: Applications would have been vulnerable from the time that you received the email from REC stating that the application was filed (and the file number was provided) until 6PM Eastern Time on Thursday.  For clients, REC filed a small number of applications on late Wednesday afternoon and filed more applications on Thursday.  Therefore, the worst case scenario was that the earliest applications would have been vulnerable for just over 24 hours with a majority of applications being vulnerable for less than 8 hours. 

Q: Is there anything an LPFM window applicant can do?

A: When you sum up the information we discuss in this document, such as the indirect method used to find the information and the “spread” of Facility ID numbers assigned from the time when they were first assigned (when you received notification that an application is “Ready for Filing”) and the number of overall facility ID numbers issued by LMS in the past few months, many of which are only used internally by LMS (ownership reports), any LPFM has about a 1 in 10,000 chance that their information could have been viewed by someone who would have known how to exploit the LMS system to view the information. 

We are unable to determine the outcome until after the window closes and all other applications are viewed and analyzed.  Because of the increased requirements for site assurance information and increased awareness in FCC Staff because of the bad actors in the 2013 Window, we feel that any residual damage, though possible, will be mostly unlikely.

Q: I am still concerned about this.  I have other channels available. Can I change my channel?

A: If your application was filed by REC on Wednesday or Thursday and you have an alternate channel that meets all spacing requirements and can achieve a second-adjacent channel waiver, REC can amend your application at no charge.  Please note that we will need to amend these applications before or on Tuesday (unless extended by the FCC).  Keep in mind though, if you change your channel, you are shifting the risk of mutual exclusivity from one channel to another.   Applications filed on Friday or later were never exposed to the original vulnerability.

Q: REC has been releasing periodic information on the number of LPFM applications filed.  How do you determine that?

A: REC collects information on all broadcast applications filed during the day through LMS which are intended to be displayed to the public through our eLMS interface.  We use the same table that drives FCC.today.  We look at the last publicly available file number issued on the previous day, add one to that, then look at the highest file number issued and then subtract the lowest file number + 1 from the highest.  We then subtract the number of applications that were filed and showing in LMS/eLMS as well as subtract any pleadings filed (as they share the file number space with applications) and the difference is our estimate. 

Tags: