NYC 911 outage at PAC2, August 18, 2026: City Council oversight hearing probes why automated monitoring failed to detect a firewall firmware upgrade problem that knocked out audio for callers routed through PAC2 (Bronx) and led the city to reroute 911 traffic to PAC1 (Brooklyn).
# What’s happening
– The August 18, 2026 911 outage began at 3:13 a.m. (system timeline).
– The Office of Technology and Innovation (OTI) manages the 911 infrastructure and vendor relationships.
– Calls were fully rerouted away from PAC2 by 10:19 a.m. on August 18, 2026.
Source: https://youtu.be/BT4l-byOlC0&t=1215
# Why it matters
– Callers in the Bronx and other areas could not hear call takers; up to hundreds of callers were affected.
– Failures in automated monitoring can delay citywide response and public notification during emergencies.
Source: https://youtu.be/BT4l-byOlC0&t=654
# Key details
– Change process: Motorola firewall firmware upgrade at PAC2 ran 9:45 p.m., Aug 17 to 4:13 a.m., Aug 18, 2026.
– Initial problem time: audio failures began at 3:13 a.m., Aug 18, 2026.
– First staff report: a call taker reported audio trouble just before 6:00 a.m.; a support ticket opened at 6:40 a.m.
– Diagnosis: by ~9:50 a.m., issue isolated to PAC2 routers; PAC1 remained functional.
– Remediation: Motorola rerouted all 911 traffic to PAC1 and disabled ~900 PAC2 trunks; reroute completed by 10:19 a.m.
– Impact estimates: testimony cited 319 unique 911 callers affected (206 connected, 113 not connected) and an OTI estimate that around 400 callers may have been unable to reach emergency services.
– Interim actions: OTI instituted a freeze on new deployments, daily executive status meetings, hourly sampling of 911 call recordings, and a strict exception protocol for critical updates.
Source: https://youtu.be/BT4l-byOlC0&t=1020
Office of Technology and Innovation (OTI)
– Role or jurisdiction: Manages city 911 infrastructure and vendor relationships.
– Action taken or responsibility: Led incident response, coordinated with NYPD, FDNY, and Motorola.
– Relevant numbers or dates: Instituted deployment freeze after Aug 18 outage; said teams are reviewing call recordings hourly.
Source: https://youtu.be/BT4l-byOlC0&t=1415
Motorola
– Role or jurisdiction: Vendor supplying firewall firmware and routing hardware used at PAC2.
– Action taken or responsibility: Performed the August 17–18 firewall firmware upgrade cited as causal and later redirected traffic to PAC1 during remediation.
– Relevant numbers or dates: Upgrade began Aug 17 at 9:45 p.m. and completed Aug 18 at 4:13 a.m.; remediation reroute completed by 10:19 a.m., Aug 18.
Source: https://youtu.be/BT4l-byOlC0&t=4141
NYPD
– Role or jurisdiction: Operates 911 call-taking staff and call centers (PAC1 in Brooklyn and PAC2 in the Bronx).
– Action taken or responsibility: Notified OTI of call problems; call takers first raised audio issues around 6:00 a.m. Aug 18.
– Relevant numbers or dates: Call takers reported audio trouble shortly before 6:00 a.m., Aug 18.
Source: https://youtu.be/BT4l-byOlC0&t=1020
The City Council hearing and OTI’s explanation
City Council committees held an oversight hearing on September 9, 2026, to examine why automated monitoring and alerts did not detect the PAC2 audio/session problem for hours. (Source: https://youtu.be/BT4l-byOlC0&t=654)
OTI and other city witnesses testified that Motorola performed a firewall firmware upgrade at PAC2 overnight Aug 17–18, 2026, and that the upgrade was believed successful when it finished at 4:13 a.m. (Source: https://youtu.be/BT4l-byOlC0&t=1020)
OTI said the first indication that callers could not hear 911 call takers came from a call taker shortly before 6:00 a.m., and a tech support ticket was opened at 6:40 a.m. (Source: https://youtu.be/BT4l-byOlC0&t=1020)
By about 9:50 a.m., the incident response team diagnosed that the issue affected calls routed through PAC2 routers while PAC1 remained functional; Motorola remedied by directing traffic to PAC1 and disabling roughly 900 PAC2 trunks, finishing the reroute at 10:19 a.m. (Source: https://youtu.be/BT4l-byOlC0&t=1215)
Why monitoring failed, per OTI testimony
OTI witnesses said existing electronic monitoring and alarms did not catch this problem. They attributed the detection failure to the way the error presented: sessions remained live and only failed after a session timeout (time-to-live expired), so no automated alert triggered. OTI testified that many monitoring systems exist across layers but that this specific anomaly did not surface in those systems. (Source: https://youtu.be/BT4l-byOlC0&t=5931) (Source: https://youtu.be/BT4l-byOlC0&t=3969)
OTI acknowledged the testing protocol had been followed for what had been considered a low-risk change, but the tests did not capture the session-timeout scenario; OTI said testing must be revised to cover such cases. (Source: https://youtu.be/BT4l-byOlC0&t=5931)
Immediate and interim changes OTI described
OTI described several interim operational changes put in place while it completes a root-cause analysis: continuous sampling of 911 call recordings to check audio quality; coordination with NYPD on defining and escalating “dead air” calls; a freeze on new deployments until fixes are identified; exception-only handling for critical updates with executive-level review across partner agencies; and daily executive status meetings to track work streams. OTI did not provide specific new alert thresholds or logging configurations during the hearing. (Source: https://youtu.be/BT4l-byOlC0&t=1415) (Source: https://youtu.be/BT4l-byOlC0&t=1215)
Responsibility and vendor management
Council members asked who manages vendors. OTI testified that OTI is responsible for managing the vendor relationship, while acknowledging that Motorola performed the firewall firmware upgrade that initiated the problem. City witnesses said they will conduct a deeper exploration to assign precise responsibility after the root-cause analysis. (Source: https://youtu.be/BT4l-byOlC0&t=7666) (Source: https://youtu.be/BT4l-byOlC0&t=4141)
Numbers on callers and remaining questions
Testimony included differing impact figures. OTI and other witnesses referenced 319 unique 911 callers affected, with 206 eventually connected and 113 not connected; elsewhere testimony estimated around 400 callers may have been unable to reach emergency services. Council members pressed for further details about the unconnected calls and possible harms; OTI said investigations to identify callers and impacts are ongoing. (Source: https://youtu.be/BT4l-byOlC0&t=5931) (Source: https://youtu.be/BT4l-byOlC0&t=654)
Public notification and next steps
Council members criticized delays in public notification; OTI said notification protocols and the role of the city’s emergency messaging system (Notify NYC / NIMS as used by the city) will be reviewed to clarify who communicates outages and under what criteria. OTI said it will complete a thorough investigation and apply lessons to change-management protocols. (Source: https://youtu.be/BT4l-byOlC0&t=7666) (Source: https://youtu.be/BT4l-byOlC0&t=1415)
What OTI declined or could not say
OTI declined to provide full technical detail about specific monitoring tools or security-sensitive logging configurations during the public hearing, citing security concerns. OTI also did not announce immediate, concrete changes to alert thresholds or specific logging fields that will be added; instead it committed to a “much closer look” and to updating testing and monitoring to capture session-timeout scenarios going forward. (Source: https://youtu.be/BT4l-byOlC0&t=3969) (Source: https://youtu.be/BT4l-byOlC0&t=5931)
GetLocalPost 
Leave a comment