All systems are operational

Past Incidents

Friday 7th August 2020

2020-08-06 Incident | RFO Report

On 8/6 at 7:37 AM Pacific local time, NocTel engineering detected anomalies with service and application availability through monitoring systems and began investigating scope of services impaired. By 7:51 AM Pacific local, the NocTel engineering team had diagnosed a storage system failure impacted the gamut of services and worked to redirect access to available resources while beginning repair of affected services and servers.

The timeline of NocTel engineering's activities to address service and system repairs:

  • 8:27 AM Pacific engineering team arrived at affected datacenter.

  • 9:01 AM Pacific local voice services on PDX10, 11, and 19 were repaired, tested, and verified.

  • 9:06 AM Pacific local voice services on PDX12 were repaired, tested, and verified.

  • 9:26 AM Pacific local, some NocTel Flow accounts were repaired, tested, and verified.

  • 9:32 AM Pacific local voice services on PDX16 were repaired, tested, and verified.

  • 10:25 AM Pacific local voice services on PDX15 were repaired, tested, and verified.

  • 11:31 AM Pacific local voice services on PDX13 were repaired, tested, and verified.

  • 11:58 AM Pacific local, all Flow services restored.

  • 12:06 PM Pacific local PDX14 voice services and faxing services repaired, tested, and verified.

  • 2:11 PM Pacific local all services repaired, tested, and verified. Direct support conducted for affected accounts with specific configuration in proceeding hours.

In reflection of the incident, NocTel engineering recognizes architectural improvements to eliminate single points of failure and to reduce extent of "blast radius" when an incident - in general - occurs anywhere in the system or infrastructure. Longer term horizon, NocTel engineering already has ongoing and in-progress discussion, design, and implementation of a more performant and fault-tolerant architecture. In the interim, measures to address specific fault prone or high impact fault points will be implemented.

Thursday 6th August 2020

NocTel Talk Services | Watching UPDATE

Inbound and outbound calling has been restored at this time. Our team is still monitoring the issue closely.

If you are still having issues with inbound, outbound, or direct extension to extension calls, please let our support team know by sending a ticket to support@noctel.com

NocTel Talk, Flow, and Control Panel Services | Investigating UPDATE

A team has been dispatched to our PDX datacenter to investigate further and resolve the problem currently affecting storage components.

Currently, users should not make any changes to their account via the control panel as the team works to resolve the issue. Some users may experience issues with outbound, and inbound calling. Voicemail and faxing are also affected at this time.

Updates to follow.

Inbound calling Talk and Flow | Investigating

NocTel Talk and Flow services are experiencing issues that are interrupting services for some users. The NocTel team is investigating the issue. Please stand by for further updates.

NocTel Control Panel Website and Control Panel | Investigating

The NocTel website site and Control Panel is experiencing issues that may make the sites inaccessible to some user at this time. The NocTel team is investigating the issue. Please stand by for further updates.

Wednesday 5th August 2020

No incidents reported

Tuesday 4th August 2020

No incidents reported

Monday 3rd August 2020

No incidents reported

Sunday 2nd August 2020

No incidents reported

Saturday 1st August 2020

Planned Maintenance - NocTel Fiber, scheduled 1 week ago

Planned infrastructure maintenance on NocTel Fiber is scheduled for August 2, 2020. Please see the following for details:

  • Planned Maintenance Date: 2 August, 2020

  • Planned Maintenance Window: 06:00 - 08:00 PST

  • Expected Service Impact: NocTel services may be temporarily affected for customers using NocTel Fiber during the maintenance window. While our engineers work hard to minimize impact, NocTel reserves the right to utilize the entire maintenance window if necessary. Long-term and chronic system stability and use issues shall not occur.

  • Maintenance Scope: Necessary infrastructure power maintenance to improve backend stability and performance.

If you have questions or concerns regarding the planned maintenance, please contact NocTel Fiber Support Team (help@noctel.com, +1.360.837.7400). Likewise, if you find your service is impacted post maintenance, please immediately contact support.