Cold Fusion instance for Student Services outage: 11 AM on Monday, Oct. 31, 2016 to 5 PM on Tuesday, Nov. 1, 2016

The applications hosted on the Student Services Cold Fusion instance (ss.cf.ryerson.ca), such as Athletics, Bookstore, CESAR Elections and TTC,  were either partially or completely unavailable between approximately 11 AM on Monday, Oct. 31, 2016 and  5 PM on  Tuesday, Nov. 1, 2016.  The issue has been isolated to one application, which has been taken offline. The rest of the applications were fully brought back online.  We continue to investigate the root cause.

We apologize for any inconvenience.

-Computing and Communications Services

Posted in All Categories | Comments Off on Cold Fusion instance for Student Services outage: 11 AM on Monday, Oct. 31, 2016 to 5 PM on Tuesday, Nov. 1, 2016

Jira, Confluence and IM/Spark outage: Wed., Nov. 2 2016 7:47 AM – 9:48 AM


Jira, Confluence and IM/Spark (the instant message service) were not available from 7:47 Am to 9:48 Am today due to a database server partition issue. The issue has been fixed and the services have returned to normal. We are looking into ways to prevent this from happening again.

We apologize for the inconvenience.

-Computing and Communication Services

Posted in All Categories | Comments Off on Jira, Confluence and IM/Spark outage: Wed., Nov. 2 2016 7:47 AM – 9:48 AM

Student printing problem: Friday, Oct. 28, 2016 9:40 AM

Printing to CCS managed lab printers, including those in the Library Commons (LIB272), TRS and KHW71 is currently not working.   The problem was reported around 9:40 AM and it is being actively investigated.   We will provide an update shortly.

UPDATE:  The printing issue was resolved around 10:40 AM.

Please accept our apologies for the inconvenience.

-Computing and Communications Services

Posted in All Categories, Lab printing | Comments Off on Student printing problem: Friday, Oct. 28, 2016 9:40 AM

XMedius Fax outage – inbound faxes: Thursday, October 27, 2016

Ryerson’s Fax service, XMedius Fax experienced an outage for inbound faxes only  on Thursday, October 27, 2016.

The issue started at approximately 11:15 AM on Thursday.

CCS continues to investigate the cause and will provide updates as they are available.

Please accept our apologies for any inconvenience this may cause.

-Computing and Communications Services

Posted in All Categories, Fax | Comments Off on XMedius Fax outage – inbound faxes: Thursday, October 27, 2016

blog.ryerson.ca outage: Monday, October 17, 2016

Ryerson’s Blog system, blog.ryerson.ca experienced a system outage today, Monday October 17, 2016

The issue started about 11:45 AM  with intermittent errors and was fixed at about 1:55 PM.

We are continuing to investigate the problem.

Please accept our apologies for any inconvenience this may cause.

-Computing and Communications Services

Posted in All Categories, WordPress blog | Comments Off on blog.ryerson.ca outage: Monday, October 17, 2016

Network outage on Friday, October 14, 2016 at 12:20 PM

On Friday, October 14, 2016, at 12:20 PM, CCS received alerts that portions of the wired and wireless networks at the west side of the campus are not functioning. Network Services team is investigating the issue.

Update:

The issue was traced to a malfunction of a network switch. Services resumed at 12:40 PM

We apologize for any inconvenience.

-Computing and Communications Services

Posted in All Categories, network | Tagged | Comments Off on Network outage on Friday, October 14, 2016 at 12:20 PM

Multiple services disruption due to server crash: Tuesday, Oct. 4, 2016 9:42 AM – 9:55 AM

One of the physical servers that hosts various virtual machines (VMs) crashed on Tuesday, Oct. 4 at approximately 9:42 AM.   The server crash resulted in a number of service outages, including admin print services applications.  All services were restored around 9:55 AM.   CCS administrators are investigating the cause of the problem.

The list of affected servers includes:

careerresources
ce-tableau
Dcnfs
dmz-mysql
dmz-mysql-01
elgg
essrkofax
Foam
food
gitccs
hisw-web-01
libwp
midasw
mps2
nolij-transfer
qnomy-db
seasons2-panorama
se-socon2
surveyweb1
svn.prodint
tagweb
trs-emssql
ubs-cbs
webconf1

We apologize for any inconvenience.

-Computing and Communications Services

Posted in All Categories | Comments Off on Multiple services disruption due to server crash: Tuesday, Oct. 4, 2016 9:42 AM – 9:55 AM

Email delivery delays on Monday Oct. 3, 2016

On Monday, Oct. 3, 2016, CCS noticed that emails were delayed for up to 25 minutes since about 7:00 am.  Mitigation steps were taken at 11:04 am and the service has returned to normal. CCS is investigating the issue.

Update: The vendor of one of the filtering appliances we use to check mail for spam has informed us that there is a bug in their software that causes intermittent memory problems that will be fixed soon.

We apologize for any inconvenience.

-Computing and Communications Services

Posted in All Categories, Mail | Comments Off on Email delivery delays on Monday Oct. 3, 2016

Email delivery delays on Friday Sep. 30, 2016

Due to a very high volume of mail there were significant delays in delivering mail with reports of email taking two hours to be delivered. We expect the backlog to clear this evening and apologize for the problems these delays have caused.

-Computing and Communications Services

Posted in All Categories, Mail | Tagged | Comments Off on Email delivery delays on Friday Sep. 30, 2016

Slow Internet access on Wednesday September 28, 2016 at 3:25 PM

On Wednesday, September 28, 2016 at 3:25 PM CCS received reports about slow or intermittent internet access.  Network services team is investigating the issue.

Update:

The network services team tuned some traffic threshold parameters and services resumed at 4:00 PM

Update October 3, 2016:

The Internet outage on September 28, 2016  was the result of a Distributed Denial of Service (DDoS) attack. The attack was first reported at approximately 3:14 PM. DDoS network traffic is designed to stress networks and servers to the point that they become unavailable. In response, CCS began to aggressively filter network traffic until Internet access from inside Ryerson was restored and the DDoS attack subsided.

During the initial stage of the attack people attempting to access the Internet from on campus were not able to do so or found access extremely slow. However, anyone off campus was still able to connect to Ryerson’s core services such as the my.ryerson.ca portal.

By 3:37 PM, when Ryerson began aggressively filtering network traffic, a small number of people were no longer able to access Ryerson from off campus. At the same time network access from on campus was restored. 

As a safety precaution, a moderately aggressive level of filtering was left on overnight. This caused connection issues for a relatively small number of people trying to connect to Ryerson during the morning of Thursday September 29 until approximately 10 AM when filtering was adjusted back to normal levels.

CCS is analysing the attack and our response to it and will work to improve our response to DDoS attacks in the future.

-Computing and Communications Services

Posted in All Categories, Internet | Tagged | Comments Off on Slow Internet access on Wednesday September 28, 2016 at 3:25 PM