Blackboard application servers issues – Wed, May 23

Posted on May 23, 2012 by Branka Halilovic

At approximately 11:56 am today, we started experiencing memory resources issues with some of the five Blackboard application servers. The servers needed to be restarted and users who were connected to these particular servers may have had their sessions dropped.

We are currently trouble-shooting the problem.

UPDATE 1: At 2 PM, the Blackboard system crashed.   We are investigating the cause of the problem.  

UPDATE 2: Blackboard services were restored  around 3 PM.   We continuing the investigation of the problem with vendor assistance. 

We apologize for the service interruption.

-Computing and Communications Services

Posted in All Categories, Blackboard | Tagged | 2 Comments

www.ryerson.ca unavailable: Wednesday, May 23 2012 11:45 AM – 12:55 PM

Posted on May 23, 2012 by Casey Carvalho

Since approximately 11:45 AM this morning, the www.ryerson.ca website was not loading properly. We worked with the Web Services team in University Advancement and the vendor’s support group to resolve the problem. The site was back up at approximately 12:55 PM.

We apologize for the service disruption.

-Computing and Communications Services

Posted in All Categories | Comments Off on www.ryerson.ca unavailable: Wednesday, May 23 2012 11:45 AM – 12:55 PM

Blackboard upgrade status: Monday, May 21

Posted on May 21, 2012 by Branka Halilovic

Update on Tuesday, May 22 at 2:00 pm: Just after 8 pm on Monday, May 21 CCS started receiving complaints from students that were not able to submit assignments on Blackboard and from one instructor who was not able to enter grades. The problem was traced to a database issue related to Blackboard’s email notification system. When the notification feature was disabled database performance gradually returned to normal and the system began functioning correctly again around 11 pm.

Update on Monday, May 21 at 4:30 pm: The upgrade of Blackboard to version 9.1 was completed on Monday, May 21 at 2:50 pm. The upgrade took longer than expected due to some database problems including slower-than-expected database performance. Blackboard, RAMSS, SoapBox and all other applications accessed through the my.ryerson portal are all back online.
If you experience any problems using Blackboard or have any questions regarding the upgrade, please contact the CCS Help Desk at help@ryerson.ca or at ext. 6806.

Please accept our apologies for the prolonged outage.
-Computing and Communications Services

Update on Monday, May 21 at 11 am: We experienced some system issues during the Blackboard upgrade which impacted our upgrade completion time which was initially scheduled for 8 am on Monday, May 21. We expect that the Blackboard upgrade will be completed early this afternoon. We will keep you posted on the status of the upgrade.

We apologize for any inconvenience this may cause.
-Computing and Communications Services

Posted in All Categories, Blackboard, RAMSS | Tagged , | Comments Off on Blackboard upgrade status: Monday, May 21

Groupwise Post Office 3 outage: Wedn., May 16 2012 05:00 AM – 08:00 AM

Posted on May 17, 2012 by Casey Carvalho

One of the Groupwise post office servers, PO3, did not complete the regular nightly maintenance process that starts at 5AM. The  server had to be rebooted at approximately 8:00 AM, Wednesday May 16.   The outage affected some Schools and a few administrative staff.

Please accept our apologies for the disruption in service.

-Computing and Communications Services

Posted in GroupWise | Comments Off on Groupwise Post Office 3 outage: Wedn., May 16 2012 05:00 AM – 08:00 AM

Brief service disruption of uPortal, CAS and HR services: Wed., May 16 11:36 AM – 11:39 AM

Posted on May 16, 2012 by Casey Carvalho

A reboot of the SSL Accelerator appliance servicing uPortal (my.ryerson), CAS (Central Authentication Service) and HR, caused a brief service outage this morning, Wednesday May 16, between 11:36 AM – 11:39 AM.   During this period, new connections were not successful.    Existing sessions were not affected by the problem.

We apologize for the service interruption.

– Computing and Communications Services

Posted in eHR, eResources, my.ryerson | Comments Off on Brief service disruption of uPortal, CAS and HR services: Wed., May 16 11:36 AM – 11:39 AM

ColdFusion Database Issue – May 9/10, 2012

Posted on May 10, 2012 by Jim Buchanan

Due to an inadvertent error when restoring a database schema, all Cold Fusion data was overwritten as of 3:51 PM on Wednesday May 9.  The Cold Fusion applications affected were running on the following systems:

  • runner.ryerson.ca
  • ccs.cf.ryerson.ca
  • hr.cf.ryerson.ca
  • ss.cf.ryerson.ca
  • alfresco.cf.ryerson.ca

A restore was completed at approximately 11:00 AM today, Thursday May 10.

All data entered between 3:51PM Wednesday and 10:00 AM Thursday has been lost. CCS is contacting system owners to help assess the impact on individual applications.

We apologize for the problems this outage will cause.

-Computing and Communications Services

Posted in All Categories | Comments Off on ColdFusion Database Issue – May 9/10, 2012

High Firewall CPU Utilization Affecting OGF and SAS Services – 8:30 am, May 08, 2012

Posted on May 9, 2012 by kwoo

The data centre firewall in front of SAS, OGF and HR systems is currently experiencing high CPU utilization, over 90%. This may be increasing the response times for users trying to access these systems.

The problem has been identified. There is an inherent memory leak in the code level and will need to be upgraded.   CCS continues to monitor the situation.  To mitigate the problem, the back-up firewall will be upgraded to the new code and “flipped” into production at off production hours.

We apologize for any service disruption.

Computing and Communications

Posted in All Categories, Firewall | Comments Off on High Firewall CPU Utilization Affecting OGF and SAS Services – 8:30 am, May 08, 2012

Webmail service disruption: Tuesday, May 1, 2012: 16:01 PM – 16:16 PM

Posted on May 1, 2012 by Casey Carvalho

A problem with Rmail caused a partial service outage with Webmail,  today, Tuesday May 1st, between 16:01 PM.- 16:16 PM.   During this period, new connections to Webmail were not successful.    Existing sessions were not affected by the problem.

We apologize for the service interruption.

– Computing and Communications Services

Posted in RMail | Comments Off on Webmail service disruption: Tuesday, May 1, 2012: 16:01 PM – 16:16 PM

Search.ryerson.ca service outage – Wednesday April 25, 2012

Posted on April 25, 2012 by Jim Buchanan

The service that provides search functions for www.ryerson.ca Web sites failed this morning.

The Google Mini Search Appliance that provides the service appears to have gone down at roughly 9 AM this morning and was restarted at 10 AM. While the search feature on www.ryerson.ca is working again, it will be some time before the system fully recovers and returns search results quickly.

Please accept our apologies for the service disruption.

-Computing and Communications Services

Posted in All Categories | Comments Off on Search.ryerson.ca service outage – Wednesday April 25, 2012

TRS-YDI Wireless Controller Fail-Over – 2012-April-20 9:30am

Posted on April 20, 2012 by kwoo

Due to a problem with the WiFi network in TRS and YDI some people may find the WiFi network has stopped working and have to login again.

We are investigating the problem. However, it appears the network interface to the active TRS/YDI controller or the interface on the distribution switch failed. The failure caused the backup controller to take over. All access points failed over to the backup controller automatically and instantly.

We apologize for the service interruption.

– Computing and Communications Services.

Posted in All Categories, wireless | Comments Off on TRS-YDI Wireless Controller Fail-Over – 2012-April-20 9:30am