Blackboard daily restart scheduled between 3 AM and 4 AM

Posted on May 26, 2012 by Branka Halilovic

The recent Blackboard 9.1 SP8 upgrade introduced a bug with the authentication code, causing a memory leak, which over time leads to the problems recently experienced with the Blackboard system.  To reduce the risk of similar incidents, the Blackboard application servers will be rebooted on a daily basis between 3 AM and 4 AM, until a patch for this issue is fully tested.


The vendor has informed us that another institution running the same version as Ryerson reported the same problem.  This other institution has not fully tested the patch provided by Blackboard, and has also implemented a daily reboot of the application servers as a temporary fix to the problem.  At this stage, we have deployed the patch in our test environment and so far the sanity testing carried out has showed good results.   We are planning to perform additional load testing to ensure that the patch corrects the problem, and we will deploy it to the production system following a positive outcome.


We will provide periodic status updates of our testing.

We apologize for any inconvenience this may cause.

-Computing and Communications Services

Posted in All Categories, Blackboard | Tagged | Comments Off on Blackboard daily restart scheduled between 3 AM and 4 AM

Blackboard Learn Unavailable, Friday May 25, 2012 between 10:25 am and 10:50 am

Posted on May 25, 2012 by Branka Halilovic

This morning around 10 am the Blackboard system was experiencing a memory leak issue, the same issue that was experienced this Wednesday after the Blackboard upgrade. To prevent the problem from getting worse, all five application servers had to be restarted one at the time. During the server restart, the users who were connected to that particular server had their sessions dropped.

CCS is working with the vendor to get the official patch for this issue. To avoid similar issues from happening again and until this patch is released, we are planning to restart Blackboard application servers on daily basis between 3 am and 4 am.

We apologize for any inconvenience this may cause.
-Computing and Communications Services

Posted in All Categories, Blackboard | Tagged | Comments Off on Blackboard Learn Unavailable, Friday May 25, 2012 between 10:25 am and 10:50 am

GroupWise web & internet access unavailable: Thursday, May 24, 2012 4:00 PM – 4:18 PM

Posted on May 24, 2012 by Casey Carvalho

A failure with one of the GroupWise systems (GW1) caused the GroupWise Web & Internet access to become unavailable around 4 PM, today, Thursday, May 24.   Services were restored at approximately 4:18 PM.

We apologize for the service disruption.

-Computing and Communications Services

Posted in GroupWise | Comments Off on GroupWise web & internet access unavailable: Thursday, May 24, 2012 4:00 PM – 4:18 PM

Blackboard application servers issues – Wed, May 23

Posted on May 23, 2012 by Branka Halilovic

At approximately 11:56 am today, we started experiencing memory resources issues with some of the five Blackboard application servers. The servers needed to be restarted and users who were connected to these particular servers may have had their sessions dropped.

We are currently trouble-shooting the problem.

UPDATE 1: At 2 PM, the Blackboard system crashed.   We are investigating the cause of the problem.  

UPDATE 2: Blackboard services were restored  around 3 PM.   We continuing the investigation of the problem with vendor assistance. 

We apologize for the service interruption.

-Computing and Communications Services

Posted in All Categories, Blackboard | Tagged | 2 Comments

www.ryerson.ca unavailable: Wednesday, May 23 2012 11:45 AM – 12:55 PM

Posted on May 23, 2012 by Casey Carvalho

Since approximately 11:45 AM this morning, the www.ryerson.ca website was not loading properly. We worked with the Web Services team in University Advancement and the vendor’s support group to resolve the problem. The site was back up at approximately 12:55 PM.

We apologize for the service disruption.

-Computing and Communications Services

Posted in All Categories | Comments Off on www.ryerson.ca unavailable: Wednesday, May 23 2012 11:45 AM – 12:55 PM

Blackboard upgrade status: Monday, May 21

Posted on May 21, 2012 by Branka Halilovic

Update on Tuesday, May 22 at 2:00 pm: Just after 8 pm on Monday, May 21 CCS started receiving complaints from students that were not able to submit assignments on Blackboard and from one instructor who was not able to enter grades. The problem was traced to a database issue related to Blackboard’s email notification system. When the notification feature was disabled database performance gradually returned to normal and the system began functioning correctly again around 11 pm.

Update on Monday, May 21 at 4:30 pm: The upgrade of Blackboard to version 9.1 was completed on Monday, May 21 at 2:50 pm. The upgrade took longer than expected due to some database problems including slower-than-expected database performance. Blackboard, RAMSS, SoapBox and all other applications accessed through the my.ryerson portal are all back online.
If you experience any problems using Blackboard or have any questions regarding the upgrade, please contact the CCS Help Desk at help@ryerson.ca or at ext. 6806.

Please accept our apologies for the prolonged outage.
-Computing and Communications Services

Update on Monday, May 21 at 11 am: We experienced some system issues during the Blackboard upgrade which impacted our upgrade completion time which was initially scheduled for 8 am on Monday, May 21. We expect that the Blackboard upgrade will be completed early this afternoon. We will keep you posted on the status of the upgrade.

We apologize for any inconvenience this may cause.
-Computing and Communications Services

Posted in All Categories, Blackboard, RAMSS | Tagged , | Comments Off on Blackboard upgrade status: Monday, May 21

Groupwise Post Office 3 outage: Wedn., May 16 2012 05:00 AM – 08:00 AM

Posted on May 17, 2012 by Casey Carvalho

One of the Groupwise post office servers, PO3, did not complete the regular nightly maintenance process that starts at 5AM. The  server had to be rebooted at approximately 8:00 AM, Wednesday May 16.   The outage affected some Schools and a few administrative staff.

Please accept our apologies for the disruption in service.

-Computing and Communications Services

Posted in GroupWise | Comments Off on Groupwise Post Office 3 outage: Wedn., May 16 2012 05:00 AM – 08:00 AM

Brief service disruption of uPortal, CAS and HR services: Wed., May 16 11:36 AM – 11:39 AM

Posted on May 16, 2012 by Casey Carvalho

A reboot of the SSL Accelerator appliance servicing uPortal (my.ryerson), CAS (Central Authentication Service) and HR, caused a brief service outage this morning, Wednesday May 16, between 11:36 AM – 11:39 AM.   During this period, new connections were not successful.    Existing sessions were not affected by the problem.

We apologize for the service interruption.

– Computing and Communications Services

Posted in eHR, eResources, my.ryerson | Comments Off on Brief service disruption of uPortal, CAS and HR services: Wed., May 16 11:36 AM – 11:39 AM

ColdFusion Database Issue – May 9/10, 2012

Posted on May 10, 2012 by Jim Buchanan

Due to an inadvertent error when restoring a database schema, all Cold Fusion data was overwritten as of 3:51 PM on Wednesday May 9.  The Cold Fusion applications affected were running on the following systems:

  • runner.ryerson.ca
  • ccs.cf.ryerson.ca
  • hr.cf.ryerson.ca
  • ss.cf.ryerson.ca
  • alfresco.cf.ryerson.ca

A restore was completed at approximately 11:00 AM today, Thursday May 10.

All data entered between 3:51PM Wednesday and 10:00 AM Thursday has been lost. CCS is contacting system owners to help assess the impact on individual applications.

We apologize for the problems this outage will cause.

-Computing and Communications Services

Posted in All Categories | Comments Off on ColdFusion Database Issue – May 9/10, 2012

High Firewall CPU Utilization Affecting OGF and SAS Services – 8:30 am, May 08, 2012

Posted on May 9, 2012 by kwoo

The data centre firewall in front of SAS, OGF and HR systems is currently experiencing high CPU utilization, over 90%. This may be increasing the response times for users trying to access these systems.

The problem has been identified. There is an inherent memory leak in the code level and will need to be upgraded.   CCS continues to monitor the situation.  To mitigate the problem, the back-up firewall will be upgraded to the new code and “flipped” into production at off production hours.

We apologize for any service disruption.

Computing and Communications

Posted in All Categories, Firewall | Comments Off on High Firewall CPU Utilization Affecting OGF and SAS Services – 8:30 am, May 08, 2012