One of the network distribution switches serving the west side of the campus was rebooted at 6:45 AM to clear up a wireless connectivity issue. Traffic was re-routed to another distribution switch during the 3 minutes of reboot. Because 415 Yonge is off-campus and connected only to that switch without a redundant path, there was a network outage in the building during the reboot.
We apologize for the service interruption.
Computing and Communications Services
One of the 5 Blackboard application servers is experiencing some problems, and we will be taking it out of the load balanced pool around 1 PM. Users connected to this application server will not be affected by this action and new connections will be directed to the other application servers. We will leave the application server up until all current connections expire.
The vendor’s support group has already been contacted and we’re working towards resolution of this issue, which does not seem to be related to the cause of the previous problems that we’ve experienced recently.
We apologize for the service interruption.
-Computing and Communications Services
The Ryecast system will be upgraded on Monday, May 28, starting at 8 AM. The upgrade is expected to take less than 24 hours. While the upgrade takes place the system will not be available.
We apologize for any inconvenience this outage may cause.
-Computing and Communications Services
The recent Blackboard 9.1 SP8 upgrade introduced a bug with the authentication code, causing a memory leak, which over time leads to the problems recently experienced with the Blackboard system. To reduce the risk of similar incidents, the Blackboard application servers will be rebooted on a daily basis between 3 AM and 4 AM, until a patch for this issue is fully tested.
The vendor has informed us that another institution running the same version as Ryerson reported the same problem. This other institution has not fully tested the patch provided by Blackboard, and has also implemented a daily reboot of the application servers as a temporary fix to the problem. At this stage, we have deployed the patch in our test environment and so far the sanity testing carried out has showed good results. We are planning to perform additional load testing to ensure that the patch corrects the problem, and we will deploy it to the production system following a positive outcome.
We will provide periodic status updates of our testing.
We apologize for any inconvenience this may cause.
-Computing and Communications Services
This morning around 10 am the Blackboard system was experiencing a memory leak issue, the same issue that was experienced this Wednesday after the Blackboard upgrade. To prevent the problem from getting worse, all five application servers had to be restarted one at the time. During the server restart, the users who were connected to that particular server had their sessions dropped.
CCS is working with the vendor to get the official patch for this issue. To avoid similar issues from happening again and until this patch is released, we are planning to restart Blackboard application servers on daily basis between 3 am and 4 am.
We apologize for any inconvenience this may cause.
-Computing and Communications Services
A failure with one of the GroupWise systems (GW1) caused the GroupWise Web & Internet access to become unavailable around 4 PM, today, Thursday, May 24. Services were restored at approximately 4:18 PM.
We apologize for the service disruption.
-Computing and Communications Services
At approximately 11:56 am today, we started experiencing memory resources issues with some of the five Blackboard application servers. The servers needed to be restarted and users who were connected to these particular servers may have had their sessions dropped.
We are currently trouble-shooting the problem.
UPDATE 1: At 2 PM, the Blackboard system crashed. We are investigating the cause of the problem.
UPDATE 2: Blackboard services were restored around 3 PM. We continuing the investigation of the problem with vendor assistance.
We apologize for the service interruption.
-Computing and Communications Services
Since approximately 11:45 AM this morning, the www.ryerson.ca website was not loading properly. We worked with the Web Services team in University Advancement and the vendor’s support group to resolve the problem. The site was back up at approximately 12:55 PM.
We apologize for the service disruption.
-Computing and Communications Services
Update on Tuesday, May 22 at 2:00 pm: Just after 8 pm on Monday, May 21 CCS started receiving complaints from students that were not able to submit assignments on Blackboard and from one instructor who was not able to enter grades. The problem was traced to a database issue related to Blackboard’s email notification system. When the notification feature was disabled database performance gradually returned to normal and the system began functioning correctly again around 11 pm.
Update on Monday, May 21 at 4:30 pm: The upgrade of Blackboard to version 9.1 was completed on Monday, May 21 at 2:50 pm. The upgrade took longer than expected due to some database problems including slower-than-expected database performance. Blackboard, RAMSS, SoapBox and all other applications accessed through the my.ryerson portal are all back online.
If you experience any problems using Blackboard or have any questions regarding the upgrade, please contact the CCS Help Desk at help@ryerson.ca or at ext. 6806.
Please accept our apologies for the prolonged outage.
-Computing and Communications Services
Update on Monday, May 21 at 11 am: We experienced some system issues during the Blackboard upgrade which impacted our upgrade completion time which was initially scheduled for 8 am on Monday, May 21. We expect that the Blackboard upgrade will be completed early this afternoon. We will keep you posted on the status of the upgrade.
We apologize for any inconvenience this may cause.
-Computing and Communications Services
One of the Groupwise post office servers, PO3, did not complete the regular nightly maintenance process that starts at 5AM. The server had to be rebooted at approximately 8:00 AM, Wednesday May 16. The outage affected some Schools and a few administrative staff.
Please accept our apologies for the disruption in service.
-Computing and Communications Services
