Rmail, my.ryerson outages – Mon. August 27, 2012

Early this morning a device on our Storage Area network failed. As a result disk storage for Rmail, my.ryerson, Blackboard, and other systems went offline. (Other services like RAMSS and eHR are not reachable when my.ryerson is down.) We are investigating the cause and working to restore service. Updates to follow.

Update 8:20 AM: A storage appliance that manages the storage connections to the Rmail, my.ryerson and other servers failed. A second storage appliance is available as a backup but for some reason the servers were not able to tolerate the storage connectivity failures before the second device took over.

Update 9:36 AM: The problem seems to be related with a failed controller on one of the storage appliances that provides storage virtualization. The vendors are working on the problem and replacement parts are being sent to us.

In the meantime, we’re also working on bringing up services on the backup appliance, with the idea that we will be able to “fail over” to the other appliance (in a controlled way) when it is working properly.

Update 12:37 PM: With the help of the vendor, whose staff are on site, we are in the process of restoring/rebuilding the failed storage appliance. We estimate that the earliest we can be back on line is some time after 2:40 PM. It may take significantly longer. In parallel with the work to restore the failed device we are also preparing another appliance which we can reconfigure to replace the failed unit.

Update 2:21 PM: We have run into other problems rebuilding the storage appliance that failed and now think we will have a replacement appliance ready before we can repair the unit that failed. This pushes back the earliest possible time to fix the problem until after 5 PM. Once again, please accept my apologies for the prolonged outage.

Update 5:40 PM: The new storage appliance is in place but it will be many hours before all services are restored. The appliance must go through a diagnostic process, test its connections to the storage devices, and be reconfigured to provide storage for each server. Initial estimates on how long that would take are proving to be optimistic. Once everything is working again we must start the process of validating that all files and databases have been recovered correctly. From what we have seen in the past this will also take hours for some services like Rmail. Based on what we’ve seen over the last hour we can’t provide a good estimate of how long it will take to get on line this evening. CCS staff will work as long as necessary to recover all services.

Update 9:30 PM: At roughly 7:30 PM we experienced a setback with restoring the configuration of the appliance. The problem means we do not have network access to the device and we are working with the vendor to continue recovering the device’s configuration. We have lost at least two hours because of this.

I know many people would like to know when the system will be restored. We are still working towards it being back online in the late evening or early hours in the morning, but at this time, we cannot predict when service will be restored.
-Brian

Update 10:54 PM: With the help of the vendor’s third-level support staff we have finally completed the automatic recovery process and are beginning the final configuration steps of the storage appliance. Network connectivity has also been established.

Update 11:17 PM: Both storage appliances are now restored and working correctly. Staff are now checking the databases and file systems that were impacted by the outage.

Update 00:40 AM Tuesday, Aug. 28: A few systems have now been recovered and services are available.   CAS, uPortal services were restored around 00:20 AM,  Blackboard was restored around 00:25 AM.   RMAIL was restored around 00:45 AM.     The GroupWise web interface had to be restarted and was unavailable between ~ 11:00 PM – 11:40 PM.   The Cold Fusion services were also restored around 00:35 AM.  We continue the recovery of other systems.

Update 01:30 AM Tuesday, Aug. 28: Alfresco services were restored at approximately 01:10 AM.   We continue working on the recovery of the RAMSS and HR systems.

Update 02:20 AM Tuesday, Aug. 28: The RAMSS services were restored at approximately 02:00 AM.   We continue working on the recovery of the HR system and we expect it to be available shortly.

Update 02:45 AM Tuesday, Aug. 28: The HR system is available as of 2:45 AM.  All services have been restored. 

Update 11:25 AM Tuesday, Aug. 28: During the outage the front-end Rmail servers that receive email continued to function while the back-end systems that store mail for each email account were offline. Before service was restored the front-end servers queued over 100,000 messages for later delivery. This morning the queues cleared and all pending mail was delivered. We do not believe any mail was lost during the outage.

We apologize for the service disruption.

-Computing and Communications Services

This entry was posted in Blackboard, eHR, my.ryerson, RMail. Bookmark the permalink.

178 Responses to Rmail, my.ryerson outages – Mon. August 27, 2012

  1. Jeremy Thomas says:

    Hi,

    My password is no longer working. I can’t get into my RMail or receive emails through my phone from this account. Is this part of the outage.

    Thank you,

    Jeremy

    • Brian Lesser says:

      Yes, this is almost certainly part of the outage. I’ll post an update as soon as I’ve heard more.

  2. Lawrence says:

    Hi,

    I assume that the outage is affecting my access to ramss and my ability to add/drop courses.

    Thanks,

    Lawrence

    • Brian Lesser says:

      Yes. That’s correct. I don’t have an estimate as to when service will be restored but I’m told we should be able to restart one of the storage appliances soon.

  3. jason nolan says:

    You folks rule. Thanks for keeping us up to date like this.

  4. Hello!
    Unfortunately, I can’t access my email, the password doesn’t work. And I can
    not access RAMSS either, and today is the day of enrollment. Please help!
    Thank you!

    • Brian Lesser says:

      I’m sorry Anastasia, our staff are working on it. We know this is a VERY bad time for this sort of problem. Once again, please accept my apologies. I’ll post an update as soon as I get more information.

  5. Hello!

    Today is a last day to enroll for the next year, please help with access to the RAMSS ASAP!!!!
    Thanks

    • Brian Lesser says:

      I’ve contacted the Registrar’s office. The initial and unofficial response I got from one person was that this is not the last day to enroll. Is there a Web page out there or something that says it is? Again, I’m sorry about today’s problems and that we still don’t have things up and running this morning.

    • Brian Lesser says:

      Just got this note from the Registrar’s office:

      “Today is the first day of Open Enrollment. Students will be able to add classes up to and including September 14, 2012.”

  6. Dawn Knight says:

    My mail password is not working but I have tried three times does that mean i will be locked out

    Courses dont appear either despite me being in one for the summer

  7. Kevin Mercado says:

    Any news yet Brian?

    And no today is just the first day to enroll for part-time students.

    But let us know asap when it is up again please thanks!

    • Brian Lesser says:

      Hi Kevin,
      Yes, and not good. We can’t just restart the storage appliance without affecting other services. So its taking time.

  8. Mervit says:

    I hope it will be working today 🙁

  9. Mervit says:

    Hope it will work today

  10. Mervit says:

    hope it will work soon!

  11. Sepideh says:

    Any estimate on the time things will be up and running again?

  12. Abdul says:

    Hey!
    sorry for bugging but do u know when the services will be back?

    do u think it gonna be today?

    • Brian Lesser says:

      Sorry no. Even if we go for an appliance restart it will take time to gracefully shut down other services that rely on the fall-back appliance. More info will be posted here when we know. Sorry this is such a complex/nasty problem.

  13. Brett says:

    Such great systems we have at ryerson not working when we have to pick courses… what a fail.. paying tuition for a crappy university system

    • Brian Lesser says:

      Sorry Brett. For what it’s worth we invested in a system that is supposed to provide redundancy so that this type of outage does not occur. If one appliance fails another is supposed to automatically take over. The systems are not inexpensive and we put a significant effort into managing them to avoid problems. Again, please accept my apologies.

  14. Hayley Fuller says:

    Just so you know, the Ryerson phone directory is down too. I have no idea if this is part of the same problem, but just fyi.

    • Brian Lesser says:

      Thanks Hayley, That’s not a surprise. The storage area network provides storage for many different systems.

  15. Quinn Wong says:

    Just a note, my.ryerson.ca wasn’t working since at least 6:45am this morning – I tried to access it then(so I could add a course on RAMSS).

  16. Revati Kinkar says:

    Correct me if I’m wrong, but I think the outage is more than just Rmail and Ramss and my.ryerson. I tried looking for the part-time calendar on the website, but I couldn’t access that either. Has there been any progress in terms of getting the sites back up and running?

    • Brian Lesser says:

      You’re right. The storage area network provides storage for many systems. Still don’t have an ETA though. We are working with the vendor of the failed hardware. To quote one of our staff: “The fastest option may be hardware replacement, but we need to be careful, because it’s very specialized equipment and not doing things right can mess things up even more.”

      • Revati Kinkar says:

        How does that affect part-time enrolment then? Most classes have already closed up before we could enrol, but now with this outage, we may not even get into evening classes..

        • Brian Lesser says:

          Good question. I have asked the Registrar’s office to discuss this with the Chang School. I’ll post here when I hear more.

          • Revati Kinkar says:

            Thank you! And as a suggestion and perhaps a compensation of sorts, is there any possible way a few spots can open up in the day time courses? I think several of my fellow part-timers who wanted to get in to day time courses would really appreciate that.

  17. Lila Pine says:

    Is this just a plot to get us to go with the cloud?

  18. Nelson says:

    Another reason why part time students and full time students should be treated equally instead of getting separated out.

  19. Sherman Griffith says:

    Everyone needs to relax. Ryerson staff will work with students to make sure they enroll in all of their courses on time

  20. Keisha says:

    Is there any estimate as to when service will be back up and working?

  21. Bianca says:

    HI,good afternoon, ah i just wanted to know if these problems with the hardware are going to affect courses that we already chosen for the fall semester, and if there is any chance that it could be fixed by today? and if not are we going to be able to pick courses for the winter after today?

    • Brian Lesser says:

      It shouldn’t have any impact on RAMSS beyond denying access as of this morning. When we are fully recovered we will check the status of the database just to make sure nothing was lost this morning. We hope to be back online some time after 2:40 but can’t predict when yet.

  22. Daniel Sarfati says:

    I am an IT infrastructure professional (as well as a part time student) and read about this…

    I’ve never heard of a SAN crash, so this must be pretty huge!

    I was also wondering about email, I know I don’t have access to webmail right now, but was wondering if emails that are being sent into my Ryerson account will be queued for later delivery.

    Thanks and good luck!

    • Brian Lesser says:

      Yes, email will be queued and delivered to your inbox later. I don’t think we’ve ever experienced an appliance fail and not fail over properly before. It is a relatively new system too.

  23. Brian Lesser says:

    With the help of the vendor, whose staff are on site, we are in the process of restoring/rebuilding the failed storage appliance. We estimate that the earliest we can be back on line is some time after 2:40 PM. It may take significantly longer. In parallel with the work to restore the failed device we are also preparing another appliance which we can reconfigure to replace the failed unit.

  24. Brian Lesser says:

    Regarding Chang School Continuing Education courses. I’ve just got this note from them:

    Today is not the deadline for enrolling. Enrollment for both the Fall 2012 and Winter 2013 term is ongoing until classes begin. There was a deadline for certificate application which was last Friday, August 24 BUT that deadline was extended to September 9. The Chang/CE website reflects the extension.

  25. Fred says:

    What is the extent of the data loss on this failed storage device?

  26. Jessica says:

    Just gave this a quick read but don’t have time to see if my question was answered already.
    My password isn’t working and when I try to log into my emails, it says service unavailable. Not only that, but I’m getting an error message on my phone saying my Ryerson password isn’t working. Could this all be related to the outage?

    Thanks,

    Jessica

  27. Rafi says:

    What will happen about class enrollment
    I was trying to enroll in classes earlier but all of them are full

  28. Ivonne Serna says:

    Thanks for the information. I tried to make a submission today into the Research Ethics Board within my.ryerson and it didn’t work as well.

    Just so you know, before the outage (last week) I tried to sign up for one of the Writing Centre workshops and it didn’t work. I tried multiple times and from different browsers, nothing worked. I called the Writing Centre to get signed-up manually and they said that’s not possible, that it has to be done online. Please help.

  29. Brian Lesser says:

    We have run into other problems rebuilding the system that failed and now think we will have a replacement appliance ready before we can repair the unit that failed. This pushes back the earliest possible time to fix the problem until after 5 PM. Once again, please accept my apologies for the prolonged outage.

  30. Margaret says:

    Thanks, by the way, for the updates; they really do help.

  31. Rafique says:

    I was expecting important e-mail for my interview. I studied Japan for five years I never experienced such problem, and here in RU I have faced many technical issues like this. RU sucks, student’s money suck.

  32. Gene Allen says:

    Are people trying to send messages to Ryerson email addresses today getting any indication of what’s going on?

    • Brian Lesser says:

      No. Right now all the mail being sent to us is being queued up on our front-end servers. So they won’t know there is a problem.

  33. Ian says:

    Will emails sent to our R-Mails today have been received successfully?

  34. Anonymous says:

    Hi Brian. Will this problem be resolved by the beginning of the school year? This is the worst time for this to be happening right now.

    • Brian Lesser says:

      Hi Ellen, if you mean before Sept 4, then yes!!! Based on what I’ve seen so far the worst-case scenario is tomorrow, though until its fixed it’s hard to make predictions. I know this is a bad time as do our staff.

  35. Muhammad Zubair says:

    Hi Brian, Just wondering will we have to enroll in courses again. Is everything or saved or it might be lost.

    • Brian Lesser says:

      You should not have to enroll again. But, please check the system after it comes back online to make sure everything is as you left it.

  36. Ukeme Noah says:

    Hey Brian, thanks a million for the updates.

  37. JMarceli says:

    I’ve tried resending an email several times but with no success. Once the issue is resolved, will the receiver receive the same email multiple times, or only once it goes through successfully? Just want to make sure as 1) It’s urgent and 2) I don’t want to spam the person’s inbox.

    Thanks.

    • Brian Lesser says:

      If you tried but could not send the email, do not assume anything worked. When the system comes back up please send a new email.

  38. Anon says:

    All the more reason to hurry up and make that google option available!

  39. John says:

    any backup system? just want see my grade for the summer?

    • Brian Lesser says:

      While the data is backed up etc, there is no separate RAMSS system we can easily bring up for view-only access right now.

  40. Scott says:

    Just wanted to say thank you Brian for keeping everyone informed and responding to people’s questions, comments and concerns.

  41. Muhammad Zubair says:

    Hi, Do you knwo when will the system be up again. I just tried nothing is still working.

  42. Skazzy B says:

    This is ABSOLUTELY unacceptable from one of Toronto’s largest universities. I have all my job applications and interview requests going to my Ryerson email account and I missed several important emails today morning. This shows a lack of a contingency plan from your CCS department.

    • joe says:

      Relax out of there control

    • jason nolan says:

      skazzy, it may show a lack of support by the university to fund CCS properly. it may show that the work CCS is doing to put in a new system, that they’ve been working on for years, was a very good move. or it could have been an unavoidable situation for which there was no ‘reasonable’ protection.

  43. Rafi says:

    Any estimated time for when everything will be back to normal?

  44. Gabby says:

    ok, it is 5 PM and nothing is working! I really want to see my summer grades!

  45. Chameera says:

    Hi,
    A quick question:
    * This Storage device failure wont affect any of my undergraduate transcripts or my grades etc. is it ? because my graduate applications are in process and I am expecting decision sometime this week.

    Thanks
    C.P

  46. Natasha says:

    Thanks for all the updates and replies, they have been super helpful.
    Things break, the best we can do is respond quickly, which it looks like is happening here.

  47. Yousef says:

    This is why you shouldn’t buy “appliances” at Walmart.

  48. Ada says:

    Seems never happened the past 10 years. Is Ryerson giving up RMail(Matrix)? Please do not. U of T does well on its Academic Mail. Ryerson should be able to catch up with them.

    • Brian Lesser says:

      U of T’s student email service is provided by Microsoft via their live@edu cloud service. Ryerson is going with Google Apps for Education but students and faculty will have the option to continue using Rmail or or to use a Ryerson Gmail account instead. In either case you will retain your ryerson email address and will not see ads in your Ryerson Gmail account. Please see our posts on http://email.blog.ryerson.ca

  49. Radmila says:

    Hi Brian,

    I as well need to acess my email account for interview information, can you please let me know if the system will work today or do I have to work until tomorro?

    Thanks.

  50. Radmila says:

    wait***, tomorrow***

  51. Brian Lesser says:

    The new storage appliance is in place but it will be many hours before all services are restored. The appliance must go through a diagnostic process, test its connections to the storage devices, and be reconfigured to provide storage for each server. Initial estimates on how long that would take are proving to be optimistic. Once everything is working again we must start the process of validating that all files and databases have been recovered correctly. From what we have seen in the past this will also take hours for some services like Rmail. Based on what we’ve seen over the last hour we can’t provide a good estimate of how long it will take to get on line this evening. CCS staff will work as long as necessary to recover all our services.

  52. Daniel says:

    Once it goes through the diagnostic process, will it automatically begin to work? Or will you guys program a set time for it to start working? Please don’t set a time because it will crash again due to overcrowding. Hopefully it will be ready later tonight!

    • Brian Lesser says:

      Recovery is not entirely automatic. Based on what we’ve seen so far it is slower than expected. So it will not be back until late this evening or in the early hours of the morning.

  53. Nelson says:

    Since we’re asking questions anyways… I’m just curious 🙂

    How much internet bandwidth does Ryerson have?
    Excluding ORION.

    • Brian Lesser says:

      1.2 Gb balanced across two separate gateways/connections. Orion/CANet is theoretically 10 Gb but under some circumstance may only be 9.4 Gb.

  54. Caroline says:

    This wait is beginning to become a little ridiculous. I have several important documents in my Rmail that I need to access by early tomorrow morning, and the deadline keeps getting pushed back. I need these documents ASAP

  55. John says:

    I am not sure why this Rmail and my.ryerson outages have been handled so unprofessionally? I am not sure if any other university student have to go through this sort of situation. This situation definitely reflects the quality of education delivered to Ryerson’s students and the unpreparedness of the Ryerson’s IT infrastructure, which is at the very least laughable.

  56. Aji says:

    How’s the progress Computing and Communications Services? Hopefully this will be up and running by tomorrow morning.

  57. Nelson says:

    Orion doesn’t count due to limited peering.

    It’s great for accessing major sites (since Orion peers at TORIX) and local sites (again due to TORIX and due to connecting to other universities) but other than it doesn’t count as a connection.

    But 1.2gbps is impressive. Waterloo only has 1 gbps.

    • Brian Lesser says:

      It turns out this week we went from 1.2 Gb to 1.6 Gb. Later this academic year we plan to go to 2.0 Gb. The upgrades are due to increasing number of WiFi connected mobile devices on campus and provisioning extra bandwidth for when we enable Google Apps.

  58. Rafique says:

    I was expecting important e-mail for my interview. I studied Japan for five years I never experienced such problem, and here in RU I have faced many technical issues like this. RU sucks, student’s money sucker.

  59. Keisha says:

    This is getting quite nervewracking. Has there been any progress in the resolution of the problem.

  60. Andrew says:

    What type of SAN are you guys using? Being in the business, I know these types of things are generally not sitting around, they are generally built to order by the manufacture.

    Good
    Cheap
    Fast

    You cannot have all 3!

    • Brian Lesser says:

      The equipment that failed was a Open Storage Solutions appliance, certified by FalconStor, to run as a FalconStor NSS device.
      The software running on this appliance is iPstor, the FalconStor virtualization software.

  61. Quinn W says:

    Thanks Brian for the updates and info. They’ve been very helpful. Great job!

  62. Sonia says:

    Brain can we have another update, when things resume, will anything have been changed due to this problem? I just haven’t seen it to go down especially for this long…

    • Brian Lesser says:

      I’m waiting for a detailed update from our staff and will post more information when I get something substantial.

  63. Lawrence says:

    Hi,

    If you guys get the system back up before midnight, we will be able to access Ramss after midnight or will the system shut down for the routine off hour maintainance (12-130am) (220-530am). Is it possible to keep ramss up for the night to help those that need to add/drop courses.

    Thx

  64. Victoria says:

    Thanks for keeping us informed, Brian. I appreciate your effort in fixing this matter asap. Do you expect Ryerson’s services to be up by tomorrow morning?

    I agree that this is unacceptable, especially due to the timing of this issue. I’m doing several interviews also and need to keep in close contact with employers, so this is making me quite frustrated.

  65. Frank N says:

    The computer staff at CCS are the best and I am sure they are working hard to get this up and running as fast as they can > Thanks CCS for keping us updated and informed about this.

  66. Ali says:

    Hi folks at CCS. I would like to thank you slong with every other person who is somehow connected with Ryerson University in the use its online services, for keeping everyone informed of the system updates. Hope everything gets back to normal soon. You guys rule!

  67. Pamela says:

    First of all. Thank you to Brian and all of the CCS team working late hours to fix this problem for Ryerson Students. It really shows how dedicated the staff is to the University.

    Second, Students who use their ryerson e-mail as a completely personal e-mail this probably is not smart. Not because this happens often or really at all but because you Ryerson e-mail is meant to be a personal inner university e-mail. IT would be much better to use gmail as an e-mail

  68. tung says:

    Omg, I can’t believe it. Too many things related to this email address and now its not working. Lesson learnt, only trust gmail

  69. Ada says:

    Pamela must be Google employee.

  70. Brian Lesser says:

    At roughly 7:30 PM we experienced a setback with restoring the configuration of the appliance. The problem means we do not have network access to the device and we are working with the vendor to continue recovering the device’s configuration. We have lost at least two hours because of this.

    I know many people would like to know when the system will be restored. We are still working towards it being back online in the late evening or early hours in the morning but we cannot predict when service will be restored.

    Once again, please accept our apologies for the problems this outage is causing.

  71. Marta Braun says:

    Brian, this is a nightmare, but you are helping us through it with the information you’re providing. Thank you. I hope you get some sleep.

  72. John says:

    Maybe virtualization is a next better solution. virtualizing from software to OS, storage to network. It can benefit recovery than rebuilding it every time hardware failure.

  73. nagina says:

    It is still not UP yet!! Any clue when this is going to be up and running. I am with RU for the last 12 years never happened like this before.

  74. Mir says:

    Time for a different vendor or redundant solution.

    • Brian Lesser says:

      The system was supposed to be redundant but fail over to the other appliance did not occur. When the smoke clears we will have some very serious conversations with the vendor and will expend significant efforts (analyzing what happened etc) to make sure this sort of prolonged outage does not happen again.

  75. Ashley says:

    I hope this is fixed soon. I have to do my course intentions for Winter!!

  76. Ben says:

    Nagina, did you read the very detailed comments from Brian before asking? Be thankful that it’s such a rare occurrence.

  77. Brian Lesser says:

    With the help of the vendor’s third-level support staff we have finally completed the automatic recovery process and are beginning the final configuration steps of the storage appliance. Network connectivity has also been established.

  78. Elyas says:

    What a great day for this outrage…today is open enrollment and i was suppose to add/drop my classes for the fall and nothing seems to be working. This is ridiculous. This is also affecting my grades…my professor was to submit my grade to the university today and now he cannot and i better get my grade on time in order for my grade change to occur! keep us updated!!!!!

  79. John says:

    So many people have been inconvenienced by this outage. Many of them are expecting or expected to make a important correspondence or transaction using the Rmail, my.ryerson portal. This much delay in restoring the service, given the fact the service went off in the peak hours of the morning, problem yet not fixed and giving inaccurate/misleading information as to the service restoration time is simply not unacceptable. I would be curious to know if the students would get anything back in return as a result of this outage and loss of this very crucial and vital service that the entire Ryerson community relies on?

    • daryl says:

      john,

      Word on the street is the ccs department is reimbursing all students atleast 35$ a piece directly from their allotted funding for the 2012-2013 term.
      Keep checking rmail for important information from the registrar

  80. Daniel says:

    I cant even log into my.ryerson at all anymore. The page wont load. Is this normal?

    • Brian Lesser says:

      It will be some time before we can bring services back online. We have to check each database and file system. This may involve rolling transactions forward based on database or file system logs.

  81. Elyas says:

    I agree with John. This has and seems like it will continue to cause an inconvenience to the students, especially since its during an important period (enrolment for classes). We need some sort of “compensation” or atleast extensions for enrollment.

  82. Brian Lesser says:

    Both storage appliances are now restored and working correctly. Staff are now checking the databases and file systems that were impacted by the outage.

  83. Ran says:

    I still cant login. Getting a uPortal error.

  84. Monika says:

    When will the rmail system be working again? I really needed to get important information from my email today.

  85. Jennifer says:

    As others seem to be forgetting, sometimes things happen which are inconvenient and yet beyond others control. I want to thank-you Brian for keeping us as best informed as you could during what was most definitely a busy and stressful time, I am happy to hear access will soon be restored. I think Ryerson is a wonderful school and has always provided me with any support necessary…including tech support when things are not going according to plan. Everyone else should relax, obviously Ryerson is aware there was a problem with the enrollment process and I am sure they will rectify the situation, if not talk to your program director regarding your timetable needs.

  86. Gloria says:

    Still not able to access either site. Crossing my fingers that it will be accessible tomorrow morning.

  87. Muhammad Zubair says:

    I cant log in to my.ryerson. The page wont even load. Please let us know when will this service resume. Thanks

  88. Elyas says:

    Brian are we able to access my.ryerson.ca because to me it says the following:

    Central Authentication Service is Unavailable
    An error occurred while complete your request on the Central Authentication Service and has been logged.

    Please try again or contact CCS Help Desk at (416) 979-5000 extension 6806.

    • Brian Lesser says:

      Thanks. It is likely because of the work staff our doing to recover systems. I have sent a note to them know that CAS is offline but I’m trying not to directly pull people away from the work they are doing.

  89. Revati Kinkar says:

    Like many comments above, I can’t log in to my.ryerson any more.

  90. Brian Lesser says:

    Regarding enrolment all I can say right now is:

    1. Today is the first day of Open Enrollment. Students will be able to add classes up to and including September 14, 2012.

    2. Regarding Chang School Continuing Education courses. Enrollment for both the Fall 2012 and Winter 2013 term is ongoing until classes begin. There was a deadline for certificate application which was last Friday, August 24 BUT that deadline was extended to September 9.

  91. Alex says:

    All the students whining in here need to calm down and stop snatching at every opportunity to demand benefits, compensation or some sort of prize for patience.

    Realize that sometimes things go wrong and just deal with it. Every major company has experienced downtime, including Google and their GMail. “IT” isn’t magic and it takes smart folks working ’round the clock to troubleshoot and fix these catastrophic issues.

    Also, please note that we have quite a few days still to fill out our course selections. So just chill everyone. Go outside and enjoy some fresh air.

  92. Mir says:

    You are doing a good job Brian.

  93. Kathryn says:

    The openness of the updates we have been receiving all day has been the saving grace in a system failure that I am sure bothered the CCS people more than it bothered any of us. While Brian has already said that redundancy was designed and the failure of redundancy will be examined, I wonder if those waiting on interview invitations will look at their own backup systems, such as providing a second email or phone number on the job application, and whether those who really need certain messages will consider copying emails on your computer instead of only on the server. Brian has provided reassurance all day that registration deadlines are NOT tomorrow, and since everyone is affected, nobody is stealing your seat while you can’t login. They will sort it out, and they’ll keep us informed.

  94. Elyas says:

    Great thanks Brian. Thank you for keeping us all updated!

  95. Alex says:

    New error: SSL certs are not kosher. Browser doesn’t see them as legit.

    “Cert is using an outdated method of encryption. Key too short”.
    Regen. longer keys.

  96. Sinéad says:

    Yes, it’s totally frustrating. BUT what’s probably more frustrating than not being able to access your mail is having a bunch of people heckle you when you’ve been spending the past sixteen hours trying to get the system up and running.

    Thank you for your hard work, Brian. I can’t even imagine the stress and pressure you’ve been dealing with today.

  97. Paras says:

    I can’t access RAMMS. It is giving the same message it did earlier – Currently Blackboard is not accessible due to a storage device hardware failure. Technical support is working on the issue to restore service.

  98. Rolandi says:

    Hey Brian, I just wanted to say, excellent job on the support. I graduated already but I still use my RMail to keep in touch with the buddies I’ve made over the years during my tenure at Ted Rogers School for Manaagement where I graduated with a 4.11 CGPA, but thats beside the point.

    Thank you brian, keep up the good work!

  99. Gerd Hauck says:

    Thanks, Brian, to you and the staff at CCS for working so incredibly hard to resolve this unfortunate mishap.

    I really appreciate the extra effort you made to keep everyone up to date on the status of the repair effort and the long hours you spent patiently responding to
    Individual inquiries and comments.

  100. Jordie says:

    Hi, my Ryerson email is still not working. I’m getting the same ‘service is not available’ message. Any estimate for how long it will be until its running again? (also, thanks for all the updates Brian)

  101. Klejdi says:

    On behalf of the Ryerson Albanian Students Association, I’d just like to say thank for the updates throughout the day Brian.

    Awesome work, you are an asset to the Ryerson Community.

  102. hamza says:

    hello brian,,, just wondering if there will be any emails lost by this incident ?

    • Brian Lesser says:

      All emails should be queued by the system and start appearing in your inbox gradually. If we discover that anything went wrong with the queuing process we’ll post an alert here.

  103. Sinéad says:

    Just a head’s up that RMail is working again.

  104. Kiaras says:

    What a catastrophic day. I had a full day of work planned, and instead I had to spend the early part of the day taking in some rays, sharing extra buttery croissants with my son, consume copious amounts of red wine, make passionate love to my stunningly beautiful wife (twice), sizzle up some steaks on the BBQ and then spend much of this evening sitting on the patio gazing at the stars and reflecting on the hardship brought upon me by this outage. I can only hope that tomorrow I can once again partake in the joys of reading and responding to emails, uploading course materials and despairing over the brevity of the 16 hour work day. Adding fire to my fury is the knowledge that all the CCS people did today was show up to work at 8am, give it their all to resolve the problem, cancel whatever plans they may have had for tonight and possibly work well into the wee hours. The gull some people have…

  105. Michael says:

    still cant access RAMSS….

  106. Daniel says:

    Is Blacboard working? You said it has been restored but it won’t load.

  107. Lawrence says:

    What a show off LOL youre pathetic

  108. David says:

    I had a question about RMail, the emails(inbox) we got during the outage… Are they lost or will they be delivered/recovered later. Thanks for the help!

  109. Gerald Mak says:

    Thanks CCS and all the others working around the clock in getting the system back up.

    -Gerald Mak

  110. Alex says:

    rms-web.ryerson.ca is still sending bad SSL certificate over.

  111. Amrinder says:

    Although RMAIL has started working, but can you please confirm whether the incoming messages have been queued or we have lost the messages. I have started receiving the current messages, so was a bit worried. Cheers

    • Brian Lesser says:

      All messages should have been queued and should have started trickling into your inbox by now. If we discover any evidence of lost mail we will post an update here.

  112. Petra says:

    hi Brian

    Thanks for the updates. I have managed to log into eHR and noticed in the process monitor that a number of batch processes ran automatically but are not posting. Is Carmen available to run the XRFWIN process and kick start the posting mechanism. Thanks, Petra

  113. Petra says:

    hi Brian

    I gather my earlier comment at 3:26am did not get through to you as the status showed ‘Your comment is awaiting moderation’. I was not able to groupwise you directly & knew RMail would be ineffective given the backlog hence why I posted here.

    Anyways, not to worry, I got ahold of Carmen at 4:45am and he has rectified the posting issue…. Petra

  114. Petra says:

    Morning Brian … Thanks for passing info. Could you please let them know that the situation was just rectified. Timing 😉 … Petra

  115. Brian Lesser says:

    Regarding the problems people experienced earlier with CAS and uPortal. We had to restart services on the second storage appliance while recovering the first. That caused a problem with the database server used by CAS and uPortal. We recovered those services around 11:47 PM.

  116. Quin says:

    Thank you for letting me know when RAMSS was back up! I’ve enrolled in the courses I wanted! Thanks for all the hard work CCS!

  117. Louise says:

    Is registration for the Chang School still down? When I go do a course search (for the Publishing Program Courses) All of the courses state that they have no classes scheduled for the Fall and Winter semesters. But I am registered for some of those courses in the fall already and I want to register for the winter term.

    Thank you

    • Brian Lesser says:

      From what I can see RAMSS is working fine. I’m about to pass your question on to a bunch of people who can look into this better than I can.

    • Brian Lesser says:

      RAMSS sends the Chang School web site course availability information. It looks like the RAMSS service that provides the information is down and we are working on getting it back into service.

      Thanks for letting us know about this!

  118. Brian Lesser says:

    Course Availability information service on RAMSS is back up. Chang School Web site should be ok now.

  119. Gabriel says:

    Hi Brian,

    Hr instance of cf.ryerson is still returning 503 errors.

  120. Charles Tam says:

    IT often doesn’t get recognition or even acknowledgment for the 99.9% of the time where they “keep the lights on” but when the lights go out, they get 100% of the blame. That said, thank you to the Ryerson IT team for pulling together, keeping us users appraised and for resolving this issue as quickly as possible. It was highly visible and complicated but you all pulled together towards a resolution. Great job and thank you! -C.Tam

  121. Judy Waalen says:

    Thank you for keeping us in the loop while you and your team were restoring our service.

  122. Vanessa Tyrell says:

    Hi,
    I’m experiencing an issue with the system saying I am not enrolled in any courses which is not the case. I was just wondering if this is a result of the system failure?

    Thank you

  123. Hei-Lok Fung says:

    I still can’t get access to RAMSS. It said, “Currently Blackboard is not accessible due to a storage device hardware failure. Technical support is working on the issue to restore service. ”

    Please help fix the problem. Thanks.

    • Brian Lesser says:

      I just checked RAMSS and it seems ok. My guess is the problem is browser cache. Please try clearing your browser cache and trying again?

  124. Hei-Lok Fung says:

    I now can get access to RAMSS by using another computer. Sorry for the trouble caused.

Comments are closed.