Early this morning a device on our Storage Area network failed. As a result disk storage for Rmail, my.ryerson, Blackboard, and other systems went offline. (Other services like RAMSS and eHR are not reachable when my.ryerson is down.) We are investigating the cause and working to restore service. Updates to follow.
Update 8:20 AM: A storage appliance that manages the storage connections to the Rmail, my.ryerson and other servers failed. A second storage appliance is available as a backup but for some reason the servers were not able to tolerate the storage connectivity failures before the second device took over.
Update 9:36 AM: The problem seems to be related with a failed controller on one of the storage appliances that provides storage virtualization. The vendors are working on the problem and replacement parts are being sent to us.
In the meantime, we’re also working on bringing up services on the backup appliance, with the idea that we will be able to “fail over” to the other appliance (in a controlled way) when it is working properly.
Update 12:37 PM: With the help of the vendor, whose staff are on site, we are in the process of restoring/rebuilding the failed storage appliance. We estimate that the earliest we can be back on line is some time after 2:40 PM. It may take significantly longer. In parallel with the work to restore the failed device we are also preparing another appliance which we can reconfigure to replace the failed unit.
Update 2:21 PM: We have run into other problems rebuilding the storage appliance that failed and now think we will have a replacement appliance ready before we can repair the unit that failed. This pushes back the earliest possible time to fix the problem until after 5 PM. Once again, please accept my apologies for the prolonged outage.
Update 5:40 PM: The new storage appliance is in place but it will be many hours before all services are restored. The appliance must go through a diagnostic process, test its connections to the storage devices, and be reconfigured to provide storage for each server. Initial estimates on how long that would take are proving to be optimistic. Once everything is working again we must start the process of validating that all files and databases have been recovered correctly. From what we have seen in the past this will also take hours for some services like Rmail. Based on what we’ve seen over the last hour we can’t provide a good estimate of how long it will take to get on line this evening. CCS staff will work as long as necessary to recover all services.
Update 9:30 PM: At roughly 7:30 PM we experienced a setback with restoring the configuration of the appliance. The problem means we do not have network access to the device and we are working with the vendor to continue recovering the device’s configuration. We have lost at least two hours because of this.
I know many people would like to know when the system will be restored. We are still working towards it being back online in the late evening or early hours in the morning, but at this time, we cannot predict when service will be restored.
-Brian
Update 10:54 PM: With the help of the vendor’s third-level support staff we have finally completed the automatic recovery process and are beginning the final configuration steps of the storage appliance. Network connectivity has also been established.
Update 11:17 PM: Both storage appliances are now restored and working correctly. Staff are now checking the databases and file systems that were impacted by the outage.
Update 00:40 AM Tuesday, Aug. 28: A few systems have now been recovered and services are available. CAS, uPortal services were restored around 00:20 AM, Blackboard was restored around 00:25 AM. RMAIL was restored around 00:45 AM. The GroupWise web interface had to be restarted and was unavailable between ~ 11:00 PM – 11:40 PM. The Cold Fusion services were also restored around 00:35 AM. We continue the recovery of other systems.
Update 01:30 AM Tuesday, Aug. 28: Alfresco services were restored at approximately 01:10 AM. We continue working on the recovery of the RAMSS and HR systems.
Update 02:20 AM Tuesday, Aug. 28: The RAMSS services were restored at approximately 02:00 AM. We continue working on the recovery of the HR system and we expect it to be available shortly.
Update 02:45 AM Tuesday, Aug. 28: The HR system is available as of 2:45 AM. All services have been restored.
Update 11:25 AM Tuesday, Aug. 28: During the outage the front-end Rmail servers that receive email continued to function while the back-end systems that store mail for each email account were offline. Before service was restored the front-end servers queued over 100,000 messages for later delivery. This morning the queues cleared and all pending mail was delivered. We do not believe any mail was lost during the outage.
We apologize for the service disruption.
-Computing and Communications Services
Hi,
My password is no longer working. I can’t get into my RMail or receive emails through my phone from this account. Is this part of the outage.
Thank you,
Jeremy
Yes, this is almost certainly part of the outage. I’ll post an update as soon as I’ve heard more.
Hi,
I assume that the outage is affecting my access to ramss and my ability to add/drop courses.
Thanks,
Lawrence
Yes. That’s correct. I don’t have an estimate as to when service will be restored but I’m told we should be able to restart one of the storage appliances soon.
You folks rule. Thanks for keeping us up to date like this.
Hello!
Unfortunately, I can’t access my email, the password doesn’t work. And I can
not access RAMSS either, and today is the day of enrollment. Please help!
Thank you!
I’m sorry Anastasia, our staff are working on it. We know this is a VERY bad time for this sort of problem. Once again, please accept my apologies. I’ll post an update as soon as I get more information.
Hello!
Today is a last day to enroll for the next year, please help with access to the RAMSS ASAP!!!!
Thanks
I’ve contacted the Registrar’s office. The initial and unofficial response I got from one person was that this is not the last day to enroll. Is there a Web page out there or something that says it is? Again, I’m sorry about today’s problems and that we still don’t have things up and running this morning.
Hi Brian, maybe this will help: http://www.ryerson.ca/currentstudents/importantdates/
Just got this note from the Registrar’s office:
“Today is the first day of Open Enrollment. Students will be able to add classes up to and including September 14, 2012.”
My mail password is not working but I have tried three times does that mean i will be locked out
Courses dont appear either despite me being in one for the summer
When the service is back up you should be ok.
Any news yet Brian?
And no today is just the first day to enroll for part-time students.
But let us know asap when it is up again please thanks!
Hi Kevin,
Yes, and not good. We can’t just restart the storage appliance without affecting other services. So its taking time.
I hope it will be working today 🙁
Hope it will work today
hope it will work soon!
Any estimate on the time things will be up and running again?
Hey!
sorry for bugging but do u know when the services will be back?
do u think it gonna be today?
Sorry no. Even if we go for an appliance restart it will take time to gracefully shut down other services that rely on the fall-back appliance. More info will be posted here when we know. Sorry this is such a complex/nasty problem.
Such great systems we have at ryerson not working when we have to pick courses… what a fail.. paying tuition for a crappy university system
Sorry Brett. For what it’s worth we invested in a system that is supposed to provide redundancy so that this type of outage does not occur. If one appliance fails another is supposed to automatically take over. The systems are not inexpensive and we put a significant effort into managing them to avoid problems. Again, please accept my apologies.
Just so you know, the Ryerson phone directory is down too. I have no idea if this is part of the same problem, but just fyi.
Thanks Hayley, That’s not a surprise. The storage area network provides storage for many different systems.
Just a note, my.ryerson.ca wasn’t working since at least 6:45am this morning – I tried to access it then(so I could add a course on RAMSS).
Correct me if I’m wrong, but I think the outage is more than just Rmail and Ramss and my.ryerson. I tried looking for the part-time calendar on the website, but I couldn’t access that either. Has there been any progress in terms of getting the sites back up and running?
You’re right. The storage area network provides storage for many systems. Still don’t have an ETA though. We are working with the vendor of the failed hardware. To quote one of our staff: “The fastest option may be hardware replacement, but we need to be careful, because it’s very specialized equipment and not doing things right can mess things up even more.”
How does that affect part-time enrolment then? Most classes have already closed up before we could enrol, but now with this outage, we may not even get into evening classes..
Good question. I have asked the Registrar’s office to discuss this with the Chang School. I’ll post here when I hear more.
Thank you! And as a suggestion and perhaps a compensation of sorts, is there any possible way a few spots can open up in the day time courses? I think several of my fellow part-timers who wanted to get in to day time courses would really appreciate that.
Is this just a plot to get us to go with the cloud?
Hi Lila! No, we’re already going Google. Not trying to make you opt in to Gmail.
Another reason why part time students and full time students should be treated equally instead of getting separated out.
Couldn’t agree more!
Everyone needs to relax. Ryerson staff will work with students to make sure they enroll in all of their courses on time
Is there any estimate as to when service will be back up and working?
HI,good afternoon, ah i just wanted to know if these problems with the hardware are going to affect courses that we already chosen for the fall semester, and if there is any chance that it could be fixed by today? and if not are we going to be able to pick courses for the winter after today?
It shouldn’t have any impact on RAMSS beyond denying access as of this morning. When we are fully recovered we will check the status of the database just to make sure nothing was lost this morning. We hope to be back online some time after 2:40 but can’t predict when yet.
I am an IT infrastructure professional (as well as a part time student) and read about this…
I’ve never heard of a SAN crash, so this must be pretty huge!
I was also wondering about email, I know I don’t have access to webmail right now, but was wondering if emails that are being sent into my Ryerson account will be queued for later delivery.
Thanks and good luck!
Yes, email will be queued and delivered to your inbox later. I don’t think we’ve ever experienced an appliance fail and not fail over properly before. It is a relatively new system too.
With the help of the vendor, whose staff are on site, we are in the process of restoring/rebuilding the failed storage appliance. We estimate that the earliest we can be back on line is some time after 2:40 PM. It may take significantly longer. In parallel with the work to restore the failed device we are also preparing another appliance which we can reconfigure to replace the failed unit.
Regarding Chang School Continuing Education courses. I’ve just got this note from them:
Today is not the deadline for enrolling. Enrollment for both the Fall 2012 and Winter 2013 term is ongoing until classes begin. There was a deadline for certificate application which was last Friday, August 24 BUT that deadline was extended to September 9. The Chang/CE website reflects the extension.
What is the extent of the data loss on this failed storage device?
Just gave this a quick read but don’t have time to see if my question was answered already.
My password isn’t working and when I try to log into my emails, it says service unavailable. Not only that, but I’m getting an error message on my phone saying my Ryerson password isn’t working. Could this all be related to the outage?
Thanks,
Jessica
Yes. This is because of the outage.
What will happen about class enrollment
I was trying to enroll in classes earlier but all of them are full
I don’t think this outage will have any impact on course sections being full.
Thanks for the information. I tried to make a submission today into the Research Ethics Board within my.ryerson and it didn’t work as well.
Just so you know, before the outage (last week) I tried to sign up for one of the Writing Centre workshops and it didn’t work. I tried multiple times and from different browsers, nothing worked. I called the Writing Centre to get signed-up manually and they said that’s not possible, that it has to be done online. Please help.
I’ll look into the writing centre problem. It sounds like a separate issue.
We have run into other problems rebuilding the system that failed and now think we will have a replacement appliance ready before we can repair the unit that failed. This pushes back the earliest possible time to fix the problem until after 5 PM. Once again, please accept my apologies for the prolonged outage.
Thanks, by the way, for the updates; they really do help.
Agreed. Having these updates are really great. Thank you so much!
I was expecting important e-mail for my interview. I studied Japan for five years I never experienced such problem, and here in RU I have faced many technical issues like this. RU sucks, student’s money suck.
Are people trying to send messages to Ryerson email addresses today getting any indication of what’s going on?
No. Right now all the mail being sent to us is being queued up on our front-end servers. So they won’t know there is a problem.
Will emails sent to our R-Mails today have been received successfully?
Yes.
Hi Brian. Will this problem be resolved by the beginning of the school year? This is the worst time for this to be happening right now.
Hi Ellen, if you mean before Sept 4, then yes!!! Based on what I’ve seen so far the worst-case scenario is tomorrow, though until its fixed it’s hard to make predictions. I know this is a bad time as do our staff.
Hi Brian, Just wondering will we have to enroll in courses again. Is everything or saved or it might be lost.
You should not have to enroll again. But, please check the system after it comes back online to make sure everything is as you left it.
Hey Brian, thanks a million for the updates.
I’ve tried resending an email several times but with no success. Once the issue is resolved, will the receiver receive the same email multiple times, or only once it goes through successfully? Just want to make sure as 1) It’s urgent and 2) I don’t want to spam the person’s inbox.
Thanks.
If you tried but could not send the email, do not assume anything worked. When the system comes back up please send a new email.
All the more reason to hurry up and make that google option available!
any backup system? just want see my grade for the summer?
While the data is backed up etc, there is no separate RAMSS system we can easily bring up for view-only access right now.
Just wanted to say thank you Brian for keeping everyone informed and responding to people’s questions, comments and concerns.
Hi, Do you knwo when will the system be up again. I just tried nothing is still working.
This is ABSOLUTELY unacceptable from one of Toronto’s largest universities. I have all my job applications and interview requests going to my Ryerson email account and I missed several important emails today morning. This shows a lack of a contingency plan from your CCS department.
Relax out of there control
skazzy, it may show a lack of support by the university to fund CCS properly. it may show that the work CCS is doing to put in a new system, that they’ve been working on for years, was a very good move. or it could have been an unavoidable situation for which there was no ‘reasonable’ protection.
Any estimated time for when everything will be back to normal?
ok, it is 5 PM and nothing is working! I really want to see my summer grades!
Hi,
A quick question:
* This Storage device failure wont affect any of my undergraduate transcripts or my grades etc. is it ? because my graduate applications are in process and I am expecting decision sometime this week.
Thanks
C.P
No. It should be fine when it comes back online.
Thanks for all the updates and replies, they have been super helpful.
Things break, the best we can do is respond quickly, which it looks like is happening here.
This is why you shouldn’t buy “appliances” at Walmart.
Seems never happened the past 10 years. Is Ryerson giving up RMail(Matrix)? Please do not. U of T does well on its Academic Mail. Ryerson should be able to catch up with them.
U of T’s student email service is provided by Microsoft via their live@edu cloud service. Ryerson is going with Google Apps for Education but students and faculty will have the option to continue using Rmail or or to use a Ryerson Gmail account instead. In either case you will retain your ryerson email address and will not see ads in your Ryerson Gmail account. Please see our posts on http://email.blog.ryerson.ca
Hi Brian,
I as well need to acess my email account for interview information, can you please let me know if the system will work today or do I have to work until tomorro?
Thanks.
wait***, tomorrow***
Could be in the early hours tomorrow. It’s proving very hard to estimate.
The new storage appliance is in place but it will be many hours before all services are restored. The appliance must go through a diagnostic process, test its connections to the storage devices, and be reconfigured to provide storage for each server. Initial estimates on how long that would take are proving to be optimistic. Once everything is working again we must start the process of validating that all files and databases have been recovered correctly. From what we have seen in the past this will also take hours for some services like Rmail. Based on what we’ve seen over the last hour we can’t provide a good estimate of how long it will take to get on line this evening. CCS staff will work as long as necessary to recover all our services.
Once it goes through the diagnostic process, will it automatically begin to work? Or will you guys program a set time for it to start working? Please don’t set a time because it will crash again due to overcrowding. Hopefully it will be ready later tonight!
Recovery is not entirely automatic. Based on what we’ve seen so far it is slower than expected. So it will not be back until late this evening or in the early hours of the morning.
Since we’re asking questions anyways… I’m just curious 🙂
How much internet bandwidth does Ryerson have?
Excluding ORION.
1.2 Gb balanced across two separate gateways/connections. Orion/CANet is theoretically 10 Gb but under some circumstance may only be 9.4 Gb.
This wait is beginning to become a little ridiculous. I have several important documents in my Rmail that I need to access by early tomorrow morning, and the deadline keeps getting pushed back. I need these documents ASAP
I am not sure why this Rmail and my.ryerson outages have been handled so unprofessionally? I am not sure if any other university student have to go through this sort of situation. This situation definitely reflects the quality of education delivered to Ryerson’s students and the unpreparedness of the Ryerson’s IT infrastructure, which is at the very least laughable.
How’s the progress Computing and Communications Services? Hopefully this will be up and running by tomorrow morning.
Orion doesn’t count due to limited peering.
It’s great for accessing major sites (since Orion peers at TORIX) and local sites (again due to TORIX and due to connecting to other universities) but other than it doesn’t count as a connection.
But 1.2gbps is impressive. Waterloo only has 1 gbps.
It turns out this week we went from 1.2 Gb to 1.6 Gb. Later this academic year we plan to go to 2.0 Gb. The upgrades are due to increasing number of WiFi connected mobile devices on campus and provisioning extra bandwidth for when we enable Google Apps.
I was expecting important e-mail for my interview. I studied Japan for five years I never experienced such problem, and here in RU I have faced many technical issues like this. RU sucks, student’s money sucker.
This is getting quite nervewracking. Has there been any progress in the resolution of the problem.
What type of SAN are you guys using? Being in the business, I know these types of things are generally not sitting around, they are generally built to order by the manufacture.
Good
Cheap
Fast
You cannot have all 3!
The equipment that failed was a Open Storage Solutions appliance, certified by FalconStor, to run as a FalconStor NSS device.
The software running on this appliance is iPstor, the FalconStor virtualization software.
Thanks Brian for the updates and info. They’ve been very helpful. Great job!
Brain can we have another update, when things resume, will anything have been changed due to this problem? I just haven’t seen it to go down especially for this long…
I’m waiting for a detailed update from our staff and will post more information when I get something substantial.
Hi,
If you guys get the system back up before midnight, we will be able to access Ramss after midnight or will the system shut down for the routine off hour maintainance (12-130am) (220-530am). Is it possible to keep ramss up for the night to help those that need to add/drop courses.
Thx
Sorry, we do not plan to run RAMSS for extra hours in the evening.
Thanks for keeping us informed, Brian. I appreciate your effort in fixing this matter asap. Do you expect Ryerson’s services to be up by tomorrow morning?
I agree that this is unacceptable, especially due to the timing of this issue. I’m doing several interviews also and need to keep in close contact with employers, so this is making me quite frustrated.
The computer staff at CCS are the best and I am sure they are working hard to get this up and running as fast as they can > Thanks CCS for keping us updated and informed about this.
Hi folks at CCS. I would like to thank you slong with every other person who is somehow connected with Ryerson University in the use its online services, for keeping everyone informed of the system updates. Hope everything gets back to normal soon. You guys rule!
First of all. Thank you to Brian and all of the CCS team working late hours to fix this problem for Ryerson Students. It really shows how dedicated the staff is to the University.
Second, Students who use their ryerson e-mail as a completely personal e-mail this probably is not smart. Not because this happens often or really at all but because you Ryerson e-mail is meant to be a personal inner university e-mail. IT would be much better to use gmail as an e-mail
Omg, I can’t believe it. Too many things related to this email address and now its not working. Lesson learnt, only trust gmail
Pamela must be Google employee.
At roughly 7:30 PM we experienced a setback with restoring the configuration of the appliance. The problem means we do not have network access to the device and we are working with the vendor to continue recovering the device’s configuration. We have lost at least two hours because of this.
I know many people would like to know when the system will be restored. We are still working towards it being back online in the late evening or early hours in the morning but we cannot predict when service will be restored.
Once again, please accept our apologies for the problems this outage is causing.
Brian, this is a nightmare, but you are helping us through it with the information you’re providing. Thank you. I hope you get some sleep.
Maybe virtualization is a next better solution. virtualizing from software to OS, storage to network. It can benefit recovery than rebuilding it every time hardware failure.
It is still not UP yet!! Any clue when this is going to be up and running. I am with RU for the last 12 years never happened like this before.
Time for a different vendor or redundant solution.
The system was supposed to be redundant but fail over to the other appliance did not occur. When the smoke clears we will have some very serious conversations with the vendor and will expend significant efforts (analyzing what happened etc) to make sure this sort of prolonged outage does not happen again.
I hope this is fixed soon. I have to do my course intentions for Winter!!
Nagina, did you read the very detailed comments from Brian before asking? Be thankful that it’s such a rare occurrence.
With the help of the vendor’s third-level support staff we have finally completed the automatic recovery process and are beginning the final configuration steps of the storage appliance. Network connectivity has also been established.
What a great day for this outrage…today is open enrollment and i was suppose to add/drop my classes for the fall and nothing seems to be working. This is ridiculous. This is also affecting my grades…my professor was to submit my grade to the university today and now he cannot and i better get my grade on time in order for my grade change to occur! keep us updated!!!!!
So many people have been inconvenienced by this outage. Many of them are expecting or expected to make a important correspondence or transaction using the Rmail, my.ryerson portal. This much delay in restoring the service, given the fact the service went off in the peak hours of the morning, problem yet not fixed and giving inaccurate/misleading information as to the service restoration time is simply not unacceptable. I would be curious to know if the students would get anything back in return as a result of this outage and loss of this very crucial and vital service that the entire Ryerson community relies on?
john,
Word on the street is the ccs department is reimbursing all students atleast 35$ a piece directly from their allotted funding for the 2012-2013 term.
Keep checking rmail for important information from the registrar
I cant even log into my.ryerson at all anymore. The page wont load. Is this normal?
It will be some time before we can bring services back online. We have to check each database and file system. This may involve rolling transactions forward based on database or file system logs.
I agree with John. This has and seems like it will continue to cause an inconvenience to the students, especially since its during an important period (enrolment for classes). We need some sort of “compensation” or atleast extensions for enrollment.
Both storage appliances are now restored and working correctly. Staff are now checking the databases and file systems that were impacted by the outage.
I still cant login. Getting a uPortal error.
When will the rmail system be working again? I really needed to get important information from my email today.
As others seem to be forgetting, sometimes things happen which are inconvenient and yet beyond others control. I want to thank-you Brian for keeping us as best informed as you could during what was most definitely a busy and stressful time, I am happy to hear access will soon be restored. I think Ryerson is a wonderful school and has always provided me with any support necessary…including tech support when things are not going according to plan. Everyone else should relax, obviously Ryerson is aware there was a problem with the enrollment process and I am sure they will rectify the situation, if not talk to your program director regarding your timetable needs.
Still not able to access either site. Crossing my fingers that it will be accessible tomorrow morning.
I cant log in to my.ryerson. The page wont even load. Please let us know when will this service resume. Thanks
Brian are we able to access my.ryerson.ca because to me it says the following:
Central Authentication Service is Unavailable
An error occurred while complete your request on the Central Authentication Service and has been logged.
Please try again or contact CCS Help Desk at (416) 979-5000 extension 6806.
Thanks. It is likely because of the work staff our doing to recover systems. I have sent a note to them know that CAS is offline but I’m trying not to directly pull people away from the work they are doing.
Like many comments above, I can’t log in to my.ryerson any more.
Regarding enrolment all I can say right now is:
1. Today is the first day of Open Enrollment. Students will be able to add classes up to and including September 14, 2012.
2. Regarding Chang School Continuing Education courses. Enrollment for both the Fall 2012 and Winter 2013 term is ongoing until classes begin. There was a deadline for certificate application which was last Friday, August 24 BUT that deadline was extended to September 9.
All the students whining in here need to calm down and stop snatching at every opportunity to demand benefits, compensation or some sort of prize for patience.
Realize that sometimes things go wrong and just deal with it. Every major company has experienced downtime, including Google and their GMail. “IT” isn’t magic and it takes smart folks working ’round the clock to troubleshoot and fix these catastrophic issues.
Also, please note that we have quite a few days still to fill out our course selections. So just chill everyone. Go outside and enjoy some fresh air.
You are doing a good job Brian.
The openness of the updates we have been receiving all day has been the saving grace in a system failure that I am sure bothered the CCS people more than it bothered any of us. While Brian has already said that redundancy was designed and the failure of redundancy will be examined, I wonder if those waiting on interview invitations will look at their own backup systems, such as providing a second email or phone number on the job application, and whether those who really need certain messages will consider copying emails on your computer instead of only on the server. Brian has provided reassurance all day that registration deadlines are NOT tomorrow, and since everyone is affected, nobody is stealing your seat while you can’t login. They will sort it out, and they’ll keep us informed.
Great thanks Brian. Thank you for keeping us all updated!
New error: SSL certs are not kosher. Browser doesn’t see them as legit.
“Cert is using an outdated method of encryption. Key too short”.
Regen. longer keys.
Yes, it’s totally frustrating. BUT what’s probably more frustrating than not being able to access your mail is having a bunch of people heckle you when you’ve been spending the past sixteen hours trying to get the system up and running.
Thank you for your hard work, Brian. I can’t even imagine the stress and pressure you’ve been dealing with today.
I can’t access RAMMS. It is giving the same message it did earlier – Currently Blackboard is not accessible due to a storage device hardware failure. Technical support is working on the issue to restore service.
Hey Brian, I just wanted to say, excellent job on the support. I graduated already but I still use my RMail to keep in touch with the buddies I’ve made over the years during my tenure at Ted Rogers School for Manaagement where I graduated with a 4.11 CGPA, but thats beside the point.
Thank you brian, keep up the good work!
Thanks, Brian, to you and the staff at CCS for working so incredibly hard to resolve this unfortunate mishap.
I really appreciate the extra effort you made to keep everyone up to date on the status of the repair effort and the long hours you spent patiently responding to
Individual inquiries and comments.
Hi, my Ryerson email is still not working. I’m getting the same ‘service is not available’ message. Any estimate for how long it will be until its running again? (also, thanks for all the updates Brian)
On behalf of the Ryerson Albanian Students Association, I’d just like to say thank for the updates throughout the day Brian.
Awesome work, you are an asset to the Ryerson Community.
hello brian,,, just wondering if there will be any emails lost by this incident ?
All emails should be queued by the system and start appearing in your inbox gradually. If we discover that anything went wrong with the queuing process we’ll post an alert here.
Just a head’s up that RMail is working again.
What a catastrophic day. I had a full day of work planned, and instead I had to spend the early part of the day taking in some rays, sharing extra buttery croissants with my son, consume copious amounts of red wine, make passionate love to my stunningly beautiful wife (twice), sizzle up some steaks on the BBQ and then spend much of this evening sitting on the patio gazing at the stars and reflecting on the hardship brought upon me by this outage. I can only hope that tomorrow I can once again partake in the joys of reading and responding to emails, uploading course materials and despairing over the brevity of the 16 hour work day. Adding fire to my fury is the knowledge that all the CCS people did today was show up to work at 8am, give it their all to resolve the problem, cancel whatever plans they may have had for tonight and possibly work well into the wee hours. The gull some people have…
still cant access RAMSS….
Is Blacboard working? You said it has been restored but it won’t load.
What a show off LOL youre pathetic
I had a question about RMail, the emails(inbox) we got during the outage… Are they lost or will they be delivered/recovered later. Thanks for the help!
Thanks CCS and all the others working around the clock in getting the system back up.
-Gerald Mak
rms-web.ryerson.ca is still sending bad SSL certificate over.
Although RMAIL has started working, but can you please confirm whether the incoming messages have been queued or we have lost the messages. I have started receiving the current messages, so was a bit worried. Cheers
All messages should have been queued and should have started trickling into your inbox by now. If we discover any evidence of lost mail we will post an update here.
hi Brian
Thanks for the updates. I have managed to log into eHR and noticed in the process monitor that a number of batch processes ran automatically but are not posting. Is Carmen available to run the XRFWIN process and kick start the posting mechanism. Thanks, Petra
Hi Petra, I’ve passed your question on to Dave and Leo. -B
hi Brian
I gather my earlier comment at 3:26am did not get through to you as the status showed ‘Your comment is awaiting moderation’. I was not able to groupwise you directly & knew RMail would be ineffective given the backlog hence why I posted here.
Anyways, not to worry, I got ahold of Carmen at 4:45am and he has rectified the posting issue…. Petra
Morning Brian … Thanks for passing info. Could you please let them know that the situation was just rectified. Timing 😉 … Petra
Thanks Petra. Will do.
Regarding the problems people experienced earlier with CAS and uPortal. We had to restart services on the second storage appliance while recovering the first. That caused a problem with the database server used by CAS and uPortal. We recovered those services around 11:47 PM.
Thank you for letting me know when RAMSS was back up! I’ve enrolled in the courses I wanted! Thanks for all the hard work CCS!
Is registration for the Chang School still down? When I go do a course search (for the Publishing Program Courses) All of the courses state that they have no classes scheduled for the Fall and Winter semesters. But I am registered for some of those courses in the fall already and I want to register for the winter term.
Thank you
From what I can see RAMSS is working fine. I’m about to pass your question on to a bunch of people who can look into this better than I can.
RAMSS sends the Chang School web site course availability information. It looks like the RAMSS service that provides the information is down and we are working on getting it back into service.
Thanks for letting us know about this!
Course Availability information service on RAMSS is back up. Chang School Web site should be ok now.
Hi Brian,
Hr instance of cf.ryerson is still returning 503 errors.
Thanks Gabriel, I’ll follow up with Paul et al.
-B
IT often doesn’t get recognition or even acknowledgment for the 99.9% of the time where they “keep the lights on” but when the lights go out, they get 100% of the blame. That said, thank you to the Ryerson IT team for pulling together, keeping us users appraised and for resolving this issue as quickly as possible. It was highly visible and complicated but you all pulled together towards a resolution. Great job and thank you! -C.Tam
Thank you for keeping us in the loop while you and your team were restoring our service.
Hi,
I’m experiencing an issue with the system saying I am not enrolled in any courses which is not the case. I was just wondering if this is a result of the system failure?
Thank you
Thanks Vanessa,
I’ll ask folks in the Registrar’s office to check into it.
I still can’t get access to RAMSS. It said, “Currently Blackboard is not accessible due to a storage device hardware failure. Technical support is working on the issue to restore service. ”
Please help fix the problem. Thanks.
I just checked RAMSS and it seems ok. My guess is the problem is browser cache. Please try clearing your browser cache and trying again?
I now can get access to RAMSS by using another computer. Sorry for the trouble caused.
No problem and thanks for the update!