VMware has been experiencing a performance degradation due to the migration of the storage from one storage RAID array to another. Technical staff are working with the vendor to determine the cause of the problem. Some virtual servers have been migrated to another RAID array. Updates will be provided periodically until the problem is resolved.
Update 1: We continue to experience performance degradation. A few virtual machines have been migrated to a different storage RAID system, and we will continue this process overnight.
Update 2 (Monday, Jan 21, 11:30pm): We are still experiencing performance issues, but a number of virtual servers have been migrated to a different RAID storage system where performance is good. The migration process is slow because the READs off the current storage is slow. We’re scheduling a number of batch migrations over night.
Update 3 (Tuesday, Jan 22, 11:20pm): Some performance issues are still present, but we have migrated many virtual servers (VMs) from the affected RAID system to a different RAID system where the performance is good. The migration process is continuing overnight. The vendors continue to investigate and are narrowing down the source of the problem. We expect the impact to be minor on Wednesday morning.
Final update (Friday, Jan 25, 8:45 am): By Tuesday evening, overall performance had improved. We continued the migration of virtual machines overnight and services behaved normally throughout the day. The migration of the virtual machines completed last night. A failing IO module on the disk array was replaced on Wednesday, but vendor support continues their investigation. However, the VMware infrastructure will stay on the new storage array system and will not be migrated back.
We apologize for any inconvenience this problem caused.
– Computing and Communications Services