Outages and Maintenance
-
Update As of 5:50 pm, Tuesday, 16 Aug 2016, the Radon cluster has been returned to service and is fully operational. Thank you for your patience. Update Due to unanticipated conflicts between the upgraded scheduler and our network configuration, the...
-
Emergency Maintenance for EXRC Datacenter
Update The EXRC system has been temporarily relocated while plumbing repairs and mold abatement are being completed. The system has been returned to service and is fully operational. Once the repairs are complete and the system can be returned to it...
-
The maintenance is complete and github.rcac is back in production, now at version 2.7.3. Apart from the usual patches and bugfixes, this is also a feature release. Release notes can be found at: https://enterprise.github.com/releases/2.7.3/notes Than...
-
Degraded performance of several systems
We have seen a significant wave of these events this morning, September 21. For the most part, this wave seems to have been linked to a storage problem that has been resolved. However, we are implementing new monitoring and response procedures toda...
-
Unscheduled scratch outage on Carter
UPDATE: ITaP engineers have implemented a temporary solution so that work may continue on Carter until the scheduled upcoming maintenance window on Tuesday. Any jobs running which were using the scratch space have been stopped in order to allow for t...
-
Emergency maintenance for GitHub
This afternoon, at 5:00pm EDT github.rcac.purdue.edu was taken down for brief emergency maintenance. GitHub announced a critical security vulnerability this week. As a result, our GitHub instance was taken down for a few minutes of patching. Users of...
-
Home Filesystem Maintenance - All Clusters
Conte has been returned to normal operations as well now. This concludes the home directory maintenance on all systems. Update: September 27, 2016 11:55pm All systems other than Conte have been successfully returned to normal operations with the ne...
-
The Conte cluster will be unavailable beginning at September 27, 2016 7:00am - September 28, 2016 11:59pm EDT, for Home Filesystem Maintenance - All Clusters. The cluster will return to partial production by midnight that day, but will remain at re...
-
Firebox Virtual Server Maintenance
This work was largely postponed due to other concerns. A few related systems were relocated successfully during this time. Original Message: The hardware powering all Firebox Virtual Servers will be unavailable beginning at Tuesday, September 27th,...
-
Unscheduled Scratch Outage on Carter
UPDATE As of about 6:30 pm, the new scratch system was brought back online, and scheduling has been restarted on Carter. Original Message The new scratch filesystem serving Carter that was just activated on Tuesday night is currently unavailable. Bot...
-
Measures taken within the first two hours of this problem seem to have resolved the issue. Original Message: A portion of the systems serving the Research Data Depot have suffered a failure. Some systems using Depot have been affected, particularly...
-
GitHub issues - Internal Server Error and Web Interface Not Updating
The issue with the GitHub web interface was resolved late yesterday evening. The website is now reflecting changes made to git repositories as normal. Please let us know at rcac-help@purdue.edu if you see any continuing issues. The sporadic "Int...
-
This issue has been resolved. Original Message: A portion of the systems serving the Research Data Depot have suffered a failure. Some systems using Depot have been affected, particularly Carter, Snyder, and systems accessing Depot over NFS. Some sys...
-
The EXRC cluster has been returned to normal operations as of Tuesday, November 1st, 2016 at 8:10pm EDT. Please let us know if you experience any problems. Thank you. Original Message: The EXRC cluster will be unavailable beginning at Tuesday, Nove...
-
The Carter Cluster was returned to production at 10:45pm on November 7. We apologize for this extended outage. Update: November 7, 2016 6:01pm Work on reinstalling the Carter nodes continues. All other systems have returned normal operations. We...
-
The Data Depot has concluded successfully, and it been returned to normal operations as of Sunday, November 6th, 2016 at 1:30pm EST. Original Message: The Data Depot will be taken down for emergency maintenance beginning at Saturday, November 5th, 20...
-
Emergency Firebox Virtual Server Maintenance
The Firebox maintenance work concluded successfully as of Saturday, November 5th, 2016 at 11:30pm EDT. Original Message: All Firebox Virtual Servers will be taken down for emergency maintenance beginning at Saturday, November 5th, 2016 at 8:00am EDT....
-
Job scheduling paused on Radon
Job scheduling was paused on Radon between 6 pm and 7 pm this evening. Node monitoring processes marked most nodes offline around 6 pm, preventing new jobs from starting. System engineers cleared the fault in the node monitoring, and nodes came back...
-
Update: Engineers were able to isolate the problem and restart the necessary systems. The Data Depot should be available again. Halstead users should double check their running work. A portion of the systems serving the Research Data Depot have suffe...
-
The Halstead cluster will be unavailable beginning at Wednesday, December 7th, 2016 at 1:00pm EST, for scheduled early-access maintenance (see Halstead Cluster Early Access Policies). The cluster will return to full production by Wednesday, December...