TOI Status Information

InfoBase unavailable 20 february

Thijs Welman — Mon, 20 Feb 2012 10:15:47 +0000

Our infoBase and portfolio services on server SRV275 are currently experiencing problems. We’re investigating the issue.

Update: The server is back up and running. The webserver was experiencing unusual high load which caused a snowball effect.

InfoBase down 16 february

Thijs Welman — Thu, 16 Feb 2012 07:22:30 +0000

Our infoBase server is down. It appears it hangs in the reboot cycle after scheduled installation of updates. We’ve seen this problem before and we’re awaiting action by the ICT department. We expect the machine to be back around 8:30.

Update: the server is back up and running.

InfoBase down 19 January

Thijs Welman — Thu, 19 Jan 2012 08:42:32 +0000

InfoBase (SRV275) was not available this morning. The machine seems to be stuck in a reboot cycle. Our ICT department is investigating.

Update 9:40: The server is back up and running.

InfoBase down for server migration

Thijs Welman — Tue, 20 Dec 2011 21:31:25 +0000

We’re moving InfoBase to a new server. This will cause the InfoBase website to be unavailable on Wednesday December 21st. We expect the main services to be available again around noon. The portfolio system will also be unavailable; handing in work will not be possible during the maintenance.

Update 12:30: InfoBase is up and running. However we still have some difficulties accessing the system from our other webserver(s). The submission portals and portfolio therefore are still unavailable. We’re waiting for our ICT department to make some changes to the firewall.

Update 12:50: ICT made the necessary changes. Everything is operational now. We’ll monitor the system the coming time for any problems that may arise.

BKTOI3 unavailable

Thijs Welman — Mon, 21 Nov 2011 10:16:38 +0000

BKTOI3 (blog.bk.tudelft.nl and adhok.bk.tudelft.nl) is unavailable since Sunday 20 November due to a problem at the central ICT infrastructure. The problem is being investigated by the TU Delft ICT engineers. Part of the storage infrastructure seems to be malfunctioning.

Update 21/11 12:00 – The problem still persists. Unfortunately we don’t have an estimated time to fix.

Update 16:00 – ICT confirmed that a piece of critical storage hardware has crashed and is malfunctioning. The vendor is working on a solution.

Update 18:00 – Several hard disks have crashed simultaneously. Chances are that the only option is to restore the system to a previous state in time from backups. All data (updates) since last Friday will probably be lost.

Update 22/11 13:00 – Still no news from our ICT department. We don’t have an estimate when the server will be back up and running again…

Update 16:00 – Just heard there are no backups of our affected server. They should be there, but the server was by mistake not included in the backup schedule. Chances are all data is lost. We’re making plans to restore the systems as soon as possible (but without the original content)

Update 24/11 – A glimmer of hope: The crashed hard-disk have been sent to the manufacturer and there is a bit more hope some or all data can be restored. We should know more on Friday. In the meanwhile most important functionality has been restored on a new server.

Update 25/11 – They did it! Seagate has been able to fix the broken hard disk and the ICT managed to get the old storage array online just long enough to get some crucial data. The data from our server has been restored and copied to a new storage system, which means our server is back up and running. We will be doing some rollbacks and merges in the coming days. The adhok website has been restored already.

infoBase down 14 November

Thijs Welman — Mon, 14 Nov 2011 20:21:17 +0000

We’re experiencing some difficulties on the infoBase server (SRV854) around 21:15. We’re investigating the problem.

Update 21:24: The server is back up and running again. Due to a unusual high load, the server became unresponsive. We’re monitoring the situation.

Renderfarm interruption 17 October

Thijs Welman — Mon, 17 Oct 2011 10:54:23 +0000

The renderfarm service was interrupted due to a render job that crashed. We’re bringing back the renderfarm now.

Update: Renderfarm is fully functional again. Several jobs were processed without any problems.

Renderfarm Maintenance 13 October

Thijs Welman — Thu, 13 Oct 2011 10:03:29 +0000

The website for the renderfarm will be briefly unavailable due to planned maintenance between 11:00 and 12:00. Also some of the rendernodes will be unavailable. Main operations should be unaffected,

Update: All maintenance has been completed. One of the nodes will remain unavailable due to a hardware problem that was discovered during the maintenance.

Update 17 Oct: The faulty controller battery has been replaced. All nodes are up and running again.

Renderfarm Maintenance 15-22 July

Thijs Welman — Sat, 09 Jul 2011 11:01:06 +0000

Due to a scheduled update and general maintenance, the renderfarm will be unavailable between 15 and 18 July.

The renderfarm is being updated to a new version of the render management software (Muster). Furthermore we will upgrade to a new version of Maya: Maya 2012-x64.

Update 18 July: Maintenance complete; the renderfarm is available for rendering again. The upgrade to Maya 2012 has been postponed due to a problem with the new license.

Renderfarm unavailable June 15

Thijs Welman — Wed, 15 Jun 2011 08:52:13 +0000

Solved – The dispatcher- and file server of the renderfarm has gone down today around 10:40, so the renderfarm is unavailable at this moment. We’re currently investigating the outage.

Update 11:35 – We notified the ICT department of the outage. It’s still being investigated.
Update 11:45 – The system had crashed. It has been restored.