Black Wednesday
No email. No internet. No website. No access to documents stored on the server. Last Wednesday, the ITER server crashed leaving the Organization homebase in Cadarache disconnected from the outside world — for 24 long hours. The only communication gateway was the good old telephone.
As the exchange of information and data via the usual line proved impossible, groups gathered in the corridors, speculating on what might have happened and on how long this interruption would last. And on how dependent on modern web-based communication tools we have become.
But what exactly happened? "A series of coincidences", is the short answer from Hans-Werner Bartels, Senior Technical Officer for Information Technology. The long version: that morning, a thunderstorm passed over Cadarache. One lightning strike must have hit the power circuit of the CEA installation which led to a power cut. In the Tore Supra building, where the French tokamak's and the 150 ITER servers are housed, the lights went out.
"Usually, a short power cut is not a problem", explains Bartels. "The nuclear installations on the CEA site have generators to step in — not so Tore Supra. The only backup for the fusion device are some UPS (Uninterrupted Power Supplies) batteries that can feed the demand for about 30 minutes. The only problem was, that the power switch that should have flicked over from the grid to the batteries, did not work. "This has not happened in over 20 years", says Bartels, who, together with seven members of his team, worked until well after midnight trying to fix the problem.
"Lesson learned", he sums up. "We will have to make sure that we don't run into this same embarrassing situation again. Therefore we should make sure that we have a diesel generator as a back up system for the new office building where the new servers will be stored. Also we are currently discussing with CEA to keep the servers here on site, as a second back-up system."
As "bad times have a scientific value", to quote the American poet Ralph Waldo Emerson, some colleagues used the exceptional peace and quiet to write that long postponed paper, or to clean up their desks. Emerson is right, something good always comes out of the bad...