ITER Scientific Data Centre comes online

The Scientific Data and Computing Centre is designed to host up to 1 MW of equipment in 46 racks, constituting both the ITER high-performance computer (HPC) and the storage and communication systems used to manage all the project’s data and transfer it to an offsite backup location.

The equipment will be relocated from its temporary home to its permanent home in the Scientific Data and Computer Centre during 2025. In the process, the HPC will be expanded to 17,000 cores. Both the temporary and the permanent locations are in the basement of ITER Headquarters. The new data centre will be energized to 50% capacity in the beginning, increasing gradually as needed. The first system, appropriately named “IBM Fusion,” has already been installed.

The design of the computing infrastructure began in 2019, with the objective of meeting Tier 3 data centre standards which, among other things, call for no more than 1.6 hours of annual downtime of the central infrastructure such as power and cooling. This is a benchmark, and hopefully the design at ITER’s will exceed that part of the requirements, keeping downtime to an absolute minimum.

To ensure maintenance can be performed on all central infrastructure without taking it offline, the data centre has multiple independent power distribution paths and redundant components for power and cooling. “We can lose one complete powertrain in the building and still do servicing, while we continue to operate using the other one,” says Peter Kroul, Computing Centre Officer. “And the same goes for generators, connectivity, and all other critical elements.”

Providing tremendous computing power and massive data storage efficiently

The ITER HPC is composed of multiple clusters designed to handle computationally intensive jobs such as Monte Carlo simulations. Neutronics, which are used to analyze neutron flows within a facility, are a key application of Monte Carlo methods. For nuclear safety, whenever the design of any part of the Tokamak Building is changed, neutronics have to be recalculated for the new structure. 

“Monte Carlo methods work particle by particle, tracking probabilities for every neutron individually,” explains David Fernandez, IT Systems and Operations Section Leader. “This involves a huge number of repetitions of the same operations, exactly the kind of intense jobs high-performance computers are designed to perform.”

In addition to the Monte Carlo algorithms, the ITER computer will be used to perform plasma physics simulations, fluid dynamics and several other compute-intensive jobs required for both science and engineering. But in most cases, the software needs to be tweaked first. “Simulations, and most other jobs that run on the high-performance computer, require source code modifications to exploit the full capacity of the large machine,” says Fernandez.

The ITER Scientific Data and Computing Centre will also fulfill one of the high-level requirements for the ITER project, which is to store, secure, process and distribute the vast amount of data produced throughout its lifetime. The new data centre will start with storage capacity in the tens of petabyte scale, using IBM’s Spectrum Scale Storage System. But the centre is designed to store an estimated 5 exabytes of data, which the project is expected to produce over time.¹

“To hold the complete data of the project throughout its duration, we are very much dependent on the continuous evolution of storage systems in terms of capacity and density,” explains Kroul. “Fortunately, we had access to technology roadmaps from some of the largest storage equipment providers in the world, which afforded us a degree of comfort in our projections. As an example, we could store 5 to 6 petabytes in a rack when we started in 2019. That number is now closer to 20 petabytes six years later—and expected to triple over the next three years.”

The main control room, where operations will be monitored and analyzed, will have 10 days of data on hand for very fast access and analysis. All data will be streamed into the data centre, which will be the vault of ITER—keeping data in an online state throughout the lifetime of the project to allow scientists to perform analysis across the different experimental phases. According to Kroul, this differs from other large-scale experiments where data is kept online only during certain phases and then passed on to “cold” storage, where it can no longer be accessed quickly. “Here the requirement from the scientists is that they must be able to perform analyses across all phases, which results in a different storage design,” he says. “Instead of using cold tiering, we will keep everything in a hot state, which means on disk or similar throughout the project.”

The data centre will be connected to a backup and distribution data centre over 50 km away in Marseille, using dedicated high-speed fiber optics, which are also redundant. Data storage and management will already begin in 2025 when the centre starts receiving data from various ITER plant systems.

Mindful of sustainability, the designers of the Scientific Data and Computing Centre included in-row cooling, where servers and storage units are chilled by towers of liquid-cooling air conditioning units placed between racks. The power consumption and the amount of cooling can be optimized for each individual rack inside the facility.

The efficiency of a data centre is typically measured with the power usage effectiveness (PUE) metric, which is a ratio of the total energy used in a data centre to the energy used by the actual computing equipment in a data centre. “I believe the ITER centre will have a PUE value below 1.3, which is decent in terms of energy and cooling consumption,” says Kroul.

¹A petabyte is equal to one million gigabytes. An exabyte is equal to one thousand petabytes, or one million gigabytes.