Polski

Projects

Computer cluster Prometheus.

Story of three innovations creating the most powerful computer cluster in Poland

On April 27, 2015 the Prometheus computing cluster was officially launched in Academic Computing Center CYFRONET (ACC CYFRONET) in Krakow, Poland. It is currently the most powerful computer cluster in Poland (#30 in the world TOP500 in November 2015). Its computational power is more than 1.6 PFlops. It is composed of over 1.7 thousand twin-processor servers connected by high- speed InfiniBand network. And it was built in record time - 16 weeks!

PROMETHEUS will serve Polish academic community in conducting complicated scientific calculations.

 

Prometheus

 

In the history of MEGATEL PROMOTHEUS is the biggest project, both in financial and technological terms. Formally, MEGATEL was the leader of the consortium MEGATEL - ACTION, that won the public tender. Nevertheless, the success of the project is the result of tremendous work and commitment of all companies and individuals involved in the project: team of ACC CYFRONET – the Client, a contractor for cooling system - ZETO PROJECT, engineers and managers of HP Poland - representing the manufacturer and ACTION – the distributor and funding provider. The team working on the project counted more than 30 people both in Poland and the USA. The project started earlier than the formal signing of the agreement and continued for 16 weeks. In the MEGATEL we spent more than 3,000 hours on the design and running the project.

Skills, knowledge and money met at the same time and place to give the groundbreaking effect.

 

Why cluster PROMETHEUS is unique?

Contrary to appearances, this is not about the computing power, because that is just millions of dollars on the way to get Petaflops results.

The uniqueness of this cluster is built on the innovative way we have chosen to tackle the three most important challenges in high-power computing: cooling, aging of an equipment and density in racks.

 

Innovative cooling

In terms of heat reception HP Apollo 8000 uses two important innovations: airtight racks and hot water. HP Apollo 8000 is, in fact, several thermos-shaped rack cabinets. The heat escapes from cabinets through pipes with water. There is no need for external air cooling. In the effect he whole installation needs only 6% more energy than the cluster itself (PUE - 1.06). The result is a 90% (!) saving in the cost of cooling, resulting in million dollars savings during lifetime of the installation.

There is no need to cool the water before it is pumped into the cabinets, it can be as warm as 30 degrees Celsius. When it is circulating in the cluster it is heated to over 40 degrees Celsius, which is even better – it can be further used for other purposes (heating, domestic hot water) or cooled down in coolers on the roof top of the building – which is additional cost saving.

See here the movie about the most important technical details:

 

 

For those interested in how we reached high tightness in water-cooling system and what is dry contact, see more here.

 

Pro-longed lifetime

Prometheus will have the opportunity to be the first system used for longer than the lifetime of components used in it. After removing servers and switches one can get a very efficient cooling infrastructure with capacity of more than 80kW per cabinet. It is enough to install a new generation of components to update the installation. Whether that will happen, it depends on how popular this solution will become in global scale as for now Prometheus is one of only four HP Apollo 8000 clusters worldwide.

 

Power of place

Place in the server room is as valuable as the cost of its construction. In HP Apollo 8000 we have achieved the record-breaking density of racks. In total, it is 144 2P servers in a single, only 60 cm wide, rack (including InfiniBand switches and Ethernet). If we count additional cabinets with pumps that operate in redundancy, the entire cluster is located in 9-meter long corridor.

Prometheus is a new generation of hardware. Energy efficient, occupying little space, suitable for re-use and producing sustainable – ready to re-use heat. We are proud that we have participated in its construction.

We wish ACC CYFRONET world-shaking scientific results on the scale of the computing power of Prometheus and ourselves - trouble-free maintenance of the installation.