Polski

Todays BigData solution implementations faces many problems, especially when it comes to hardware and software management of entire clusterbrain. Administrators are looking for software familiar to vCenter in Vmware culsters, which they could use to administer their whole HPC and BigData Cluster containg hundreds or even thausands of nodes. This article fucuses on HP CMU Software, which can be implemented to administer BigData Apache Hadoop clusters.

 

 

Looking at Our 10 years HPC experience we figure Out, that the best solution for HPC Administrator is to have unified servers and storage base which can be managed from one management window. This is the place where HP CMU comes in.

Main HP CMU management functions:

  • – firmware patching
  • – automatic new hardware add/remove
  • – management and creation of OS golden images (i.e. HPC nodes, Storage Nodes, Mgmnt Nodes etc.)
  • – deployment of images on new hardware
  • – hardware and OS monitoring
  • – visualization of hardware and OS statuses, load, etc

The same situation applies to BigData clusters, where we have similar kinds of nodes like Worker Node, Name node etc. HP and Cloudera worked together to integrate HP solutions with Cloudera Apache Hadoop Software Solution. It turned Out that integrating together HP Hardware with Cloudera Enterprise and HP Insight Cluster Management Utility ended with very agile and robust solution, which gives Administrator full control over their HPC Big Data Clusters.

Best HP hardware for Hadoop clusters:

  • – HPE Apollo 2000 servers for High Performance Interactive/SQL workloads
  • – HPE Apollo 6000 or DL 580 gen9 servers for High Memory and GPU intensive workloads
  • – HPE Moonshot servers for specialized workloads like ARM, FPGA, GPU  workloads
  • – HPE Apollo 4500 servers for storage nodes and capacity based workloads

 

The most interesting fact is that all of above hardware can be managed by one software, from one window – HP Cluster Management Utility.

 

What is exactly HP Insight Cluster Management Utility ?

space

 

HP Insight Cluster Management Utility (CMU) is an efficient and robust hyper-scale cluster lifecycle management framework and suite of tools for large Linux clusters such as those found in High Performance Computing (HPC) and Big Data environments. A simple graphical interface enables an “at-a-glance” real-time or 3D historical view of the entire cluster for both infrastructure and application (including Hadoop) metrics, provides frictionless scalable remote management and analysis, and allows rapid provisioning of software to all nodes of the system.

a

 

We recommend using HP CMU for all types of Apache Hadoop installations. You can use it with MapR, HortonWorks, Cloudera or open source Apache Hadoop cluster. It allows one to have comparison of Hadoop metrics with cluster infrastructure metrics, such as CPU Utilization, Network Transmit/Receive, Memory Utilization and I/O Read/Write. In addition, Insight CMU also greatly simplifies the deployment of Hadoop, with its ability to create a Golden Image from a node and then deploy that image to up to 4000 nodes. Insight CMU is able to deploy 800 nodes in 30 minutes.

We have implemented HP CMU Clusters with big success. It manages clusters with more than one hundred nodes. We also created proof-of-concepct  implementation of CMU with Cloudera in Our LAB to check how it works. It confirmed that implementing HP CMU with HP hardware and Hadoop is very good, agile and easy to manage solution.

You can read more about HP CMU in Our other post.