High Performance Clustering Part 4

For the first time in this series, we dwell into two completely different clustering techniques. We see how to build a clustered NAS as well as preview the first commodity HPC initiative from Microsoft

Saturday, April 08, 2006

Print

Comment

Email

Digg

Del.icio.us

Reddit

Twitter

This time instead of doing a single topic on clustering, we've actually done two. One is based on our good old 20-nodes cluster, and is about a topic we promised to bring to you two months ago. It's about building a storage cluster, which is basically a NAS running across multiple commodity machines. While the idea sounds adventurous, its setup is equally challenging, so get ready for some adrenaline pumping action. Our second topic is a preview of an upcoming technology, called the Microsoft Compute Cluster 2003. This is Microsoft's first initiative to bring out a software to create HPC clusters out of commodity hardware. The final release is expected sometime in June this year.

Building a clustered NAS
We'll use one of the most popular Open Source NAS Operating Systems called OpenFiler and build a custom kernel for it, which has a OpenMosix patch. This will enable us to add multiple compute nodes to the NAS box, which can make any kind of batch processing on the device faster. You can go for aggregating the storage of all the boxes for the NAS devices by using MFS. But we won't advise that because this will reduce the data redundancy significantly. But if you have hardware or software RAID configured on all the cluster nodes, you can also try that out.

The installation
The first time we did an article on OpenFiler was about a year and half ago, and at that time, it was so primitive that the only way of installing it was by first installing a RedHat 8.0 installation and then installing a huge set of RPMs manually on top of that. But today OpenFiler has its own customized distribution. This distribution is based on CentOS and comes on a single CD. You can download it from http://www.openfiler.com/download/ . Installing it is pretty easy and the installation looks very much similar to the installer of PCQLinux/Fedora. All you have to do is boot the machine with the CD and follow the graphical installer. One thing you have to keep in mind is that for OpenFiler to work properly you will have to create the partitions manually. Basically you have to create three partitions. But before that remember to delete all other existing partitions. The partitions you have to create are /boot, swap and /. While creating the partitions make sure that both the /boot and / are in ext3 format.

While preparing the partitions for openfiler, make sure that you have all the partitions exactly in this format

The size of the /boot partition should be at least 128 MB and the size of the swap partition should be double the size of the amount of RAM you have in your machine. And the size of the / (root) partition should be at least 4 GB. Remember that this partition is just for the OS to work. The partitions and disks that will be used over the NAS should be different.

After you are done with the partitioning, follow the installation exactly as you do with the PCQLinux installation. The installation will take around an hour depending on your machine's speed.

Now start the following services as follows:

Now your first openfiler node is up and running. The next thing to do is to build an OpenMosix kernel on top of your open file node.

Installing OpenMosix 
The openfiler installer CD doesn't comes with kernel source and for installing OpenMosix on top of it you will need to recompile the kernel. So first of all you have to find out the exact version of kernel that's running on the OpenFiler OS. For this run the following command

This will show you the exact version of the kernel of Openfiler you are using. Now what you have to do is to first download the source of the exact kernel version. You can find the exact source from http://kernel.org. After you're done with this you have to download the openmosix tarball. This is basically a kernel patch. But here one thing you have to make sure is that you should download the openmosix patch of the exact version of kernel, which you have with your OpenFiler.

The next step is to recompile the kernel with the openmosix kernel patch. For this follow the steps given below

Step 1--Uncompress the Kernel source tar ball by running the following commands:

Step 2--Get all the settings and configurations from the old kernel to this new kernel source file by running the following commands

Step 3--Now patch the kernel with openmosix kernel patch by running the following commands

Step 4--The next step is to compile the kernel with the patch and all the old configurations from the Openfiler kernel. For that run the following commands

For creating new volumes with Openfiler, click on the 'Create' link and provide the volume details

And this should complete the installation part. Now you have to reboot the machine. While rebooting, you will see a new entry in your grub menu. Select this menu and boot the machine normally.

Now your one node is ready with the OpenFiler which has a openmosix kernel patch. the next thing you have to do is to install the usermode and openmosixview tools on top of this machine and you will be able to add new nodes as well monitor your Open Filer Cluster. 

To do so, download the userland tools from http://prdownloads.sourceforge.net/openmosix/openmosix-tools-0.2.4-1.i386.rpm and downlaod openmosixview from http://www. openmosixview.com/download/openmosixview-1.5-redhat90.i386.rpm   

And install them with the following commands

Installing other nodes
Now your first node of the cluster is ready. For adding new nodes, you have two ways. Either use the openmosix live CDs and boot the other cluster nodes using that. Or you can install any flavor of Linux which have the same kernel version which the OpenFiler node has and install the openmosix-kernel patch of the same kernel rpm on top of that.

Monitor the cluster 
To monitor your cluster, start X-Window on any cluster node that has the openmosixview rpm installed, and run the following command:

This will pop-up a window that will show a list of all the servers in the cluster, the amount of RAM and processor utilization in each. From here you can configure the load-balancing efficiency of the cluster. Each node has a slider bar that you can move to do the load balancing. When you do this, openMosix will automatically adjust the processes running on each machine. It will automatically migrate them to another node if the existing node is overloaded. You can also migrate the processes manually from one node (processor) to another by selecting the processes option from the toolbar. It will show you a list of the processes that are running. By double-clicking on any process you'll get another window in which you can choose the node you want to migrate the process to.

Only if your Windows 2003 server is a domain controller, can you enable the option for installing a real node

Using Openfiler
Now your openfiler cluster is ready to manage your storage network. To start working with it, fire up any Web browser and open the link https://yourmachinename:445, where 'yourmachinename' stands for the host name of the machine where openfiler is running. Remember to use https as openfiler works over a secured SSL connection. On the first screen it will ask you for the username and password. Use 'openfiler' as the username and 'password' as password. After logging into it you will find a neat and easy to use storage-management workplace. From here you can manage the existing volumes and create new volumes. You can also manage users and can set disk quota for these users. These users can be of any form, such as LDAP, CIFS, Kerprose and NIS .

In open source and Linux world there used to be quite a few Clustering solutions available and that too of different types. Such as Load balancing and HPC (parallel processing) clusters. Load balancing clusters are those which can run any batch operation over a cluster infrastructure and don't need any special modification in the software while HPC are the clusters which need customized applications that are aware of MPI Libraries/APIs and then can run accordingly.

MS Compute Cluster
Microsoft has released its first commodity hardware based computer cluster environment. The softwares are still in their beta stage. So, we decided to give it an early look. This is mainly an HPC (parallel processing) type cluster network and needs the applications to be aware of the MPI API. 

The MS Compute Cluster Pack provides support for MPI2 libraries. It also contains an integrated job scheduler and the cluster resource management tools. MPI is a standard application programming interface (API) and specification for message passing. It has been designed specifically for high-performance computing scenarios executed on large computer systems or on clustered commodity computers.

It uses the MS MPI which is an MPI version of the Argonne National Labs Open Source MPI2 implementation that is widely used by existing HPC clusters. MS MPI is compatible with the MPICH2 Reference Implementation and other MPI implementations and supports a full-featured API with more than 160 function calls.

The MS Visual Studio 2005 also includes support for developing HPC applications such as parallel compiling. And this I think is the best part that will turn out to be a great plus point for Microsoft's compute cluster initiative. The reason being that, now a developer will get a familiar interface for developing HPC programs and subsequently deploy and run them on familiar environments.  

Requirements
All you require is a two CD set of Microsoft Compute Cluster 2003 which includes a specialized Windows 2003 x64 version called MS Windows 2003 Compute Cluster Edition and the Compute Cluster Pack.

Windows Server 2003 Compute Cluster Edition is a specialized 64-bit Windows Server OS based on the 64-bit edition of Windows Server 2003 to support high-performance software. Windows Server 2003 Compute Cluster Edition is a full version of the Windows Server 2003 64-bit operating system. However, it is not intended to be used as a general purpose server.

You can also install the Compute Cluster Pack on top of any Windows 2003 Server edition for a 64-bit architecture.

Installation
The installation of MS CC consists of two parts. Creating a Head node and the Compute nodes. The Head Node acts as a server and all the management is done from this node.It is very easy to install. All you have to do is to take a machine which has a 64-bit processor and then install MS Windows 2003 CC edition on it. Subsequently, you can either make it a domain controller by running dcpromo.exe or add it into an existing Domain. After this is done, install the  MS CC Pack on it. After you start the installation of the Compute Cluster Pack, it will ask you whether you want to install the components for the Head node, Compute node or Administrative nodes. Insert the Compute Cluster Pack CD and let the autorun start. Now when the wizard starts, select the first option which says 'Create a New Compute Cluster with this node as a Head node' and proceed with the installation wizard.

This wizard will install around seven components out of which three will be installed from the CD and the rest of the four from the Internet (first download and then install). It will take around 15 to 20 minutes to complete the installation. When the installation is over, you can go to StartàProgramsàMicrosoft Compute Cluster Pack and click on Microsoft Compute Cluster Pack. Here, you will see a window where the total number of processors available in the cluster and the number of jobs running are shown. But at this time as there are no nodes connected to the cluster, the software will only see the resources available on the Head node.

In this window you can see the summary of your cluster like number of CPUs, total jobs, etc

Installing Compute Nodes
After you are finished with the creation of the Head node, the next thing you have to do is to install and add the Compute nodes. For this, you will need another set of 64-bit machines (numbers will depend on how big you want to build your cluster; for our test we created a two node cluster) having Windows 2003 Server Edition or Cluster Edition installed on all of its machines. 

Now you have to install the Compute Cluster Pack on all the machines. For this, you can either use an RIS server or do it manually. We created a small two node cluster, and so,  we preferred to install it manually. For this, place the CC Pack in the CD drive of the node machine and let autorun start. When the wizard window pops up, select the second option which says 'Join this cluster to an existing Compute Cluster as a Compute node.' Selecting this option would enable the 'Enter the name of the Compute Cluster Head node' option. Here, enter the fully qualified domain name of the Head node and follow the wizard to complete the installation.

Adding the Compute Nodes
After the installation is over, you have to add the nodes to the cluster. To achieve this, go to the Head node and open the Cluster Management console. Now, from the Node Management pane, click on the Add Nodes button. Also, on the option 'Before You Begin Page,' click Next. On the Select Add Node Method page, select manual addition. This will now ask you for the FQDN of the node machines. After you provide the necessary details it will ask for the Administrator password and once you are done with the nodes will be installed.

The node you have added will be displayed in the Node Management page as 'Pending for Approval.' Approve or reject the node for inclusion to the cluster by right-clicking that node and choosing Accept.

This month we have seen how to build a Commodity Cluster using MS Compute Cluster Pack. Next month, we will see how to build an MPI compliant application using Visual Studio and run it on the cluster. 

Anindya Roy

Page(s)   1