Now You can afford a Supercomputer in your Enterprise

Saturday, January 21, 2006

Print

Comment

Email

Digg

Del.icio.us

Reddit

Twitter

From universities to the enterprise, from high end to mid level. Machines that can deliver very high computing powers is no longer limited to a lucky few. Ordinary hardware can now be combined into clusters to give the same power. And enterprises can now utilize the same technologies to improve the productivity of their applications manifold. Starting this month, we will take you through how to set up these clusters and how to run useful applications on them

Three and a half years back, in the May 2002 issue, for the first time, we talked about how to build your own supercomputer. Since then, many changes have happened in the supercomputing/clustering space. Supercomputing, like a studious child, having passed its childhood in colleges and universities has now grown up and is ready to face the corporate world.

Till some time back, supercomputers were affordable only by research facilities, universities or government departments like defense and space research, due to their cost as well as the complexities involved in setting up and running these systems. One would go as far to guess that it was more the latter than the cost which kept many enterprises away from using supercomputers for meeting their computing requirements.

But that is now in the past. Today, adopting easy-to-use clustering technologies and software along with low cost, commodity hardware, creating a supercomputer (or one that is almost there) is very easy.

The technology and the implementation have become so simple that there are those who have networked two or three home PCs and notebooks to create simple clusters so that they can rip their music faster. Obviously, building a high-performance cluster is not at all rocket science! And using one impacts the productivity immensely (not to count the bragging rights).

So, starting with this issue, we are dedicating some of our pages to take you on a journey to understand where and how such systems can be used in the enterprise. We will tell you how to build one and how to put it to good use.

Let's start off on this journey by understanding the various the types of clusters.

Clusters?
You must have noticed that we have used the terms cluster (rather high-performance clusters) and supercomputer synonymously.  Why? Before that, to bring in a sense of perspective, let us look briefly at the evolution of supercomputers. A supercomputer is loosely designed as one of the fastest computers available.

Remember that today's fastest ones may be left far behind tomorrow.

The earliest of supercomputers were built up of scalar processors. Scalar processors are those that process only one item at a time, as against vector processors which can process multiple items in parallel. Most CPUs traditionally do scalar processing while GPUs are vectors. Modern CPUs include some vector capabilities. The second generation of supercomputers were again monolithic machines, but built using vector processors. The third generation is where the monolithic architecture gave way to the current parallel design, with a large super computer being built of many smaller units. In the beginning of the cycle, even these building blocks were actually specialized, high-performance units. It is only recently that cheap, off the shelf building blocks started getting used to make high performance machines, or high-performance clusters.

So, what is a cluster?
A cluster is a set of computers that are interconnected (networked) to perform as one. High performance computing is just one of the things that a cluster can do. You can have failover clusters, load-balancing clusters, and so on, using the same commodity hardware.

Here, the server is fully loaded, while the cluster is only under a 10% load (simultaneously converting 75 WAV files to OGG). The performance deference is about 50% less time on the cluster.

Here the cluster is loaded up to full load and the same load was applied on the server too (zipping and taring 55 MB files in batches of 100, 150 and so on up to 300. The cluster reached full load at 250 files). The cluster gives about 6 times better performance.

In this series, the term cluster will mean a HPC (High Performance Cluster) and not a Fail Safe or other cluster unless specified explicitly.

Within high performance clusters, you can have two types of setups.

SSI based clusters
No, we are not recommending clusters to small scale industries, at least not yet. SSI (Single System Image) is a clustering technology which can use a number of nodes (say n) on a network and make them work like a single virtual machine with 'n' processors. The best thing about SSI is that it doesn't require any specific modifications to your applications to run on the new virtual machine. But because of this, it has some drawbacks too. SSI works very well when you run many tasks simultaneously on the virtual machine, for instance, like converting hundreds of media files from one format to another. In such a situation, the SSI cluster will migrate all tasks evenly to all the machines available in the cluster and complete the job significantly faster than if it were on a single machine.

On the other hand, if you deploy a single job or thread which requires a large amount of number crunching, then SSI cluster will not give you the performance improvement. This is because it cannot divide a single task into multiple threads and spread them across the nodes of the cluster.

An example of clustering middleware for SSI is OpenMosix.

The charm of SSI based clusters is that you can deploy any standard (Linux) application on the cluster. And it doesn't require any modification to be made in the application. So for enterprises which want to migrate an existing application (mainly batch processing applications) on to a cluster, but don't want to invest in re-creating their applications with the PVM/MPI library, or are using third-party application where they do not have access to the code, then SSI based clusters are the best solution for them.

Talking of Linux running these clusters, SSI based clusters offer another benefit. There are some live distributions available which can convert an existing network into an SSI cluster almost on the fly, and once the application has run, you can reboot back into the original environment and continue working as before. In this case you do not even have to invest for any additional hardware to build your cluster. Your existing network is already the cluster.

This kind of live CD based approach is best for those who need huge computing power temporarily and particularly for testing whether your applications will benefit from being deployed on a cluster.

There are some projects currently on, using which let you create a cluster out of a heterogeneous environment. We will explore one such project later on in this article.

What are the typical applications you can put an SSI cluster to do? You could speed up your batch processing jobs.  For example, backing up (basically taring and zipping a large number of files).  Or you could create a Web content filtering cluster using software like Dansguardian or even a mail virus scanning cluster using AMAVIS.

PVM clusters
PVM (Parallel Virtual Machine) is the other type of clustering technology. The biggest difference from SSI is that, here you need to recompile or build the application which you want to run on this cluster with PVM/MPI support. This means that you cannot run any ordinary existing application as is on this kind of cluster.

The commonly used clustering middleware is OSCAR.

The benefit of using such a cluster is that if you are running a single application which needs a huge number crunching capability on a PVM cluster, then the application will automatically take care of thread management and job migration between the nodes.

What would you use a PVM cluster for? Scientific applications for one are best suited for PVM clusters. If you want to build a cluster which can do genome mapping for example, then PVM is your best answer. Similarly, data modeling and forecasting jobs are best run on this kind of cluster.

Specialized clusters
These are some special types of clustering software which are designed to do some very specific tasks. For example the Deep Blue supercomputer is designed to just play chess, or Deep Crack is designed to do only DES cracking.

In a similar fashion, we have some software and even live CDs which can do some specific tasks rapidly. One example is Cinelerra, which is a video rendering application but having the capability of running on clusters to execute the renders real quick. Similarly ChaOS can be deployed for checking security levels (for password cracking).

Economics first
Before we set out to build the cluster, it is important to know, how much it would cost us.

The true answer is, it depends. Depends on your budget, and the effort you are willing to put into it. If you are low on the budget front, but have a decent network and are willing to put in the effort, you can have a working cluster for next to nothing. On the other hand, if the concept is proven to give you good benefits, then budgets should be the least of your limitations.

As an indicator, let us look at how much we spent on the 20 node HPC we built for doing this story.

We built this cluster using an array of 20 entry level P4 PCs. Of course we did away with luxuries like extra graphics cards, keyboards and monitors. The cluster cost us under Rs 3 Lakhs, without racks and cooling, which is very much comparable to a decent dual processor server in the market and will give many times the comparable performance.

We also did some comparisons of this cluster (using 18 nodes) with an IBM e-server with duel Xeon processors and 1GB of RAM (see PCQuest, October 2005)  and the results that we got were astonishing. For benchmarking, we did a large number of backup tasks at the same time and measured the time taken to complete the process for both the server and the cluster.

The cluster makes more sense from a productivity perspective as well as an economics perspective. The most obvious negatives for the cluster we created are the space it consumed (twenty boxes against one), the power consumed and the heat generated. Once our cluster started getting regularly used, we realized that enough heat was being generated to increase the temperature of Labs to such an extent that we had to do extra air conditioning in summer and could do away with heating during winter!

What if you do not have a budget to start building a cluster? Remember that we had said that you can start with your existing network and no budget in hand. Let's see how you can do that.

Let's suppose that you have around 50 PCs which has say Windows installed, sitting on a network and that your office hours are 9 am to 6 pm. So the systems are not in use for about 15 hrs per day. You can convert this setup into a cluster by night. The only thing missing here is software which can convert this network into a cluster when needed. There are quite a few tool-sets which can do the job.

Let us say you want to build an SSI based cluster which can run in your heterogeneous (Linux and Windows) network to speed up your backup jobs after office hours.  Then what we are going to create is an OpenMosix based cluster running natively on the Linux machines and over a virtualization layer on the Windows machines. You can use this kind of a setup during normal working hours as well, because this uses your idle CPU time available. The only problem is that the network loads will go up and performance could degrade. So, it is advisable that you do not do this during working hours, or alternatively that your network has enough head room to accommodate the load.

A heterogeneous cluster
Like we just discussed, we are going to run OpenMosix natively on Linux machines and on a virtualisation layer on Windows ones. The challenge is to select the right virtualization tool for this purpose. In this case we need a virtualization tool which can run on windows with minimal footprint, so that it can use the maximum resource of the node for the cluster. We chose coLinux.

Why coLinux?
coLinux (www.colinux.org) or Cooperative Linux is a port of the Linux kernel that allows it to run cooperatively along with another operating system. It actually runs as a Windows kernel driver. Unlike most virtualization software, which does full machine virtualization and uses up a good amount of system resources, this small piece of software allows the Linux kernel to run natively (as a Windows kernel driver). So there is no "bridge" between the host kernel and the guest kernel. That's why the Linux guest OS can run at a relatively near-native speed.

Lets get on with the job
The first step is to install coLinux on all the windows machines. For this, download the EXE file from 'http:// prdownloads.sourceforge.net/colinux/coLinux-0.6.2.exe' and install it on C:\coliux on all your windows machines. You will need some more software to get going-a file system image for the virtual machine. You have two options here, either you the Fedora Core 1 file system image or the Debian image. We used the FC1 file system and that worked pretty well for us. You can download it from “http://prdownloads.sourceforge.net/colinux/colinux_minimal_fedora_ core_1.zip”.

This zip file will yield some bz2 files.  So you must first download this file on a Linux machine,  unzip and unbzip it and then move it to the Windows 'c:\colinux' folder. Unzipping and unbzipping this file will generate a 2GB image file. You need this much free space in each machine on the cluster. 

Now you have to get a custom kernel for your coLinux which has an OpenMosix patch.

For the geeks-the only way is to build one yourself by first patching the kernel-2.4.26 source with the coLinux patch and then patching it with the OM patch and then building it.

But that will take a huge amount of time and cause a huge amount of heartburn, when things do not work exactly as advertised. So, the smart way is to download a pre-compiled kernel with the coLinux and openMosix patch from 'http://www.minet.uni-jena.de/~gentryx/harpy.tgz'. On your Linux machine, you can extract it by running the following command

#tar –zxvf harpy.tgz

After extracting, you will get the kernel file called 'vmlinux'. Now copy and replace this file in the existing vmlinux file at 'c:\colinux'.

You have to now install WinPCap. This is required for the network driver in coLinux to work properly. You are creating a bridge network between coLinux and the host Windows. Download and Install WinPCap from the following link http://www.winpcap.org/install/bin/WinPcap_3_1.exe.

Now you have to modify the default.colinux.xml file. This file has all the settings for coLinux. You have to make changes to reflect the following.

1.   The path to the FC1 filesystem image file image on your machine

2.   The path to the new kernel

3.   Change the network type from 'taped' to 'bridged'

4.   Give your network card type

If you have followed the steps mentioned above exactly, then file in the box “coLinux configuration file” will work perfectly for you with just one change: the name of the network card.

For this check the model number of your LAN card installed on the windows machine and replace the 'name=” RTL8139” value with your model name in the last but one line. You can identify the model name in Windows by right clicking on the LAN connection icon and checking its properties.

 Now run coLinux with the following command in Windows.

C:\colinux\>colinux-daemon.exe –c default.colinux.xml

And coLinux will start running on the Windows box. Now you need to install the openmosix tool (userland tool) on top of coLinux.

For this, download the OpenMosix userland tools from the URL 'http:// prdownloads.sourceforge.net/openmosix/openmosix-tools-0.2.4-1.i386 .rpm' and install it with the following command:

#rpm-ivh openmosix-tools-0.2.4-1.i386.rpm

Note that we are using an older version of the tool set because the patched kernel we are downloading is not compatible with the newer versions of the tools. If you want to use the newer version, you will have to compile your own patched kernel.

Now modify /etc/mosix.map and enter the list of all the nodes present in the network. This is nothing but a list of all the nodes that are going to be in the cluster, with their IP address. You can use their names if you have a DNS server or host file that OpenMosix can recognize. To be safe, use IP addresses.

 If your network has 40 nodes in the subnet 192.168.1.0 and the IP addresses are in the range of 1 to 40, then the format of mosix.map is like this:

1            192.168.0.1    40

If there is another set of 10 machines on the same subnet, that you want to add to the cluster, but with IP addresses starting at 50 (you do not want to add the intervening 10 machines to the cluster because they are notebooks), then you add another line to the file saying

41            192.168.0.50  10

Remember that coLinux doesn't support OM auto-discover. So this file is a must. Now start OpenMosix with the following command

#service openmosix start

Your Windows node is now cluster ready.

For Linux machines, you have to install kernel 2.4.26-openmosix from 'http://prdownloads.sf.net/openmosix /openmosix-kernel-2.4.26-openmosix1 .i686.rpm' and reboot the machine with this kernel.

Now if you have any Linux machine which is running the same version of OM kernel (kernel 2.4.26-openmosix) and it has the openmosixview installed on it then you will be able to see this windows node in the cluster list and will be able to use it as a part of the cluster.

One Linux machine (running the X Window GUI system) should have OpenMosixView installed from 'http:// www.openmosixview.com/download/ openmosixview-1.5-redhat90.i386 .rpm', so that you can monitor the nodes in your cluster. With this your heterogeneous SSI based, cluster-by-night is up and running. In the next issue we will see how much this cluster can speed up your routine batch jobs. We will also see how to deploy our full time, dedicated cluster.                                      ¨

By Anindya Roy, Krishna Kumar with help from Vijay Chauhan

Page(s)   1