DUXSoft : Project SPARX

Tuesday, June 03, 2008

In the last decade or so, the Animation, VFX (Visual Effects) and gaming arena have seen some rapid progress. Each of these technologies has moved out of the computing laboratories to become a full-fledged industry in itself. This progress has also brought with it some new challenges.

Along with the regular management challenges that most organizations face, these studios have dealt with challenges like Pipelining, Render Management and Digital Asset Management.

The production process in a studio is usually very serial and iterative in nature. Due to this, the work flow is referred to as the 'Pipeline' by the industry. The problem is that, even though Animation, VFX and Gaming have a common base in 'Computer Graphics', yet each of these industries follow different production practices. Not just that, unlike other industries, each project itself may require a different workflow. To complicate things further, due to the highly artistic nature of the work, the skill set required may also vary from task to task and not just from project to project. An improper workflow could derail the project at it crucial stages.

Rendering is the process of creating images of a movie scene and is a highly compute intensive task. These individual images are sequenced together to form a movie. One second of a movie scene is made up of 25 very high resolution frames, while one second of television scene is made of 50–60 normal resolution frames. Depending on the complexity of the scene, rendering a single frame may take up to hours.

But fortunately this is a process that can be easily parallelized. If we have enough resources, each frame can be rendered by different computers. For example, if one has 100 Computers, 4 seconds or 100 frames of the scene can be rendered in the same time that it takes to render one frame. Such an array of computers used for rendering is called a 'Render Farm'. Managing a Render Farm is a challenge, as one has to deal with multiple jobs (scenes) and multiple frames being rendered simultaneously.

Has over 14 years of experience in infrastructure development and technical support. Possesses vast knowledge of web app servers and mailing systems on UNIX and MS platforms, and is experienced in planning and management of LAN/WAN, etc. Indivar Nair, VP – Technical Services Q What challenges were you facing before the deployment? SPARX aims to be a one stop solutions to problems one faces in a studio. Along with regular management challenges that most organizations face, studios have to deal with challenges like Pipelining, Render Farm Management and Digital Asset Management. SPARX aims to automate the complete production life cycle of Digital Content creation, right from requirement capture to final delivery and archival. It will help studios dynamically change their production practices without having to undergo a major change management process with every project. Q What according to you is the USP of this project? SPARX Render Management Module uses Sun Grid Engine in the background. Grid Engines are built from ground up to handle thousands of jobs (running on as many nodes) simultaneously. This makes it a very powerful tool for Render Management too.We removed the complexity of using a Grid Engine by giving it a very easy to use, intuitive File Manager type Web Interface.

A huge number of images, scene files, movie clips etc. are generated and used during a project. Due to intellectual / artistic value attached to it, these are known as Digital Assets. Storing these assets securely as well ensuring that they can be retrieved easily when required is another challenge that most studios face.

This is the problem space that DUX Soft tries to address using SPARX, a software application that provides end to end production support for studios of all types and sizes. SPARX aims to automate the complete production life cycle of Digital Content creation, right from requirement capture to final delivery and archival. SPARX will help studios dynamically change their production practices without having to undergo a major Change Management process with every project. And to enhance the speed of the rendering at the backend of SPARX a mammoth 100 Core Render farm is sitting.

The Implementation
Right from the start, SPARX has been developed keeping in mind 'ease of use' for the end user. Therefore the whole system has been designed like a file system explorer. A graphics artist intuitively knows what to do as soon as he / she logs in. Over and above this, a context sensitive help screen guides the user on every step. SPARX web interface has been developed on PHP 5.2.x and uses various additional plug-in to enhance its features. The web pages have been written in plain HTML, with very little JavaScripting and AJAX support. It has been tested to work with most new browsers like IE6, IE7, Firefox and Safari.

SPARX uses Sun Grid Engine 6.1 in the background for its job scheduling and management activities. The excellent features that Sun Grid Engine provides allows one to prioritize job queues, pause jobs, manage compute loads, offload an activity from one compute node to another, have detailed accounting of the jobs completed, etc. SPARX can be integrated with Microsoft ADS or any other LDAP service to provide a common authentication and single sign-on to any SPARX module. At the backend SPARX uses four Dell PowerEdge 2550 servers for ADS, Storage and SPARX Master Node. For the render farm they have used 25 Dell PowerEdge 1950 for Render Farm with two dual core processors which essentially mean 100 cores. The nodes runs OpenSuse 10.3 as the operating system which is stripped down to an extent that they have minimal OS overhead on the processors. And as this render farm is connected to the website from where the customers can initiate process at any time, they needed to keep all these servers running throughout. But to save power what they have done is that, they have created a script which turns on all the 25 nodes when any job is initiated. And once the job is finished all the nodes again goes to shutdown state automatically. This simple trick helps them to cut down power consumption in a big way.

The Impact
This kind of a outsource workflow management for rendering will essentially help the small and medium companies which are into this space. Now they can concentrate on the core job of creating the animation footage instead of wasting a huge amount of CPU and Human time in rendering the created footage without even investing a bomb on deploying their own render farm.

Page(s)   1