Afanasy

From Odwiki

Free render farm manager (open source).

Afanasy Features

Render computer resources monitoring

You can monitor CPU usage: user, nice, system, I/O wait and load average. Memory, swap and HDD usage. Network traffic. Disk I/O operations speed and utilization percentage. This can help you to diagnose what slows rendering process. Especially swap, I/O wait and network traffic. You can write your own custom resources meters on python.

Job - Block - Task

Job consists of blocks, which produce tasks. There are two types of blocks numeric and string. Task inherits most attributes from block (for example all block tasks has the same working directory). But some other attributes can be stored for each task (commands or command arguments - string block), block can have a rule to generate command for tasks (command pattern to fill with numbers - numeric block).

1 000 000 Tasks Job !

As Tasks are generated by job blocks "on demand", Afanasy (and Server and GUI) can handle numeric blocks with huge tasks number.

Dependences

Job can depends on other job(s) of the same user (depend mask) or on some job(s) from any user (global depend mask). Job will wait other jobs to be done matching this mask regular expression (Afanasy uses Qt Regular Expressions, they are Perl-like).

Job Block can wait other block(s) to be done (depend mask). Block tasks can wait other block tasks to be done (tasks depend mask).

Sub-task Dependence

Tasks can depend on other task(s) sub-progress. They can run when some task finished partially. It is useful to render simulations. As simulations usually performed by a single task on a whole frame range. Now render tasks can run when the entire simulation is not finished.

Priorities

User priority - user with greater priority can get more hosts. Job priority - to simple sort user jobs queue.

Multiply Tasks Hosts

Render host can run several tasks at the same time. Useful for "powerful" hosts with several CPUs. Render and Task has a capacity attribute, each task checks available capacity on render, and if its enough, task launches and take its capacity value from render.

Tasks can variate capacity 'on-the-fly'

Capacity variation can be described through its minimum and maximum coefficients. If task with 1000 units of capacity runs on render with 4000 free capacity, task can take capacity coefficient 4. This coefficient can be caught in task command line (so Mantra can be launched 'mantra -j 4 ...', or 1000 capacity can be used by one 'houdini' task, for example to generate something, which use only one CPU, and 'mantra -j 3'). Hosts capacity values described in 'Farm Setup' ('xml' file).

Hosts Masks

User, job and block has a mask of host names to be able to run on and exclude mask for hosts to avoid. Maximum Hosts

User, job and block has maximum number of hosts - no new hosts will be available above this limit.

Errors Solving

Job Blocks Errors Hosts List - block store host names where error was occurred and number of errors for each host. If this number greater it's limit, block will 'avoid' this host (not to run tasks on render with such host name).

Blocks Tasks Errors Hosts List - the same as described above, but for each task.

Tasks Auto Retry - maximum number of errors to retry task automatically.

Tasks Maximum Run Time - the time after running considered as an error task.

User define this default values for his new jobs and can to override them in each job, watch and reset any error hosts list.

Services and Parsers

Service - 'block tasks type', for example: 'hbatch', 'mantra', 'prman', 'nuke', 'maya'.

Service setups a default task output parser type (which can be overridden later). Services are python classes. They all inherits a base 'service' class. Service class describes command line manipulations, such as task capacity (how to specify how much CPUs to use ('mantra - j @AF_CAPACITY@')).

Parser - read task output to calculate running progress. Parsers are python classes and can be combined (multiply inheritance), for example: 'hbatch_mantra' - inherits 'hbatch' and 'mantra' parsers to listen to tasks when Houdini generates a files for mantra, such parser listen for frames switching (when several frames per render set) and calculates valid percentage.

Paths Map

Every host can has different paths and a map to translate paths to server and from server. With this feature you can setup a multi-OS render. You can use different operating systems on farm hosts and to submit jobs from any OS to render it on any OS.

Python API

You can create and send job using python.

Setup services and parsers through python classes inherited from base service and parser.

Farm setup

Map hosts names and their abilities through patterns described in XML file. You can describe job and block 'need' properties to be able to run only on hosts which satisfy this needs (OS, free memory etc.). Server can reload this file (and reconfigure itself) "on-the-fly", without restart and keep tasks running.

Services Limits - Software Licenses Setup

You can describe various services (tasks) limits: - Maximum number of total service starts on an entire farm. - Maximum number of hosts which started a service (each host can start several tasks of a service). - Maximum number of service starts on each host.

Watch - minimalistic GUI to afanasy

Watch - afanasy monitor. It can be in three modes - USER, VISOR and GOD. USER ( common mode ) - user can change any his and any his job parameter. He can manipulate render host which was registered with it's user name or if it contains user computer name VISOR ( super user mode ) - can do same as user but he change or remove job of any user. GOD ( super user mode ) - can do anything. Change any parameter of any user, job or host. Add or delete users. Start, stop, restart any render host. To switch user mode you need to type password in active Watch window (you do not need to relogin to switch between modes). To reset super user mode you can type password again.

Web Visor

Web interface to afanasy database to show current state tables or draw history usage statistics diagrams.

All calculations and database queries are on the server side, simple HTML text and PNG images provided to client. So you need no libraries or plug-ins for it, any browser will be enough.

SQL Database Connection

Afanasy server can connect SQL database. It process it through Qt classes, you can query available SQL drivers for your system by afcmd command line utility. Postgre SQL, MySQL and SQLite are available for most common systems. It can be chosen in configuration file. Postgre SQL (QPSQL) is set as default. Afanasy stores jobs and users information in database. It get this information back on start. So you can shut down afanasy, and it will restore it's state on start again. Also you can type any SQL query to get some afanasy information yourself.

Command Line Interface

Afanasy has a command line interface for various purposes. It can connect to afanasy from remote Linux machine. You can set some server parameter "on-the-fly". Change some user, job and host attributes. Query users, jobs, hosts attributes and traffic statistics.

Job Example

The simplest job:

#!/usr/bin/env python

import af

job = af.Job('example job')

block = af.Block('block of tasks')
block.setWorkingDirectory('/home')

task = af.Task('simple task')
task.setCommand('ls -l')

block.tasks.append(task)
job.blocks.append(block)

job.send()

You'll need to right jobs directly on python only for custom jobs. It's better to use some GUI to generate jobs. Use some already written in Afanasy or write your own.

Farm Patterns Example

< farm >
  < pattern name="Some Host" >
     < mask >.*< /mask >
     < description >Unrecognized machine< / description >
     < capacity >1000< / capacity >
     < maxtasks >1< / maxtasks >
     < power >100< / power >
     < service name="generic" / >
  < / pattern >
  < pattern name="Render Host" >
     < mask >r.*< / mask >
     < description >Some render machine< / description >
     < os >linux< / os >
     < capacity >2200< /capacity >
     < maxtasks >5< / maxtasks >
     < power >1000< / power >
     < properties >intel nvidiagpu< / properties >
     < service name="generic" / >
     < service name="nuke" / >
     < service name="hbatch" / >
     < service name="prman"  count="2" / >
     < service name="mantra" / >
     < service name="hbatch_prman" / >
     < service name="hbatch_mantra" / >
     < service name="maya" / >
     < os >linux< / os >
  < / pattern >
< / farm >

When some machine registers on server, it finds last pattern with mask that match it's host name. So '.*' first mask put at the top to give any machine some properties.

Houdini Integration

ROP operator written to construct and send job using python soho program. It launches another python program which load and render a scene.

Houdini Launch Script Example

You need to set up some environment to use CGRU scripts and Afanasy OTLs:

# Setup CGRU houdini scripts:
export HOUDINI_CGRU_PATH=$CGRU_LOCATION/plugins/houdini

# Set Afanasy houdini scripts version
export HOUDINI_AF_VERSION="10"
# Set Afanasy houdini scripts location
export HOUDINI_AF_PATH=$AF_ROOT/plugins/houdini/houdini$HOUDINI_AF_VERSION

export HOUDINI_AF_OTLSCAN_PATH=$HIH/otls:$HOUDINI_AF_PATH/otls:$HH/otls

if [ "$HOUDINI_OTLSCAN_PATH" != "" ]; then
   export HOUDINI_OTLSCAN_PATH="${HOUDINI_AF_OTLSCAN_PATH}:${HOUDINI_OTLSCAN_PATH}"
else
   export HOUDINI_OTLSCAN_PATH=$HOUDINI_AF_OTLSCAN_PATH
fi

All examples provided with afanasy ( and Release and SVN)

See site with full documentation for details

Personal tools