MantraRemoteRendering

From Odwiki

What is remote rendering?

Remote rendering refers to Mantra's ability to render using multiple machines for the same frame (viz. Distributed rendering). Sometimes, this can also be thought of as a form of network rendering since the machines are connected via a network.

Houdini Apprentice

Remote Rendering is disabled in Houdini Apprentice.
This includes the -n option.

Licensing

For all of your remote machines, they must have mantra licenses available to them. If you have a node-locked license, you probably don't have mantra licenses for any other machine. In that case, remote rendering can only work on your local machine (ie. the -n option).

How to invoke remote rendering

To perform remote rendering, use mantra's -H option. eg.

 mantra -H host1,host2

One can also use multiple mantra processes (see MultiProcessing) on the same computer by using the hostname localhost. eg.

 mantra -H localhost,localhost

This is equivalent to:

 mantra -n 2

You can also invoke multithreaded renders using the -j command-line option or specified via a RenderingProperty present on the MantraROP. The following will run with two threads:

 mantra -j 2

The following will render with as many CPUs/cores your machine has:

 mantra -j 0

How does the remote rendering work?

The mantra that is run communicates to the hserver running on the remote machine. This is done, even if the remote machine is the local machine (i.e. -n 2).

The remote hserver then starts up a mantra which connects back to the master mantra. The master mantra then sends the IFD to the remote clients asking them to render. The clients then send the rendered pixels back to the master mantra where it composes the final image.

Which mantra is invoked on remote machines?

The mantra started on the remote machine is highly dependent on the locally running hserver. The mantra executable on the remote machine is searched for in the following order:

Entries specified in $HH/hserver.opt (Unix) or $windir/hserver.ini (Windows). Note that on Unix, hserver.opt is searched for using the HOUDINI_PATH environment variable at the time when hserver was started on the remote computer.
(Windows only) The latest installed Houdini version as found in the HKEY_LOCAL_MACHINE\SOFTWARE\Side Effects Software\Houdini registry entry.
Whatever "mantra" executable is found in the PATH environment variable at the time when hserver is started

Where are assets found?

An IFD contains a description of the scene that mantra understands. The scene is comprised of many different assets: Geometry, Shaders, Textures, Point Clouds etc. Some assets are transmitted by the master, while others need to be accessed locally by the remote host.

The basic rule of thumb is that any assets which are embedded in the IFD are transmitted to the remote host, while any assets which are referenced by disk files have the reference transmitted to the remote host. So, if geometry is embedded in the IFD, that geometry would be sent to the remote host. If geometry is specified as a disk file, the name of the disk file will be sent to the remote host. This means that the remote host needs to be able to load the geometry from disk. It's recommended that you use Delayed Read MantraProcedurals if you can, as it results in smaller IFDs to be sent to the remote hosts, and so decreases the wait time until the image rendering begins.

Shaders which are built in VOPs are embedded in IFDs (since there is no disk file), so they are transmitted to the remote host. However, shaders specified from OTLs will have their OTL reference file transmitted to the remote host. Which means that the remote host needs to be able to find the OTL.

Textures (including point clouds, i3d textures, etc.) are assets which are usually specified as disk files. This means that the remote host needs to be able to access the texture.

See the Misc Issues section for some issues with specifying paths.

Misc Issues

Remote rendering (ie. the -n and -H options) is disabled on the Apprentice editions of Houdini.
Make sure that ports 1714 and 1715 are unblocked on all machines in question and that all Hserver versions (check with "hserver -l") are the same or greater than the Houdini builds on the machines.
For textures, all your remote machines must be able to read them using the same path. This means using properly mounted NFS shares on Unix and mapped drives (or UNC paths) on Windows.
On Windows, to use texture maps stored on network drives, the Houdini License Client service (aka hserver.exe) must be set up to run as a user with sufficient permissions to access those network drives. This is true even when using UNC paths (//machinename/mountpoint/maps/file.rat). For paths that use mapped drive letters (x:/maps/file.rat), the user that hserver runs as must also be logged in to the machine, and the network drives mapped in that login session. Thus UNC paths are recommended so that the remote rendering machines will work without having anyone logged in.
1. To do this, go to Services (likely in Control Panel->Administrative Tools) and log in as a specific user.
Finally, note that on Windows, all slashes should be forward slashes and not backslashes to prevent erroneous escaping.
Remote mantra processes are run as the owner of the hserver application. Permission issues may exist if the hserver process is owned by a different user.
There is no central manager for the process, i.e., in the case that multiple artists wish to render on the same pool of remote machines, there is no central server to control the load and distribution of the render buckets. If you have the situation where many artists are wanting to render using remote machines then it is currently up to you to find a way to manage which hosts are in each artists host list. Currently if remote host is deemed too busy to render, it will skip that machine for the entire frame and not repeatedly attempt to render for each bucket. Consider using HQueue.

Troubleshooting

Techniques

In the abovementioned hserver.opt files, there is an undocumented line you can use use:

logfile [logfile]

If you add this line and restart hserver on the remote host (or run hserver -p to reload the hserver.opt file), and then attempt to render the scene once again, you should be able to view the contents of this log file to find out what seems to be going wrong.

An example of the contents of this file could look like this:

%> tail -f hserver.log
12:37:26 09/06/06 hserver[638655c023]: C[11]
12:37:27 09/06/06 hserver[71254667de]: C[0]
12:38:06 09/06/06 hserver[638655c3ba]: write IP access failure
12:38:06 09/06/06 hserver[638655c023]: C[0]

This error, for instance, tells you that it could not send data to the receiving hserver. Checking the hserver.opt file revealed that the subnet mask for readIP and writeIP (default to +.+.+.*) didn't allow communication between two machines on different subnets. Marking this into +.+.*.* solves the issue.

There is also an undocumented switch for the hserver executable itself to override the logfile setting in hserver.opt:

hserver -L [logfile]

Messages

mantra: Network Error[myhost-l] HServer was unable to start mantra
- Check the logfile output as described above. This particular message may have many meanings.
  - The readIP and writeIP subnet masks in the hserver.opt file may be too restrictive by default. Its set to +.+.+.* - which would be to narrow, especially for larger facilities where it's very likely you'll have more than 256 machines.

Personal tools