Renderfarm

From ETC Public Wiki
Jump to: navigation, search

The ETC has a renderfarm consisting of 46 nodes, and currently supports batch rendering for Maya, 3DS MAX, NUKE, After Effects, Houdini and manual scripting via Shell script and Python. GPU-assisted rendering support will be added in the future. The render management software is Muster. The version of Muster installed on this farm (as of Apr, 2019) is 9.0.10. Note that this may not be the latest version available.

The renderfarm is reserved for official use by designated ETC project teams only; there is no public queue. If your project team will be using it, Jon and/or Steve will have already contacted you about it.

Renderfarm File System

The ETC provides a network file system for storage of asset files to be accessed and modified by the renderfarm nodes. You can access it via \\old-macdonald\etc. If you're on a team that has been given access to the farm, then a sub-folder for your team will have been pre-created for you (i.e. \\old-macdonald\etc\render-files\2019-semester-1\sirena\).

You must make your project portable by making sure all files referenced are relative to a set project folder (Maya example) and then copy that to your team's folder on Old-MacDonald, as this is the only server that the user that runs the farm has access to (it cannot read files from Randon). This is the path that you'll enter when submitting a job to the farm.

Renderfarm Workflow

In brief, a renderfarm consists of the following components:

  • Dispatcher Service: The Dispatcher Service manages connections with every renderfarm node, retains a job queue, accepts commands from consoles and sends notifications to the users.
  • Render Client Service: The Render Client Service is installed on each renderfarm node and retains a connection with the dispatcher service, waits for server commands and starts the scheduled render jobs.
  • Console: The Console is a GUI front-end to the Dispatcher Service. It allows the user to schedule jobs, manage and configure connected nodes and configure the dispatcher service.

When you open the console tool, it will connect to the Dispatcher Service to manage the jobs and nodes. Each job will be scheduled by the Dispatcher Service and dispatched to different renderfarm node in chunks (usually several frames). The Dispatcher Service will monitor the connection of each node and send the status back to your console interface.

The Dispatcher Service and Render Client Service have been configured for the Renderfarm. You only need to manage all the jobs with the Muster Console tool.

Getting Started

The ETC Renderfarm is accessible directly through Muster Console tool. You can either get it from Jon and/or Steve, or download it directly from Muster's website. Only the client tools are required to run the Muster Console.

Connect to Renderfarm Dispatcher

  1. Click the "Connect" button on the toolbar. Fill in the account information as follows:
    Renderfarm Login Panel.png
    Note: The password is the one you used in ETC's website and other services.
  2. The Muster Console is displayed as follows:
    Muster Console Overview.png
    1. The Instances view shows all the Renderfarm nodes and their current status.
    2. The Queue view shows all the queued jobs and their progress.
    3. The Logs view shows all the log message on the renderfarm.
    4. The Submission panel shows the data for the job you are going to submit to renderfarm.

Submit a Job

  1. The submission panel has all the information needed to create a job. To begin with, select a template engine first because all the job settings will be reset when you switch to another template.
    To begin with Maya, check Maya Arnold option. For other platforms and renderers, please check the #Platform-specific Workflow section and the official documentation for more details.
  2. The next part you need to focus on is the Multiframe options. The Packet size field specifies how many consecutive frames will be dispatched to a single node as a chunk. You can also assign the renderfarm nodes you would like to use in the Include/Exclude Pool(s) option in the General options.
  3. What follows is the specific settings for each template. Suppose you are using Maya Arnold template, it will look like this:
    Muster Maya Arnold Settings.png
    Here we need to focus on the following options. These may also apply to other templates:
    • (Maya) scene file name: The scene file to be rendered. All the render settings should be configured (e.g. Maya Render Settings) in this file before you assign it.
    • Project directory: The project workspace that the scene file should be located in. This is very important for software like Maya that depends heavily on the project workspace directory to handle with path referencing. Check the #Platform-specific Workflow section for more details.
    • Frames destination: This option enables you to change the output directory of the rendered images.
    • Frames: The range of frames to be rendered. Note it will override the render settings in your scene file.
    • Digits: The number of digits in the output filename. For example, if the digits are 4, the output file name would be like image-0001.exr. Note this will override the render settings in your scene file.
    Note: Before you submit the job, make sure all the files selected are on the remote drive (\\old-macdonald.etc.cmu.edu) instead of local drives.
  4. When you are ready to render, click the Submit button at the bottom of the window. Muster will push the job into the queue and start assigning render jobs to each node soon.
    Note: You can click any job in the Queue view to reload the job configurations in the Submission panel.

Check Job Progress

  1. The queue view shows the current progress of the job, and you can observe the progress through the progress bar. Generally, green chunk means success; yellow chunk means processing; red chunk means error or warning occurred.
  2. The detailed information on which chunk triggers error can be inspected in the log view. You can right click on one log message and select Open chunk's log to check the message output from the corresponding renderfarm node. It is recommended to check the project files on local PCs first before inspecting the problem in the Renderfarm environment.
  3. If you need to redo any chunk, select the chunk in the output view, right click Select chunk and open the Chunk detail window; then right click the selected chunk and click Requeue chunk(s) to push it back to the rendering queue.

Platform-specific Workflow

Maya

Maya is based on a project based workspace system to manage file referencing. All the files outside the project directory will be referenced with absolute path directly, which may cause problems on the Renderfarm environment. It is recommended to set the directory where the scene file exists, and to always set the project workspace before working on these files.

The dispatch template of Maya on the renderfarm is utilizing Maya's batch rendering feature to render the scene. You have to make sure the render process succeeds under the Batch Render mode (in the "Render" menu) before you start rendering on the renderfarm. Otherwise you may face the same issues as you occurred using Batch Render mode on your local PC. For more information, check Render from the command line and Batch render a still or animation on Autodesk's knowledge base.

For dynamics like clothes and fluids, it is highly recommended to bake the dynamic system first (into Alembic cache or other types of cache) before rendering to accelerate the rendering process and avoid potential issues with batch rendering.

Maya supports multiple AOVs and render layers (configured in the Render Setup window) in the render settings. By default, all the render layers and AOVs will be rendered on the renderfarm. If you need to render specific layer or AOV, check the render flags and append it into job configurations.

Again, make sure your scene file works on any local PC first before moving to the renderfarm. Be prepared to have a backup plan to manually render anytime in case any renderfarm node crashes or produces wrong results.

Houdini

Houdini has a well-designed workspace system and it is always using $HIP to manage the files under the working directory where the scene file locates. Unless you are referencing files outside this directory, you don't need to worry about file referencing issues on Houdini.

However, the default template script provided by Muster has issues resolving backslash \ paths. A fixed version of template should be provided on the renderfarm called Houdini hython for Sirena. You can also replace backslash with slash manually when setting up the job. In addition, the Frames output path field does not work for the moment, so you have to specify it in the render settings of the ROP node (for example, /out/mantra_ipr by default).

The Houdini hrender template script does not work as well. Any fixes to this script are welcome and we hope you could provide more information on it.

Likewise, we recommend baking all the dynamic systems first (DOPs, ROP Geometries, file caches, etc.) because it will reduce rendering time a lot. It is also recommended to optimize the shader settings first, especially for volume shaders, because volume rendering will cost pretty long time in Mantra.

Sometimes you will encounter licensing issues with the error message no licenses could be found to run this application. In most cases you can cancel the job, wait for several minutes and it should be solved. If not, please contact Jon and(or) Steve for help.

Template System

Muster features a powerful template system based on Python 3. A template library containing commonly used software has been already included in Muster. However, you might need to modify the template file manually to support your specific demand. For example, some versions of Houdini does not support backslash in command line rendering, and you might have to change it to slash characters manually in the template file.

An admin access is required to modify the template system. It is recommended to restart the whole renderfarm after a new template is added to make sure all the renderfarm nodes receive the latest version of the template library.

For more information, please check Muster's official documentation.

Known Issues

  1. The XGen plugin does not work well on the renderfarm due to its limited support for batch rendering mode in Maya.