Freezerburn : a Replacement for BackBurner

Tuesday, March 11 2008 @ 10:47 AM CDT

Contributed by: Yeraze

Anyone who's used Autodesk's 3D Studio Max for a while knows about BackBurner.  BackBurner is the Queue-controlling software they ship with Max to allow you to render jobs across multiple scenes in a naive way.  Rather than pooling resources to work on a single frame, it hands out individual frames of the sequence to each node in the backburner queue.  (You can pool resources using the "Strip" rendering, but it's pretty much the same thing).  However, the way we use BackBurner at work has proven to us that it's just a piece of crap.  Thus, I wrote my own queue manager that I've dubbed "Freezerburn".  Let me elaborate.. 

In our environment, we typically get a dataset of 1000 or so timesteps.  We extract 1000 isosurfaces (one from each timestep), and then need to render these 1000 files into 1000 frames of an animation (For those interested, we typically save the isosurfaces as Stanford PLY files and use the PLY import plugin I wrote, along with a MaxScript, to convert them all into MAX files).  So we create a simple setup scene with the context, lights, and camera.  We then use another custom MaxScript I developed to embed a callback into the Max File's "onLoad" event that will look at the current frame number, merge in the appropriate Max file, and apply whatever material properties we need.  That way we have a single MAX file that can render all 1000 frames of the animation.  Unfortunately, since it's in the onLoad, it means we need to load this Scene file from disk for each image.

With Backburner, you have no control over this.  With a large scene, it typically starts off assigning individual frames to nodes, which is exactly what we want.  After those complete, however, it starts assigning larger & larger batches of frames to each node (to reduce overhead spent loading the file & transmitting information).  This means that the onLoad callback is only triggered once but renders multiple frames, which isn't what we want.

The technical amongst you may be wondering "Why not just use preRenderFrame"? Well, there's a few reasons while the various preRender callbacks won't work like that:

  1. preRender - Only called once at the beginning of the entire render, basically no better than onLoad
  2. preRenderFrame - Called between every frame, but after geometry is cached.. Attempting to change geometry (eg. Merge in a new Max File) results in a SegFault
  3. beginRendering* - Same as preRenderFrame
  4. preRenderEval - I had high hopes for this when I first saw it, but it seems to operate identically to preRender (only once at the beginning of the entire render)
So, the only way to guarantee that backBurner will process the frames as individual jobs, is to submit each frame as an individual job.  Now, we have MaxScripts to do this for us with a single buttonclick, so it's not that bad.  However, Backburner simply can't take it.  BackBurner was designed to handle a handful of several-thousand-frame jobs, and does so beautifully.  However, hand it a few thousand single-frame jobs, and it dies.  It dies horribly.  It dies a slow excruciating death as you watch job submission times go from half a second to 3 or 4 minutes per.  It's truly sad and ridiculous.

So my solution was to use my new Python skills to write our own queue manager.  It's written entirely in Python, and offers the following features:
Over the next few weeks I'll post screenshots and articles of how it works, to share the knowledge I gleamed about Python, Windows Programming, HTTP servers, and 3d Studio Max.  So be sure to come back!
 

Technorati Tags: , ,

Comments (0)


Yeraze's Domain
http://www.yeraze.com/article.php/2008031110473221