Yeraze's Domain 3.0

Wired.com posted the list of SciDAC 2009 winners, with the videos.  I’m on Wired!

Impact of a Copper Bullet (video not available, sadly. Long story)
Breaking Waves

Best Science Visualization Videos of 2009 | Wired Science | Wired.com.

Tonight was the SciDAC 2009 Visualization Night, and I submitted three videos for them to include.  Yes, I know I’m a DoD guy and that’s a DoE conference, but they wanted our submissions and we were happy to oblige.
I submitted:

2000lb Class Device in a Bunker
Projectile Impact on Kevlar Fabric
and a special “Greatest Hits” compilation of [...]

Anyone who’s used Autodesk’s 3D Studio Max for a while knows about BackBurner.  BackBurner is the Queue-controlling software they ship with Max to allow you to render jobs across multiple scenes in a naive way.  Rather than pooling resources to work on a single frame, it hands out individual frames of the sequence to each node in the backburner queue.  (You can pool resources using the "Strip" rendering, but it’s pretty much the same thing).  However, the way we use BackBurner at work has proven to us that it’s just a piece of crap.  Thus, I wrote my own queue manager that I’ve dubbed "Freezerburn".  Let me elaborate.. 

In our environment, we typically get a dataset of 1000 or so timesteps.  We extract 1000 isosurfaces (one from each timestep), and then need to render these 1000 files into 1000 frames of an animation (For those interested, we typically save the isosurfaces as Stanford PLY files and use the PLY import plugin I wrote, along with a MaxScript, to convert them all into MAX files).  So we create a simple setup scene with the context, lights, and camera.  We then use another custom MaxScript I developed to embed a callback into the Max File’s "onLoad" event that will look at the current frame number, merge in the appropriate Max file, and apply whatever material properties we need.  That way we have a single MAX file that can render all 1000 frames of the animation.  Unfortunately, since it’s in the onLoad, it means we need to load this Scene file from disk for each image.

With Backburner, you have no control over this.  With a large scene, it typically starts off assigning individual frames to nodes, which is exactly what we want.  After those complete, however, it starts assigning larger & larger batches of frames to each node (to reduce overhead spent loading the file & transmitting information).  This means that the onLoad callback is only triggered once but renders multiple frames, which isn’t what we want.

The technical amongst you may be wondering "Why not just use preRenderFrame"? Well, there’s a few reasons while the various preRender callbacks won’t work like that:

1. preRender – Only called once at the beginning of the entire render, basically no better than onLoad
2. preRenderFrame - Called between every frame, but after geometry is cached.. Attempting to change geometry (eg. Merge in a new Max File) results in a SegFault
3. beginRendering* – Same as preRenderFrame
4. preRenderEval - I had high hopes for this when I first saw it, but it seems to operate identically to preRender (only once at the beginning of the entire render)

So, the only way to guarantee that backBurner will process the frames as individual jobs, is to submit each frame as an individual job.  Now, we have MaxScripts to do this for us with a single buttonclick, so it’s not that bad.  However, Backburner simply can’t take it.  BackBurner was designed to handle a handful of several-thousand-frame jobs, and does so beautifully.  However, hand it a few thousand single-frame jobs, and it dies.  It dies horribly.  It dies a slow excruciating death as you watch job submission times go from half a second to 3 or 4 minutes per.  It’s truly sad and ridiculous.

So my solution was to use my new Python skills to write our own queue manager.  It’s written entirely in Python, and offers the following features:

• Queue & priority control
• For Mental Ray scenes it gives you % complete on each of the 3 major stages of render (Preprocessing, Final Gather, Rendering)
• Vastly improved "at-a-glance" status information
• Controllable via a Web Interface (with SSL certificates and CAC authentication which is required where I work)
• And much much more…

Over the next few weeks I’ll post screenshots and articles of how it works, to share the knowledge I gleamed about Python, Windows Programming, HTTP servers, and 3d Studio Max.  So be sure to come back!
[tag:autodesk][tag:backburner][tag:python]  

A friend of mine tipped me off to the fact that MAKO finally made their IPO last month.  I had completely missed it (which is probably a good thing).  So I hit up Google’s Financial pages to see the trading charts..

While this may look good at a glance, a quick look shows nothing of note.  They opened just under $10, spiked to $12, and settled down around 10.50 for today.  What I find more interesting is the harsh commentary on it… Like the Wall Street Journal:

MAKO closed at $9.18 a share on the Nasdaq, down 8% from its IPO price of $10. It sold 5.1 million shares at the low end of a reduced $10 to $11 price range; it originally was expected to price between $14 to $16 through underwriters J.P. Morgan Chase & Co. and Morgan Stanley.
…
Meanwhile, Fort Lauderdale, Fla.-based MAKO specializes in making robotic instruments and implants for knee surgery. Its products, which were first cleared by the Food and Drug Administration, allow surgeons to avoid complete joint replacement for early-to-mid-stage osteoarthritic knee disease, and instead resurfaces damaged joints and places implants through a small incision.

MAKO has never been profitable and expects to continue losing money as it develops its business.

So, looks like noone will be selling their shares and buying a Yacht anytime soon.  Guess we’ll just have to wait and see how it plays out.
[tag:mako][tag:stockmarket]

I was kinda surprised to find this (Thanks to some researchers in the office for tipping us off to this), but there’s some conceptual animation one of my coworkers did of a Rapid Levee Repair possibility.  A somewhat far-fetched possibility, but a possibility nonetheless (Man, if you think this is far-fetched, you should’ve seen some of the other… wow).

It’s courtesy of an article in New Scientist entitled "Air Dropped Dams could fix levee breaches".  The YouTube clip is below in pretty poor quality.  Our version was significantly higher-resolution, somewhere around 720p.

It’s strange how stuff shows up in public sometimes, without rhyme or reason.
[tag:work][tag:daac][tag:neworleans][tag:levee]

So for the last few weeks I’ve found myself doing alot of web design.  First with VizWorld (had to build that entire layout from scratch), then with the ezViz Script Generator, and now with a redesign of the ItsOnSirius Website.  In all my previous webwork I generally stayed away from JavaScript, partly because of it’s non-cross-platform nature and partly because I just didn’t want to learn it.  This time, tho, I decided it’s time to learn and dive in to learn how all the new fancy “Web 2.0″ Ajax stuff works.

It’s been a fun trip.  It’s always best with stuff like this to have a project to implement your education as you go.  Just reading is one thing, but actually implementing stuff in code shows all the corner cases and odd behaviour not covered in most texts.  Also, trying to design code that works with IE6, IE7, and FireFox is a serious PITA.

So here’s a few tips for others attempting the same….[tag:html][tag:javascript][tag:development][tag:ajax]

As a followup to a previous story, I just found a press release on Biomet’s Website entitled “Biomet Announces Fourth Quarter and Fiscal Year-End Results, and Cash Dividend“. It has the following sad information:

So that’s it, Acumen is finished (At least with Biomet/EBI). Sorry hear it guys, I hope all you that got shafted in the “relocation” 2 years ago do ok.
[tag:biomet][tag:ebi][tag:acumen][tag:zkat]

While looking for an answer to my previous 3dsMax problem, I came across a lightly-documented tool called the “BackBurner Web Monitor”. 

It’s not talked about much, but it’s basically a CGI tool that can duplicate all of the monitor’s functionality through a Web Page.  From the WebPage you can view jobs, view servers, suspend and restart jobs, and more.  It’s a great tool because all of the communication with the Manager is done from a single system, the Web Server.  Of course, they won’t let us run a webserver here (Be it Apache or IIS) so we’ve been unable to use it.  This means we have 3 or 4 people all running a monitor individually, which puts a huge load on the manager (Especially once you top the 1000 jobs mark).

I took it upon myself to dig into this deeper and came up with a neat solution.  Being a CGI program, it simply takes arguments via a HTTP GET, and spews out an HTML webpage to relay to the user.  So why can’t I emulate this behavior, and then copy the resulting HTML file to our fileserver, for all of us to view?  A few hours later, and after researching how to pass GET parameters, I had a prototype in Tcl working.  With a bit more work, I even had it postprocessing the HTML files to remove links (that no longer work), correct paths to correctly resolve to the Stylesheets and images, and disabling the buttons (since they no longer work).  I even went so far as to completely scrape and rebuild the Servers listing page so that it’s in a basic table format.

Sure, the result is read-only, but that’s 90% of what we need.  I have a TCL Script that I can run to update the pages every 30 seconds on the fileserver, and the HTML documents have META Refresh tags to refresh every 30 seconds.  This way we can each simply open the job listing and leave it running to see how the renderings progress.  It puts basically no load on the manager, and lets everyone stay updated on what’s going on.  It’s been a big hit, and I’m curious if such a thing would be useful to others.  I’m sure I’ll add a few more bells and whistles to it, but it’s working great for us right now.

Post a comment or drop me an email if you’re interested.
[tag:3dsmax][tag:autodesk][tag:backburner][tag:webpage][tag:cgi][tag:tcl]

Well, this last week has proven interesting at work.  A few weeks ago I was contacted by someone curious about the differences/advantages between various file formats for Visualization.  Specifically, they were looking at XMdf, XDmf, Exodus, and VTK.  I told them what I knew, and that has started off some very interesting discussions.

Most recently, tho, we started discussing the possibility of me adding support for their existing formats, rather than them converting to a new format (at least in the short term).  I had already written a file reader for a similar format, ADCIRC, so I didn’t think this would be hard.   The format they’re using was listed as “NetCDF CF“.  NetCDF is a pretty generic file format, similar to HDF5, and CF is a “Climate Forecasting” convention.  NetCDF has a pretty strange way of doing things, but it didn’t take long before I figured it out.  I’ve already got some code written to decipher & read the grid into VTK structures, but the data is proving difficult because it’s “edge centered”, instead of cell or point centered.  I haven’t figured out how to deal with that, but luckily the guys on the other end have made my life a little bit easier by switching from this format to the “Quoddy” Format, which is all point-centered.

So basically, I’m up to my waist in File Format issues.  It’s ok though, I actually enjoy it.  It’s a fun form of cryptography, deciphering cryptic files and data structures, often with little to no direction (As evidenced by ytnef).  So I’m having fun.. Now, if only our new viz cluster would finally get approved and installed, so it didn’t take me 5 minutes to compile my code.
[tag:work][tag:hpc][tag:simulation][tag:format]

Today I found an interesing quirk in Microsoft Visual Studio .NET 2003, specifically Visual C++.

I was attempting to compile a linux app on my machine that used CMake & VTK. It all went pretty well until I got an error “Unresolved symbol atanh”. I quickly noticed that I wasn’t including math.h and tried again, only to be met with the same error. Figuring that maybe I wasn’t doing it the “Microsoft Way”, I highlighted it and clicked F1 only to find nothing.

After some digging around, I found that atanh, asinh, and acosh are all missing from Visual C++. Instead, they provide a table of derived formulae to compute them. I was amazed. Why would they leave these functions out? If they went through all the trouble to provide a table in their documentation, why not just add the functions, or #define’s? After some research, I finally came across GSL – Gnu Scientific Library, that comes with high precision replacements for alot of math functions, including the missing atanh I needed. A little more digging came up with GnuWIn32, a collection of several libraries compiled for Visual C++, including GSL For Windows.

Now if I can just get “Debugging” priveledges so I can figure out why it still crashes. Seems that pass-by-pointer doesn’t function the same in GCC vs Visual C++.

Update: 5/24/2006 10:30am
As an experiment, I downloaded CodeBlocks with the MinGW compilers. Cmake supports MinGW Makefiles as an output, so I recompiled VTK and my App. I wasn’t able to use the IDE, just ran make from the command line, but now it seems everything is working. It’s faster than Visual Studio, although I’ll freely admit that I am not a VS guru so I may have missed a few optimization flags. But where the VS code returned 0 everywhere and required the GSL library, the MinGW build returns valid values using the built-in atanh function.

Update 9/12/06:
I thought I added this long ago, but Microsoft acknowledges the shortcoming and publishes a workaround here.
[tag:math][tag:gsl][tag:visualc++][tag:microsoft][tag:mingw][tag:codeblocks]