Yeraze's Domain 3.0

This weekend I was digging through old bookmarks and dug up an old reference to a game called Aquaria.  I first heard of Aquaria when it won the IGF2007 Grand Prize, but it wasn’t available for download at the time.  Now, they have a good 60M Demo of the game (good for 2 hours or so of playtime), and for $30 you can get the full 250M version.  I’ve been playing it all weekend, and after 6 hours of playtime (according to my savegame) I’m still finding new stuff and continually being astounded by the game.

The graphics are simply beautiful.  The entire game is a side-scroller basically, but with lots of little bells and whistles.  Seaweed and flowers flow as you swim by, matching your movements.  Schools of fish following true flocking behavior swimming around.  The environment changed from lush green landscapes, to barren castle hallways, to eerie organic fleshly tunnels, and all done seamlessly.  The music is even more impressive.  An eerie lonely tune that adds to the mystery of "Who am I?", with fast-paced music added for boss fights and action scenes.  The cut-scenes are great, and the british accent voiceovers are a perfect match.

At it’s heart, Aquaria is a puzzle game with a little bit of combat.  Primarily you’re searching for new "forms" (spells) to give you new abilities, but usually each form comes with solving a puzzle.  Boss fights are there, but are usually puzzle-based. Eg. The boss has only a single special weakness that you have to discover and exploit.

The game is a blast, and I’m catching myself humming the theme music constantly.  I look forward to seeing how the story plays out, and seeing what other fantastic environment there are for me to explore.
[tag:game][tag:indie][tag:aquaria]

Working in the DOD, there are a few things you just come to accept.  Webservers require security (SSL), and SSL requires Common Access Card Authentication.  I had hoped that when I implemented the HTTP Monitor for Freezerburn, I could ignore all the security aspects and simply say that "It’s behind all the government firewalls" and "It requires an account on the fileserver".  I wasn’t so lucky.

After alot of research from articles like:

• NRL DoD PKI Notes
• CommonAccessCard – Ubuntu Documentation
• HOWTO CAC Server – Gentoo documentation

I finally came up with a way to make it work, that required very little variation from Python’s built-in HTTPServer object.  Starting with the ASPN Cookbook recipe for SecureHTTPServer, I came up with the following:

import socket, os
from SocketServer import BaseServer
from BaseHTTPServer import HTTPServer
from SimpleHTTPServer import SimpleHTTPRequestHandler
from SocketServer import ThreadingMixIn
from OpenSSL import SSL
import sys

class SecureHTTPServer(HTTPServer):
    def __init__(self, server_address, HandlerClass,
                dodcerts, serverkey, servercert):
        BaseServer.__init__(self, server_address, HandlerClass)
        #  Based on online Documentation, the v23 actually enables TLS1 as well.
        ctx = SSL.Context(SSL.SSLv23_METHOD)
        #ctx = SSL.Context(SSL.TLSv1_METHOD)

        print "Loading Private Key from %s" % serverkey
        ctx.use_privatekey_file (serverkey)
        print "Loading Certificate from %s" % servercert
        ctx.use_certificate_file(servercert)
        print "Loading DOD Certifications from %s" % dodcerts
        ctx.set_verify_depth(2)
        ctx.load_client_ca(dodcerts)
        ctx.load_verify_locations(dodcerts)

        print "Creating SSL socket"
        callback = lambda conn,cert,errno,depth,retcode: retcode
        ctx.set_verify( SSL.VERIFY_FAIL_IF_NO_PEER_CERT | SSL.VERIFY_PEER, callback)
        ctx.set_session_id(‘Freezerburn’)
        self.socket = SSL.Connection(ctx, socket.socket(self.address_family,
                                                        self.socket_type))
        self.server_bind()
        self.server_activate()

class SecureHTTPRequestHandler(SimpleHTTPRequestHandler):
    def setup(self):
        self.connection = self.request
        self.rfile = socket._fileobject(self.request, "rb", self.rbufsize)
        self.wfile = socket._fileobject(self.request, "wb", self.wbufsize)

This creates a new object called the "SecureHTTPServer" that acts just like the regular HTTPServer, except it allows you to specify the location of the DOD Root Certificates, the Server Private Key, and the Server SSL Certificate.  The only real difference from the ASPN one is that it turns on Client Certificate Verification, which is the core of the CAC Authentication scheme.  With that little snippet of code, SSL & CAC were enabled in one fell swoop!
[tag:ssl][tag:python][tag:cac]

Laura made this for her MySpace page the other day, and I thought it was worth showing to the world here Enjoy.

[tag:myspace][tag:pics]

One thing I decided very early on with FreezerBurn was the need for a Web-Based monitoring tool.  I didn’t want to have to deploy a Python tool with a full user-interface to every desktop and then deal with all the Security implications of that.  With a web interface, I could simply let everyone use FireFox or IE or Safari and load it however they wanted.  At least, that was the initial idea.

Very quickly I came across Python’s HTTPServer class which handles a good 90% of what I needed.  After some experiments, though, I found that it tended to die under heavy loads (eg, loading my page with external JS, CSS, and Images).  I needed it to be fairly reliable, and thusly discovered alot more then I ever intended to about HTTP Caching.

HTTP implements caching 2 ways: ETags & "Last-Modified" dates.  Etags are simply checksums where the Browser says "The version I have has this checksum", and the server checks it and tells you "That’s right" or "That’s old, here’s a new one".  In my case, I chose to instead implement the "Last-modified" version where I could simply check the last modified date on the files.  The resulting code looks like this:

class MonitorServer(HTTPRequestHandler):
    def do_GET(self):
        global jobQueue
        global servers
        global mQueue
        global initTime
        # process self.path
        (scm, netloc, path, params, query, fragment) = urlparse.urlparse(self.path, ‘http’)
        if scm != ‘http’ or fragment:
            self.send_error(400, "bad url %s" % self.path)
            return
        print ‘HTTP: Serving %s’ % path
        # Write to self.wfilt
        if path == "/":
            # Raw Index Page
            self.wfile.write(""" blah blah blabh """)
        elif path == "/job":
            blah blah blah
        else:
            # Serve files from the system
            path = path.split(‘/’)[-1]
            if os.path.exists(constants.MonitorPath + ‘\\’ + path):
                (ctype, enc) = mimetypes.guess_type(path)
                info = os.stat(constants.MonitorPath + ‘\\’ + path)
                lastmod = datetime.datetime.fromtimestamp(info[ST_MTIME])
                if self.headers.get(‘If-None-Match’,”):
                    self.send_response(304)
                    return
                if self.headers.get(‘If-Modified-Since’,”):
                    dt = self.headers.get(‘If-Modified-Since’).split(‘;’)[0]
                    try:
                        modsince = datetime.datetime.strptime(dt, "%a, %d %b %Y %H:%M:%S %Z")
                        if modsince >= lastmod:
                            print "HTTP: No new version of %s" % path
                            self.send_response(304)
                            return
                    except:
                        pass

                self.send_response(200)
                self.send_header(‘Cache-Control’, ‘max-age=864000′)
                self.send_header(‘Expires’, "Fri, 30 Jan 2010 12:00:00 GMT")
                self.send_header(‘Content-Length’, info[ST_SIZE])
                self.send_header(‘Last-Modified’, lastmod.strftime("%a, %d %b %Y %H:%M:%S GMT"))
                self.send_header(‘Content-Type’, ctype)
                self.end_headers()
                self.copyfile(open(constants.MonitorPath + ‘\\’ + path, ‘rb’), self.wfile)
            else:
                print "MONITOR:Can’t find %s" % (constants.MonitorPath + ‘\\’ + path)
                self.send_error(404, "Unknown url %s" % self.path)
      

So, this piece of code creates a "MonitorServer" object  and defines the "GET" function.  In there it parses the requested URL, and if it’s in one of a few selected forms then send the  dynamically generated content.  Otherwise, send the requested file directly to the User.  As a security precaution, I don’t allow the user to specify a path (the path is stripped and the file must exist within 1 specific directory).  If the browser provides a "If-None-Match" header, then I return a 304 code which indicates the Cache is up-to-date.  If they provide a "If-Modified-Since", then I parse the date and compare it against the file, and returna 304 if appropriate.  One interesting thing I learned from this is that the Browser doesn’t actually return the date of the file, but rather returns the "Last-Modified" you send it, in identical formatting. 

So far, this works great!  It significantly improved the response of the server and has greatly reduced load-times (Even though it’s on a local-network).
[tag:python][tag:webserver][tag:caching]

Anyone who’s used Autodesk’s 3D Studio Max for a while knows about BackBurner.  BackBurner is the Queue-controlling software they ship with Max to allow you to render jobs across multiple scenes in a naive way.  Rather than pooling resources to work on a single frame, it hands out individual frames of the sequence to each node in the backburner queue.  (You can pool resources using the "Strip" rendering, but it’s pretty much the same thing).  However, the way we use BackBurner at work has proven to us that it’s just a piece of crap.  Thus, I wrote my own queue manager that I’ve dubbed "Freezerburn".  Let me elaborate.. 

In our environment, we typically get a dataset of 1000 or so timesteps.  We extract 1000 isosurfaces (one from each timestep), and then need to render these 1000 files into 1000 frames of an animation (For those interested, we typically save the isosurfaces as Stanford PLY files and use the PLY import plugin I wrote, along with a MaxScript, to convert them all into MAX files).  So we create a simple setup scene with the context, lights, and camera.  We then use another custom MaxScript I developed to embed a callback into the Max File’s "onLoad" event that will look at the current frame number, merge in the appropriate Max file, and apply whatever material properties we need.  That way we have a single MAX file that can render all 1000 frames of the animation.  Unfortunately, since it’s in the onLoad, it means we need to load this Scene file from disk for each image.

With Backburner, you have no control over this.  With a large scene, it typically starts off assigning individual frames to nodes, which is exactly what we want.  After those complete, however, it starts assigning larger & larger batches of frames to each node (to reduce overhead spent loading the file & transmitting information).  This means that the onLoad callback is only triggered once but renders multiple frames, which isn’t what we want.

The technical amongst you may be wondering "Why not just use preRenderFrame"? Well, there’s a few reasons while the various preRender callbacks won’t work like that:

1. preRender – Only called once at the beginning of the entire render, basically no better than onLoad
2. preRenderFrame - Called between every frame, but after geometry is cached.. Attempting to change geometry (eg. Merge in a new Max File) results in a SegFault
3. beginRendering* – Same as preRenderFrame
4. preRenderEval - I had high hopes for this when I first saw it, but it seems to operate identically to preRender (only once at the beginning of the entire render)

So, the only way to guarantee that backBurner will process the frames as individual jobs, is to submit each frame as an individual job.  Now, we have MaxScripts to do this for us with a single buttonclick, so it’s not that bad.  However, Backburner simply can’t take it.  BackBurner was designed to handle a handful of several-thousand-frame jobs, and does so beautifully.  However, hand it a few thousand single-frame jobs, and it dies.  It dies horribly.  It dies a slow excruciating death as you watch job submission times go from half a second to 3 or 4 minutes per.  It’s truly sad and ridiculous.

So my solution was to use my new Python skills to write our own queue manager.  It’s written entirely in Python, and offers the following features:

• Queue & priority control
• For Mental Ray scenes it gives you % complete on each of the 3 major stages of render (Preprocessing, Final Gather, Rendering)
• Vastly improved "at-a-glance" status information
• Controllable via a Web Interface (with SSL certificates and CAC authentication which is required where I work)
• And much much more…

Over the next few weeks I’ll post screenshots and articles of how it works, to share the knowledge I gleamed about Python, Windows Programming, HTTP servers, and 3d Studio Max.  So be sure to come back!
[tag:autodesk][tag:backburner][tag:python]  

A friend of mine tipped me off to the fact that MAKO finally made their IPO last month.  I had completely missed it (which is probably a good thing).  So I hit up Google’s Financial pages to see the trading charts..

While this may look good at a glance, a quick look shows nothing of note.  They opened just under $10, spiked to $12, and settled down around 10.50 for today.  What I find more interesting is the harsh commentary on it… Like the Wall Street Journal:

MAKO closed at $9.18 a share on the Nasdaq, down 8% from its IPO price of $10. It sold 5.1 million shares at the low end of a reduced $10 to $11 price range; it originally was expected to price between $14 to $16 through underwriters J.P. Morgan Chase & Co. and Morgan Stanley.
…
Meanwhile, Fort Lauderdale, Fla.-based MAKO specializes in making robotic instruments and implants for knee surgery. Its products, which were first cleared by the Food and Drug Administration, allow surgeons to avoid complete joint replacement for early-to-mid-stage osteoarthritic knee disease, and instead resurfaces damaged joints and places implants through a small incision.

MAKO has never been profitable and expects to continue losing money as it develops its business.

So, looks like noone will be selling their shares and buying a Yacht anytime soon.  Guess we’ll just have to wait and see how it plays out.
[tag:mako][tag:stockmarket]