Plausible Environment Lighting in Two Lines of Code

a group post by McGuire, Evangelakos, Wilcox, Donow, and Mara
(Because of intense interest in this post, we archived it as a technical report at http://graphics.cs.williams.edu/papers/EnvMipReport2013/ suitable for scientific citation)


This post describes a trick used in the G3D Innovation Engine 9.00 to produce reasonable real-time environment lighting for glossy surfaces. It specifically adds two lines of code to a pixel shader to reasonably approximate Lambertian and glossy reflection (under a normalized Phong model) of a standard cube map environment. This trick resulted from an afternoon discussion/challenge last week in the Williams Computational Graphics lab between Morgan McGuire (that's me), my summer 2013 research students, Dan Evangelakos (Williams), James Wilcox (NVIDIA and University of Washington), Sam Donow (Williams), and my NVIDIA colleague and former student Michael Mara.

There are many more sophisticated ways of computing environment lighting for real-time rendering (see Real-Time Rendering's survey or a more informal Google search). Unfortunately, those techniques tend to require an offline asset build step, so they can't be directly applied to dynamic light probes, which are particularly interesting to us...although there are some clever things that one can do with spherical harmonics to reduce the cost of that precomputation a lot. Those methods also tend to require a specific setup for the renderer that doesn't work with our default shaders. For research and hobbyist rendering, one often has to work with off-the-shelf or legacy assets. This situation also arises in production more than one might think, especially when working with games that run across a variety of devices.

A series of increasingly shiny teapots, before and after applying this technique.
In the before row, only the perfect-mirror (rightmost) teapot reflects the environment.

Z-Prepass Considered Irrelevant

The G3D Innovation Engine directly supports easy switching between forward, forward+, and deferred rendering. Most programs, including the starter sample, begin rendering with a z-prepass regardless of which mode they are operating in. This ensures that (except for translucent surfaces and partial coverage) shading occurs once per sample. The z-prepass operation is a depth-only render pass that bypasses the fragment shader (except when a surface has an alpha mask), and it processes fragment-bound scenes in about 1/3 the time of a regular shading pass on NVIDIA GPUs, which have special fixed-function support for depth-only rendering.

A Really Smart Enum in C++

For the G3D Innovation Engine, I wanted much richer enumerated type functionality than C++ enum (or even recent extensions) provide, without the template error nightmares and complexity of something like the Boost library's enums or X macros. I also did not want to use non-standard compiler extensions or an offline code generator; portability and minimal maintenance are important for my multi-platform code and small development team. I call my solution "really smart enums" because they extend the so-called C++ Intelligent Enum design pattern.  The solution that I used allows all of the following:

Tracewords Makeover

I'll be releasing Tracewords shortly.  This is a casual word game written with codeheart.js that is designed for one-handed (one-thumb, really) play on mobiles. I implemented the mechanics a year ago. To prepare for release I needed three major features:

Favorite Films from Annecy 2013

Top three:

1. Miss Todd

Miss Todd

Kristina Yee 2013
A paper-cutout musical, historical fiction (pleasantly heavy on facts), feminist tract, and fun story. That it was also a student graduation film is amazing. This was my favorite film of the festival.

2. Will
Eusong Lee 2012
The saddest and most emotionally effective short I have ever seen at the festival.  I cried and wanted to run from the theatre once it became clear where this was going.  If the last 15 seconds were cut, then it would be perfect.

3. La Grosse Bête
Pierre-Luc Granjon 2013
Well animated, with a compelling story. This also acts as a precise allegory in a world debating civil rights vs. security.  The timing and direction are flawless and masterful.

Other highlights:

Les Voiles du Partage 
Pierre Mousquet and Jerome Cauwe 2013
Mad Max meets Charles Bronson for a fast-paced dystopian satire. It is great to have some films that are just fun!

Rabbit and Deer

Nysuzi es Oz (Rabbit and Deer)
Peter Vacz 2012
2D meets 3D, and science is the original sin.

Duku Spacemarines
Nicolas Liautaud, Alice Suret-Canale, Hugo Paquin, and Nicolas Dubois 2012
For every student whose film or game design included "and one more awesome thing..."

Beach
Pawel Prewencki 2012
A simple vignette of the beach, with one cynical note.

Betty's Blues
Claymation and animated woodcut bringing to life a set of three blues songs; the stories are dark but presented more cheerfully than you might expect.

Trespass
Paul Wenninger 2012
A stop-motion live-action journey of epic scope; the music-video like novelty wears off after a minute and the impact then begins. I believe that the actor was also the director.  His performance is particularly impressive--it would be hard to maintain that composure over such long stop-motion scenes.

But Milk is Important

The Caketrope of Burton's Team
Axelandre Dubosc 2012
A chocolate cake zoetrope in homage to Tim Burton.

But Milk Is Import
Anna Mantzaris, Eirik Gronmo Bjornsen 2012
Shy guy meets shy girl in stop motion with lots of knitting and felt.

Kick-Heart
Masaaki Yuasa
Japanese insanity, with manga tropes.

History of Pets
Kris Genijn 2013
The pets don't fare so well in this funny and lighthearted tragedy.

A Girl Named Elastika
Guillaume Blanchet
The journey of a rubber band girl across a cork board in thumb-tack stop motion.  A fresh look and self-deprecating presentation.

Balance and Swing

Balance and Swing

Anne Beal 2012
A beautiful watercolor from RISD.

Aux Gambettes Gourmandes
A light powder (think sand and sugar painting) animation with a fun and accurate twist.

Boles
Spela Cadez 2013
Extremely smooth and self-contained Kafkaesque stop-motion. Women in Annecy shorts regrettably tend to caricatures: nurturing mothers, children, thin beautiful 20-something sex objects, grandmothers, or extremely fat.  This one had a much more interesting character.


The program this year seemed heavy on slow, depressing films about child abuse and refugees. These are seldom my own favorites because, while the subjects are of course important, they very hard to present in a novel or nonobvious way, and thus tend to disappoint as art and are underrepresented in my list.

I noted fewer films from North America and from Japan this year.  Monsters University was shown in 3D and was good, and technically perfect although I found the story and cinematography less compelling than Pixar's previous best films.  New work by Monty Python and Bill Plympton was good but not as exceptional as we have come to expect.

Bonus! A favorite from last year is now online: Una Furtiva Lagrima

---

Morgan McGuire is a professor of Computer Science at Williams College and a professional game developer. He is the author of The Graphics Codex, an essential reference for computer graphics that runs on iPhone, iPad, and iPod Touch.

Big Graphics Codex changes in Aug & Sept

Over 1000 of 3D programmers, researchers, and graphics students already use the Graphics Codex app--I can see not only the original sales, but how many download the free updates each month. For the last two years the app has been essentially underground. Sales are driven solely by word of mouth and I have no advertising. This was a quiet start appropriate for an app that began by just releasing my course lecture notes in a mobile form factor, and only later grew towards comprehensive coverage.

Now that the Graphics Codex has 224 entries, including an entire course of lecture notes on physically-based rendering and new topics ranging from HTML5 to the biology of the human eye, it is time to upgrade both the app and the distribution strategy. In mid-August, I'll release version 2.0.  I have already submitted it to Apple and am just waiting for App-store approval for release. This contains a reworking of the UI code for responsiveness and clarity. Here's the full change log:

This really is a "2.0"--the entire scrolling, selection, and layout engines have been rewritten for performance and robustness.  The 1.9 version introduced the iOS 7 interface, which is retroactively available back to iOS 5.

I plan two more rapid-fire updates, for August and September. The first will include more UI features that need additional testing, such as the interactive graphs and retaining your place on a page when navigating between topics. Other UI features like printing are on the task list but may not make these short-term updates. I try to ship once a month with whatever is ready, rather than holding a release on a specific feature.

This update will be accompanied by an academic marketing campaign run by Alice Peters (cofounder of A K Peters), whom I've had the pleasure to work with on many projects in the past. It is time to get the word out about the app as the perfect complement for traditional textbooks in the classroom. In the last six years (long before it was publicly available!), I used the Graphics Codex in teaching at Williams as a companion to Fundamentals of Computer Graphics, Real-Time Rendering, and Computer Graphics: Principles and Practice in different semesters. I tuned the material for the app to work well with all three of these popular texts.

The September update will follow shortly after Apple's expected early-September product announcements. It will include additional iOS 7 support and increased coverage of GPU and rasterization topics. I'll post more information about this and another significant announcement about the Graphics Codex at the end of August.

Morgan McGuire is a professor of Computer Science at Williams College and a professional game developer. He is the author of The Graphics Codex, an essential reference for computer graphics that runs on iPhone, iPad, and iPod Touch.

Starfield Shader

I'm preparing assets for an upcoming mini-game jam/hackathon.  My current plan is to make a spaceship video game inspired by Galaxy Trucker, in which you build and then fly a ship.  This post is about the starfield shader that I created by modifying Kali's amazing Star Nest shader on Shadertoy to fit into the game framework.

The starfield

The starfield renders in 3.7ms at 1080p on GeForce 650M (the Kepler GPU in a 2012 Macbook Pro) on Windows 7 64-bit, including upsample time.  It is procedural, so it has effectively infinite extent and fine detail down to the pixel level, and has parallax as it scrolls.

I was only willing to spend 25% of the frame time on the background. To keep the performance high, I render downsampled by 2.5x in each direction to an RGB8 texture and then bilinearly interpolate up to full screen resolution.

Here's the GLSL shader:

// -*- c++ -*-
// \file starfield.pix
// \author Morgan McGuire
//
// \cite Based on Star Nest by Kali 
// https://www.shadertoy.com/view/4dfGDM
// That shader and this one are open source under the MIT license
//
// Assumes an sRGB target (i.e., the output is already encoded to gamma 2.1)
#version 120 or 150 compatibility or 420 compatibility
#include <compatibility.glsl>

// viewport resolution (in pixels)
uniform float2    resolution;
uniform float2    invResolution;

// In the noise-function space. xy corresponds to screen-space XY
uniform float3    origin;

uniform mat2      rotate;

uniform sampler2D oldImage;

#define iterations 17

#define volsteps 8

#define sparsity 0.5  // .4 to .5 (sparse)
#define stepsize 0.2

#expect zoom 
#define frequencyVariation   1.3 // 0.5 to 2.0

#define brightness 0.0018
#define distfading 0.6800

void main(void) {    
    float2 uv = gl_FragCoord.xy * invResolution - 0.5;
    uv.y *= resolution.y * invResolution.x;

    float3 dir = float3(uv * zoom, 1.0);
    dir.xz *= rotate;

    float s = 0.1, fade = 0.01;
    gl_FragColor.rgb = float3(0.0);
    
    for (int r = 0; r < volsteps; ++r) {
        float3 p = origin + dir * (s * 0.5);
        p = abs(float3(frequencyVariation) - mod(p, float3(frequencyVariation * 2.0)));

        float prevlen = 0.0, a = 0.0;
        for (int i = 0; i < iterations; ++i) {
            p = abs(p);
            p = p * (1.0 / dot(p, p)) + (-sparsity); // the magic formula            
            float len = length(p);
            a += abs(len - prevlen); // absolute sum of average change
            prevlen = len;
        }
        
        a *= a * a; // add contrast
        
        // coloring based on distance        
        gl_FragColor.rgb += (float3(s, s*s, s*s*s) * a * brightness + 1.0) * fade;
        fade *= distfading; // distance fading
        s += stepsize;
    }
    
    gl_FragColor.rgb = min(gl_FragColor.rgb, float3(1.2));

    // Detect and suppress flickering single pixels (ignoring the huge gradients that we encounter inside bright areas)
    float intensity = min(gl_FragColor.r + gl_FragColor.g + gl_FragColor.b, 0.7);

    int2 sgn = (int2(gl_FragCoord.xy) & 1) * 2 - 1;
    float2 gradient = float2(dFdx(intensity) * sgn.x, dFdy(intensity) * sgn.y);
    float cutoff = max(max(gradient.x, gradient.y) - 0.1, 0.0);
    gl_FragColor.rgb *= max(1.0 - cutoff * 6.0, 0.3);

    // Motion blur; increases temporal coherence of undersampled flickering stars
    // and provides temporal filtering under true motion.  
    float3 oldValue = texelFetch(oldImage, int2(gl_FragCoord.xy), 0).rgb;
    gl_FragColor.rgb = lerp(oldValue - vec3(0.004), gl_FragColor.rgb, 0.5);
    gl_FragColor.a = 1.0;
}

The preamble uses some G3D-specific shader preprocessing. #expect declares a macro argument, G3D allows multiple version numbers and promotes the highest one to the first line, and implements #include.  The compatibility.glsl file smoothes over differences between GLSL versions and between GLSL and HLSL.

At its heart is the "KaliSet" iterated function p := | p | / | p |² - k.  I tuned the constants for the specific look I was targeting and for performance.  It is of course very performance sensitive to the number of iterations of the inner loop and the number of volumetric steps.  I also extracted some of the expressions into uniforms in a way that the Shadertoy framework wouldn't have allowed for the original.  I tried using the max component or the average component in place of the expensive length() call, but this creates a 3-fold symmetry on some "suns" that is undesirable.

In my game, the background will move slowly as the ship flies. Very bright single pixels flicker in and out of existence as they are undersampled with the unmodified shader, so I suppressed these cheaply using derivative instructions and blending in against the previous frame.  The previous frame blending is primarily there to create motion blur and soften the background under motion to keep attention on the shops. I also wiggle the screen by a random offset of up to 1/3 pixel when the camera is still.  This makes stars appear to twinkle and closes the visual difference between the slightly-incoherent rendering under motion and when still.

Morgan McGuire is a professor of Computer Science at Williams College and a professional game developer. He is the author of The Graphics Codex, an essential reference for computer graphics that runs on iPhone, iPad, and iPod Touch.