Follow Me

Dropbox. ?Spread the word.  The first 2GB is free!

Journal Index
« Getting Good Fur from 3ds max | Main | Peanut Candy Bar »

De-constructing Vray's Anti-Aliasing

Many people take for granted the details involved in getting good anti-aliasing.  Sometimes, people don't know exactly which settings should be changed to get the AA quality up without blowing out the render speed for no reason.

Here is the truck rendering at the beginning. Notice how bad the anti-aliasing is overall. 

click to enlarge


Here's a close up that shows the worst parts of this rendering.  Look at the first image.  The grill is so bad it looks like tin foil!  The other image is so bad, that the highlight line is broken in 3 places!  This is awful.  Then, wait till you get this animating.  These spot will CRAWL making your CG look even faker.



The first thing I want to talk about is understanding the display of the anti-aliasing samples.  This next image shows what happens when you render with the "Show Samples" checkbox turned on. Alot of these concepts work with metal ray also, but I will talking specifically about Vray. Click the image below for a larger one.

click to enlarge

 Vray is a raytracer. I does things by throwing rays at the image.  When you throw more rays at a pixel in an image, that pixel begins to smooth with the pixels around it and looks anti-aliased.  (Some programmer is rolling his eyes at that explanation.) The colors in the sample image represent where the renderer is putting it's rays.  Dark blue is less rays, and light blue is more. Broad areas of the image that have similar color should be darker blue, and areas of high detail should be lighter.  

.0011.0The balance of this is all about the  Color Threshold value in the AA rollout. This spinner's range is from 0 -1.  0 will push your sampler to light blue everywhere (bad cause it's slow) and 1 will push it to dark blue everywhere (Bad cause it will look like shit.) Here are 2 settings, .001 and 1.0  So, with this in mind, you can now balance the threshold to your needs.


Now, experience tells me we can solve a lot of this with a different anti-aliasing method.   The image above is Adaptive subdivision.  This type of AA does under-sampling. (Notice the -1 set as the min rate)  And while this really helps with the speed of an image, sometimes, it just doesn't do the trick. Under-sampling can lead to small details being lost. So lets switch our AA over to Adaptive DMC (Deterministic Monte Carlo if you're wondering) Now with Adaptive DMC chosen, turn off the setting "Use DMC sampler thresh" This will link your adaptive threshold with the renderer's general DMC noise setting, and for now, we want to just work with the clr threshold in the AA rollout.  The default  settings are 1 min and 4 max, sampler threshold is .01. 

Adaptive DMC 1 - 4 .01Notice how the DMC sampler is different than the Subdivison one?  To me, it's like the image has greater contrast and seems to overlap the problem areas with more light blue coloring.  This is much better now, but I need more samples at the high end.  I'll change the upper sample from 4 to 6.

Adaptive DMC 1 - 6 .01Ok, a little better. It's hard to see in the RGB image, but you can see that the grill is looking a bit better.  I'll reduce threshold from .01 down to something smaller like .005. Lets see where that gets us.

Adaptive DMC 1 - 6 .005Pretty damn good now.  Probably a lot slower, but much better.  I think in the end, I didn't go this high.  I left it at .01 because of what I did next.  Can you see that the edges of the grill are still a little noisy? Do you see those little dots around where the hood and grill meet?  At this point, the AA isn't the problem, it's really  the chrome itself is.  The chrome is trying to reflect an HDRI image. Those glitches are due to the default chrome not having enough reflection samples.

 I'm gonna double the samples to 16, which will help when reflecting the corners. Now, while I'm here, lets talk about the glossiness.  When glossiness is at 1, I believe Vray only throws a single ray at it. (Since it's like a mirror)  When glossiness gets below 1, Vray then considers it a glossy reflection and uses the subdivision setting below.  So, for this image I want to get a smoother chrome, so I will also change the glossiness to .9 to ensure that more rays are throw at the chrome itself.  

Adaptive DMC 1 - 6 .005 | Gloss .9, Subdivs 16, Area Filter

 Notice how the small highlight glitches near the hood are gone! The edges around the grill are a cleaner, and I can move forward with confidence on rendering the animation out later. However, there is still one more thing that can soften this image.  That's changing the filtering type.  "Area" is a filtering type that was created by the max scanline rendering back in the day.  I think release 4 implemented many known anti aliasing filters that are in max today.  Back 10 years ago, using these filters with a SD frame (720x540) would give blurry results.  But today, with everything being in HD (1920x1080) these filters are now worth even more.  I like the Soften filter.  I did some tests with all the types, and Soften seems to smooth the results without looking very "blurry".

 Left Area Filtering - Right Soften Filtering

Final Rendering


Final Samples

PrintView Printer Friendly Version

EmailEmail Article to Friend

References (4)

References allow you to track sources for this article, as well as articles that were written in response to this article.
  • Response
    NFL is truly 1 of the largest sports in America. It has a big following.
  • Response
    Football is definitely one particular of the most significant sports in America. It has a major following.
  • Response
    NFL is genuinely a single of the most significant sports in America. It has a significant following.
  • Response
    Response: copysniper
    Ruff's Stuff - CG Blog - Journal - De-constructing Vray's Anti-Aliasing

Reader Comments (3)

--From DADAL over on the chaos groups board--
I've finally read it ! Quite interesting, but I'd go with Blend filter... it allow to blend fixing lights hot spots + keep the edge when necessary...

By the way ruffstuff what is your idea of filtering at 5-10-15k render size ?

You know I think that Blend filter was an idea that Frank Delise threw out to an engineer and he implemented it. Back then he was the product designer and was an application engineer. We would always throw a blur on a second layer in Photoshop to soften up the look. (The blur and the opacity of the blurry layer) But now I'd rather keep it sharp and do that effect in post. At the same time, I anti aliasing does more at the sub-pixel level than some blur in composite.

As to the larger formats, I'd have to say that I haven't really thought about it that much. Are you thinking the anti aliasing should be scaled with the render size?

December 4, 2010 | Registered CommenterFred Ruff

Hey Ruff,

Nice article about filtering.

If I could make a suggestion it would be worth mentioning "Clamping Output" - this is most often the factor that I have to explain to people who are having trouble with aliasing. No amount of sampling will remove the apparent aliased pixels in the render if the pixel are overbright (i.e greater than 1). Of course doing this makes the image not HDR anymore.

December 15, 2010 | Unregistered CommenterAdam

Thanks for the feedback. You know, Iv'e never tried clamping the output, but I can see how that would help. I'm a fan of linear worklow so i probably wouldn't clamp it that often since like you said the resulting image about be clamped. But it's a good point to bring up.

December 15, 2010 | Registered CommenterFred Ruff

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>