Project Rembrandt: Difference between revisions
Line 23: | Line 23: | ||
:*<u>Ambient pass</u>: the diffuse buffer is modulated with the ambient color of the scene and is drawn as a screen-aligned textured quad | :*<u>Ambient pass</u>: the diffuse buffer is modulated with the ambient color of the scene and is drawn as a screen-aligned textured quad | ||
:*<u>Sunlight pass</u>: a second screen aligned quad is drawn and a shader computes the view position of every pixel to compute its diffuse and specular color, using the normal stored in the first stage. The resulting color is blended with the previous pass. Shadows are computed here by comparing the position of the pixel with the position of the light occluder stored in the shadow map. | :*<u>Sunlight pass</u>: a second screen aligned quad is drawn and a shader computes the view position of every pixel to compute its diffuse and specular color, using the normal stored in the first stage. The resulting color is blended with the previous pass. Shadows are computed here by comparing the position of the pixel with the position of the light occluder stored in the shadow map. | ||
:*<u>Additional light pass</u> | :*<u>Additional light pass</u>: the scene graph will be traversed another time to display light volumes (cone or frusta for spot lights, sphere for omni-directional lights) and their shader will add the light contributed by the source only on pixels receiving light. | ||
:*<u>Fog pass</u>: a new screen aligned quad is draw and the position of the pixel is computed to evaluate the amount of fog the pixel has. The fog color is blended with the result of the previous stage. | :*<u>Fog pass</u>: a new screen aligned quad is draw and the position of the pixel is computed to evaluate the amount of fog the pixel has. The fog color is blended with the result of the previous stage. | ||
Revision as of 09:23, 28 March 2012
Work in progress This article or section will be worked on in the upcoming hours or days. See history for the latest developments. |
Why this name ?
Rembrandt was a dutch painter living in the 17th century, famous as one of the master of chiaroscuro.
This project is about changing the way FlightGear renders lights, shadows and shades, and aims at making Rembrandt painting style possible in FG.
What is it ?
The idea driving the project is to implement deferred rendering inside FlightGear. From the beginning FlightGear had a forward renderer that tries to render all properties of an object in one pass (shading, lighting, fog, ...), making it difficult to render more sophisticated shading (see the 'Uber-shader') because one has to take into account all aspects of the rendering equation.
On the contrary, deferred rendering is about separating operations in simplified stages and collecting the intermediary results in hidden buffers that can be used by the next stage.
- First stage is the Geometry Stage
- we render all the scene into 4 textures, using multi render targets, to do it in one pass: one for the depth buffer, one for the normals (lower left of the image), one for the diffuse colors (lower right) and one for the specular colors (upper right).
- Next stage is the Shadow Stage
- we render the scene again into a depth texture from the point of view of the lights. There will be one texture for every light casting shadows.
- Then comes the Lighting Stage, with several substages
-
- Sky and cloud pass: the sky and the clouds are first drawn using classical method.
- Ambient pass: the diffuse buffer is modulated with the ambient color of the scene and is drawn as a screen-aligned textured quad
- Sunlight pass: a second screen aligned quad is drawn and a shader computes the view position of every pixel to compute its diffuse and specular color, using the normal stored in the first stage. The resulting color is blended with the previous pass. Shadows are computed here by comparing the position of the pixel with the position of the light occluder stored in the shadow map.
- Additional light pass: the scene graph will be traversed another time to display light volumes (cone or frusta for spot lights, sphere for omni-directional lights) and their shader will add the light contributed by the source only on pixels receiving light.
- Fog pass: a new screen aligned quad is draw and the position of the pixel is computed to evaluate the amount of fog the pixel has. The fog color is blended with the result of the previous stage.
- All lighting computations are accumulated in a single buffer that will be used for the last stage, in addition of the one computed by the Geometry stage.
- In the end, the Display Stage, with optional Post-Processing effect
- The results of the previous buffers are pushed to the main framebuffer to be displayed, optionally modified to show Glow, Motion blur, HDR, redout or blackout, screen-space ambient occlusion, anti-aliasing, etc...
In FG, we end the rendering pipeline by displaying the GUI and the HUD.
All these stages are more precisely described if this tutorial that is the basis of the current code, with some addition and modifications.
Caveats
Deferred rendering is not capable to display transparent. For the moment, clouds are renderer separately and should be lit and shaded by their own. Transparent surfaces are alpha-tested and not blended. They would have to be drawn in their own bin over the composited image.
It also don't fit with depth partitioning because the depth buffer should be kept to retain the view space position, so for the moment, z-fighting is quite visible. Depth partitioning with non overlapping depth range might be the solution and should be experimented at one point.
The glow pass can make certain MFD (that use emissive color) unreadable because blurred. Should be treated as transparent.
Implementation
Repositories
Source code and data are available in gitorious repositories :
- https://gitorious.org/~fredb/fg/fredbs-simgear
- https://gitorious.org/~fredb/fg/fredbs-flightgear
- https://gitorious.org/~fredb/fg/fredbs-fgdata
The code and data are in project/rembrandt branches
The code is not yet optimized
and may put the graphic card under pressure.
Rendering of transparent surfaces
Transparent surfaces are detected by OSG loader plugins and their state set receive the TRANSPARENT_BIN rendering hint. In the culling pass, the cull visitor orders transparent surfaces in transparent bin. In a cull callback attached to the Geometry camera, after the scenegraph traversal, the transparent bins are removed from the render stage and saved in a temporary collection. In a cull callback attached to the Lighting camera, after the scenegraph traversal, the transparent bins saved at the previous stage, are added to the render stage of the Lighting camera with a high order num. That way, the transparent surface are drawn on top of the scene lighted from the Gbuffer. |
Memory consumption
For each camera defined in the camera group, there is a separate shadow map, so the video memory usage is :
- G-buffer and Lighting buffer: 48 bytes per pixel. For an HD screen (1920x1080) memory requirement is 95 Mb
- Shadow map: 3 x shadow_map_size x shadow_map_size bytes (if size is 8192, whole map size is 192 Mb
Not counting textures, display list or vertex buffers for models and terrain
3 HD screens require 237 Mb of memory for the buffers (shadow excluded), you're asking 3x8192x8192x3 = 576 Mb (megabytes) of memory for the shadows alone.
If you are seeing error messages during startup or FlightGear doesn't start up properly, it's probably because you don't have enough free video memory. Reduce the size of the shadow map in preferences.xml by locating
<map-size type="int">8192</map-size>
And put 4096 or 2048 instead. You can also use a startup parameter: --prop:/sim/rendering/shadows/map-size=2048
Guidelines for shader writers
Predefined uniforms
These glsl uniforms don't need to be declared in the effect file.
Name | Type | Purpose |
---|---|---|
fg_ViewMatrix | mat4 | In fullscreen pass only, view matrix used to transform the screen position to view direction |
fg_ViewMatrixInverse | mat4 | In fullscreen pass only, view matrix inverse used to transform the screen position to view direction |
fg_ProjectionMatrixInverse | mat4 | In fullscreen pass only, projection matrix inverse used to transform the screen position to view direction |
fg_SunAmbientColor | vec4 | |
fg_SunDiffuseColor | vec4 | |
fg_SunSpecularColor | vec4 | |
fg_SunDirection | vec3 | |
fg_FogColor | vec4 | |
fg_FogDensity | float | |
fg_Planes | vec3 | Used to convert the value of the depth buffer to a depth that can be used to compute the eye space position of the fragment |
fg_BufferSize | vec2 | Dimensions of the buffer, used to convert gl_FragCoord into the range [0..1][0..1] |
osg_ViewMatrix | mat4 | Defined by OSG, used only when working on actual geometry |
osg_ViewMatrixInverse | mat4 | Defined by OSG, used only when working on actual geometry |
They still have to be declared in the fragment or the vertex shader to be used.
Geometry Stage
The Geometry Stage is there to fill the G-buffer. Shading doesn't occur at this stage, so light or fog computation should not be part of the shader. The required operation in the Fragment Shader is to fill every individual buffer with sensible value :
- Depth buffer, modified with gl_FragDepth, will record the distance between the fragment and the camera. Default behavior is to avoid to touch it, living the GPU rasterizer doing sensible things by interpolating vertex gl_Position from the Vertex or the Geometry Shader. If altering the computed depth is required, like in the Urban shader, the value of gl_FragDepth should be set.
- Normal buffer, modified with gl_FragData[0].xyz, will record the normal of the fragment in eye coordinates. gl_FragData[0].w is reserved for future use. The interpolated normal is usually simply stored but bump mapping or relief mapping affecting the normal can be computed here.
- Diffuse color buffer, modified with gl_FragData[1].rgb, will hold the unshaded color of the fragment, usually modulating the material diffuse+ambient color with the texture map. Diffuse color from environment mapping should also go here.
- Specular color, modified with gl_FragData[2].rgb, and specular shininess in gl_FragData[2].a, will retain the specular color of the fragment.
- Emission color, modified with gl_FragData[3].rgb
In anyway, don't use gl_FragColor as it is incompatible with MRT (Multi Render Target) and would affect the four last buffers with the same value. In that case, the model will glow (emission buffer initialized) and parts will disappear at certain view angles because normals are not initialized properly.
This layout is going to change in the official repository when the merge will be done. The goal of this change is to fill more useful data into less texture memory and get rid of a stupid IP issue on float textures
Additional light pass
There would be a single shader for each light type used. The plan is to create lights like animations in the model XML file. The light shader will retrieve scene geometry by combining screen space position converted in view space ray by the inverse of the projection matrix (an helper function should be provided), and the fragment depth at that screen position read from the depth buffer. With the help of the fragment normal, the diffuse and specular color and the properties of the light the shader implements, it will be possible to add to the lighting buffer the contribution of the light rendered.
Fog Pass
Using the fragment depth, it will be possible to compute any fog distribution. For the moment, the simple fog equation is implemented.
Bloom Pass
This is a two-pass effect that blurs the lighting buffer in a small texture. This texture is then added to the lighting buffer at the display stage.
Required Effects
Several pass are implemented using the effect system. For this purpose, some effects are referenced in the core code using reserved names. these effects are:
Name | Kind | Purpose |
---|---|---|
Effects/ssao | Works on a full screen quad | Compute ambient occlusion from the normal buffer and the depth buffer |
Effects/ambient | Works on a full screen quad | Copies the diffuse color buffer multiplied by the ambient light to the lighting buffer. Ambient Occlusion can also affect ambient light. |
Effects/light-spot | Works on real geometry of the light volume | Computes the light contribution of a spot light defined in a light animation having a light-type of spot |
Effects/fog | Works on a full screen quad | Computes the fog from the G-buffer and the lighting parameters |
Effects/display | Works on a full screen quad | Renders the composite final image from the G-buffer and the lighting buffer |
Guidelines for modelers
Porting aircraft
- Rembrandt computes shadows => no more fake shadows in the model
- Rembrandt computes ambient occlusion => no ambient occlusion baked into textures
- Rembrandt has light => static lightmap are not needed, emissive color to see models at night is not needed and would interfere
- Rembrandt has glow => incorrectly used emissive colors may blur displays and make some text unreadable. Light size may have to be adjusted
- Rembrandt has strict needs with shaders => shaders need to be adjusted to comply with the new framework otherwise the view will be plain wrong
- Rembrandt can't do transparent surfaces => transparent surface need to be properly registered to render them with the classical path
Registering all translucent surfaces
Every model is, by default, rendered using the Effects/model-default effect. This effect initialize the G-buffer, ignoring transparent surfaces, by doing alpha testing and rendering all the geometry in the default bin. It is not possible to redirect rendering to transparent bins when the associated texture has alpha channel because most models use a single texture atlas and even opaque parts are rendered with texture with alpha channel.
If a model needs to have transparent or translucent surfaces, these surface objects need to be assigned a different effect that sets explicitly the render bin to "DepthSortedBin", or sets the rendering hint to "transparent". This tells the renderer to render this object using forward rendering, so lighting and fog need to be enabled, and if a shader program is used, they should be computed in the classical way. The Effects/model-transparent can be used to register simple transparent/translucent surfaces.
If opaque surface need to have special effect, for example to apply bump mapping, this effect should use the "RenderBin" bin, or the rendering hint set to "opaque", and the G-buffer needs to be initialized correctly in the Geometry stage.
Adding lights to a model
There are two things to consider: the appearance of the light source and the illuminated area. For the appearance of the light source (what you see when you look at the bulb), you need a model with an emissive material that will produce the glow effect and that is visible at night.
For the effect of the source on its environment (the lit area), we must have in the 3D model (the .ac file) a volume that includes the effect (Light Volume). It can be a large cone for spotlights or a sphere for point light. It's important that the light volume is closed, convex and it's normals are oriented outward.
The light volume must be part of the geometry of the model and be referenced in the animation file. No need to add a color or an effect to this volume. Light calculation is only done on the fragments covered by the light volume, but has no influence on the color or the attenuation of the light.
All available animations are possible on the light volume, except material and texture. It is not possible to change color of lights for the moment, except switching to another animation. Axis and position are in object space and are transformed by the subsequent animations.
Spotlights
<animation>
<type>light</type>
<light-type>spot</light-type>
<name>LightSrcRight</name>
<object-name>LightRight</object-name>
<position>
<x>0.169</x>
<y>0.570</y>
<z>0.713</z>
</position>
<direction>
<x>-0.9988</x>
<y>0.0349</y>
<z>-0.0349</z>
</direction>
<ambient>
<r>0.03</r>
<g>0.03</g>
<b>0.03</b>
<a>1.0</a>
</ambient>
<diffuse>
<r>0.7</r>
<g>0.7</g>
<b>0.6</b>
<a>1.0</a>
</diffuse>
<specular>
<r>0.7</r>
<g>0.7</g>
<b>0.7</b>
<a>1.0</a>
</specular>
<attenuation>
<c>1.0</c>
<l>0.002</l>
<q>0.00005</q>
</attenuation>
<exponent>30.0</exponent>
<cutoff>39</cutoff>
<near-m>3.5</near-m>
<far-m>39</far-m>
</animation>
|
|
Point lights
<animation>
<type>light</type>
<light-type>point</light-type>
<name>LightSrcRight</name>
<object-name>LightRight</object-name>
<position>
<x>0.169</x>
<y>0.570</y>
<z>0.713</z>
</position>
<ambient>
<r>0.03</r>
<g>0.03</g>
<b>0.03</b>
<a>1.0</a>
</ambient>
<diffuse>
<r>0.7</r>
<g>0.7</g>
<b>0.6</b>
<a>1.0</a>
</diffuse>
<specular>
<r>0.7</r>
<g>0.7</g>
<b>0.7</b>
<a>1.0</a>
</specular>
<attenuation>
<c>1.0</c>
<l>0.002</l>
<q>0.00005</q>
</attenuation>
<near-m>3.5</near-m>
<far-m>39</far-m>
</animation>
|
|
References
- Deferred Shading by Shawn Hargreaves and Mark Harris
- Deferred Shading Tutorial by Fabio Policarpo and Francisco Fonseca
- Deferred Rendering in Killzone 2 by Michal Valient
- Battlefield 3 Graphics Tech Talk by Johan Andersson
- Rendering Techniques in Split/Second by Jeremy Moore and David Jefferies
TODO List
- Fix shadow rendering when using multi threading in OSG
Implement Cascaded Shadow Map (need to be optimized - frustum calculation and night)- Honor <noshadow> animation directive
- See what happens with glow in fog
- Test multi-screen
- Restore splashscreen
Draw transparent objects with forward rendering (may need to capture the transparent bin from the geometry stage and move it in the display stage)(OK - needs model contribution)Add spotlights as animations (nearly finished)find a solution for ambient and emissive color of material (may need an additional buffer)- implement strength of glow (in the emissive buffer alpha channel)
- provide levels 0 to 5 - we are currently at level 5
- level 0 should be ok for MFDs that are currenly unreadable because blurred
- Provide a shader for transparent objects that could render to the emissive buffer too (using MRT)
Use stencil buffer to limit light range(no - done in light shader)- needed for cockpit light to implement fake shadows and avoid lighting the runway from the cabin through the airframe
- Use effect system instead of hard-coded shaders (mostly done)
- Convert existing shaders to deferred rendering
- Modify shadows to allow multiple casters (limited list)
- Implement a priority list of light sources, based on priority and distance from the viewer
Add new animation to link a light source to a model(need to provide point light animation duplicating spot light)
- Tidy up the architecture
- Restore depth partitioning using depth ranges
- Restore stereo and other options currently available in CameraGroup
- Implement quality vs performance user control
Gallery
Model modification log
- c172p
- Add an effect to the propeller disk (object Propeller.Fast) to put it in a transparent bin
- Models/Airport/apt-light.xml & Models/Airport/apt-light-ba.ac
- Add a spot light animation and a light volume
- Scenery/Objects/w130n30/w123n37/942050.stg
- Change KSFO_light.xml to apt-light.xml
- Aircraft/followme/Models/followme.xml & .ac
- Add light volumes and spotlight animations for headlights
Effect/Shader modification log
- Default shaders
- render to the G-buffer
- Urban effect
- render to the G-buffer
- Spot light effect
- new effect to render spot lights from the animation file
- model-transparent
- new effect to classify transparent surfaces (those that are not bound to the glass shader or other shader that use explicitly the transparent bin )