Tech/LxEngine/Tutorials/Tutorial 3
From Athile
Contents |
Overview
Tutorial 3 adds to the spinning cube from Tutorial 2 by adding more advanced geometry and materials.
Tutorial 3 sets up command-line arguments for the program, loads up an XML document and processes its elements to load the geometry and materials, uses the BlendLoad subsystem to load Blender g .blend files to use as the geometry, uses the ShaderBuilder subsystem to create complex shaders from simple JSON g descriptions, and lastly sets up some basic user controls to cycle through the combinations of geometry and materials.
Controls
-
ESC- exits the application -
G- advances to the next model (geometry) -
F- rewinds to the previous model -
M- advances to the next material -
N- rewinds to the previous material
Concepts Introduced
- The LxEngine Document Object Model
- Engine::globals()
- Engine::parseCommandLine()
- LightSet
- BlendLoad
- ShaderBuilder
Prerequisites
- Tutorial 2
- Basic familiarity with XML
- Basic familiarity with the XML/HTML DOM (Document Object Model)
- Basic familiarity with the JSON data format
Results
If all goes well, you should end up with a viewer app that allows you toggle through various materials and models.
(If you can't see the video below, you may want to try refreshing the page.)
Headers
Let's start with the headers:
//===========================================================================// // H E A D E R S & D E C L A R A T I O N S //===========================================================================// #include <iostream> #include <lx0/lxengine.hpp> #include <lx0/subsystem/rasterizer.hpp> #include <lx0/subsystem/shaderbuilder.hpp> // <-- new #include <lx0/util/blendload.hpp> // <-- new
Two new headers of note have been included since the last tutorial: shaderbuilder.hpp and blendload.hpp. These will be used, respectively, to dynamically build shaders from a convenient JSON-syntax and to load Blender model files.
Program Start-up
Next, let's compare the start-up from the previous tutorial and from this one. The start-up prior to the main loop (i.e. Engine::run()) was fairly simplistic last time. For Tutorial 3, however, we'd like to add some command-line arguments as well as load all the application models and materials from a file (rather than hard-coding that data).
Here's the code from last time. It creates the Engine and then creates an empty Document that we never did anything with.
lx0::EnginePtr spEngine = lx0::Engine::acquire(); spEngine->initialize(); { lx0::DocumentPtr spDocument = spEngine->createDocument();
Command-line parameters
In Tutorial 3, we add several lines of code that add to the Engine::globals() and then make a call to parseCommandLine(). This new code should be a bit self-explanatory as it sets up Tutorial 3's command-line arguments:
lx0::EnginePtr spEngine = lx0::Engine::acquire(); spEngine->initialize(); // // Add several global configuration variables. These will be exported as // command-line options. // spEngine->globals().add("shader_filename", lx0::eAcceptsString, lx0::validate_filename(), "media2/shaders/glsl/fragment/normal.frag"); spEngine->globals().add("params_filename", lx0::eAcceptsString, lx0::validate_filename(), lx0::lxvar::undefined()); spEngine->globals().add("model_filename", lx0::eAcceptsString, lx0::validate_filename(), "media2/models/standard/suzanne/suzanne_subdivided.blend"); spEngine->globals().add("view_width", lx0::eAcceptsInt, lx0::validate_int_range(32, 4096), 512); spEngine->globals().add("view_height", lx0::eAcceptsInt, lx0::validate_int_range(32, 4096), 512); // // Have the Engine parse the command-line, checking if any configuration options // have been set. // if (spEngine->parseCommandLine(argc, argv)) {
In more detail...
The Engine::globals() method returns a special table of variant data that for storing global configuration parameters: i.e. int values, string values, even arrays and nested data. This provides a convenient means for storing "named" application data in a way that will be accessible to the whole application. Using string named variables like this makes it easier for the engine and its components expose this data to the user via command-line arguments (as in this case), more generally via an in-application console and/or scripting language, or any sort of user configurable application-wide variables and settings.
The table entries get three parts:
- a name
- a set of flags in this case describing what type of data that variable can hold
- a validation function to ensure the data being set is a valid value
- a default value.
This is enough information that the Engine object can now expose each of these variables as a command-line parameter on application start-up.
In Tutorial 3, the Engine::parseCommandLine(...) method is called next. This uses the globals table along with Boost Program Options to construct a set of command-line options. This makes it very easy to add new options to the command-line and do so in a standardized fashion (since Boost Program Options is based on Unix standards). For example, now "view_width" can be set to 1024 pixels via the "--view_width=1024" command.
But of course, if you don't like this approach or you require a non-standard argument on the command-line, there's no necessity to call parseCommandLine() - you can set up your own parser and completely ignore this method. It's there for convenience and is not a necessity.
Read-back
The parseCommandLine() checks for the command-line arguments, but what about using them?
Trivial:
lx0::lxvar options; options.insert("title", "Tutorial 03"); options.insert("width", spEngine->globals()["view_width"]); options.insert("height", spEngine->globals()["view_height"]); spView->show(options);
The table entries are lxvar data objects which now should either be storing the user provided value or the default.
We won't go into lxvar in too much detail other than to say it's used throughout the system in non-performance critical areas and is a convenient way to store variant data. Also note the flags and validation functions set up earlier ensured that the value stored are the right type and value; for example "view_width" is going to be an int between 32 and 4096 so there's no need to double-check those data values before passing them off to the View.
The Document
The command-line arguments are handled, but now what about the main application data which we'll be getting from disk rather than hard-coding this time?
Loading the Document
This starts with the change from the call to createDocument() in Tutorial 2 to a call to loadDocument() in Tutorial 3...
lx0::DocumentPtr spDocument = spEngine->loadDocument("media2/appdata/tutorial_03/document.xml");
The loadDocument() call now creates an actual, useful data object (or 'model' in MVC terms) that our 'view' can now read its data out of. The Document is an XML document composed of Elements as the individual nodes of the document. This is using the LxEngine DOM (Document Object Model) which is a simplification of a full XML/HTML DOM.
In LxEngine, the XML nodes are all Element objects: there is no Node class, no TextNode, or other specializations. The Element object contains one single value and a set of attributes. The value is usually a JSON-value as are the attributes (though attributes are often a 'simple' JSON value such a single string or number). These JSON values are stored in the code as lxvar objects.
There are a lot of details about the LxEngine DOM versus a proper XML DOM, but those don't really matter in the context of these early tutorials. Think of Document as an XML Document storing a tree of Elements each composed of a set of attributes and a single JSON value and references to their direct children.
- Document is a...
- Tree of Elements, each with...
- 0 or more named attributes, each with an lxvar value
- 1 lxvar value for the Element itself
- Tree of Elements, each with...
What's the Document look like?
Rather than continuing to speak abstractly about the Document, let's take a look at a snippet of the actual XML document being loaded:
<Document> <Library> <!-- Geomtrey --> <Geometry src="media2/models/unit_cube-000.blend" /> ... <!-- Materials --> ... <Material id="phong_checker"> { graph : { _type : "phong", diffuse : { _type : "checker", color0 : [ 0, 0, 0], color1 : [ 1, 1, 1], uv : { _type : "spherical", scale : [ 4, 4 ], }, }, }, } </Material> </Library> </Document>
So the XML document contains a <Library> Element which contains a set of <Geometry> and <Material> elements. The Geometry element is pretty straight-forward: it just has an attribute indicating the .blend file to use as the source for that geometry. The Material element is a little more interesting simply because it is using JSON as it's value to describe its data. The use of JSON for the value is common as there are many cases where XML document is aimed more towards conveniently storing structured data than plain text.
Renderer Initialization
Ok, so we've shown how to get a bunch of data into the app. Now it's time to do something with that data. As we'll see the Renderer::initialize() method has changed rather substantially.
Note the new code after the call to initialize() on the rasterizer:
mspRasterizer.reset( new lx0::RasterizerGL ); mspRasterizer->initialize(); // // Process the data in the document being viewed // _processConfiguration(); _processDocument( spView->document() ); lx_check_error( !mMaterials.empty() ); lx_check_error( !mGeometry.empty() );
What's happening here?
Two new important calls to protected methods we've added Renderer:
-
_processConfiguration()to go through the values on theEngine::globals()we set up earlier -
_processDocument()to go through that XML document we just loaded up
As you might guess from those checks for empty - the above two methods will be populating the Renderer with some materials and geometry
_processConfiguration()
void _processConfiguration (void) { // // Check the global configuration // lx0::lxvar config = lx0::Engine::acquire()->globals(); if (config.find("model_filename").is_defined()) { _addGeometry(config["model_filename"]); } if (config.find("shader_filename").is_defined()) { std::string source = lx0::string_from_file(config["shader_filename"]); lx0::lxvar parameters = config.find("params_filename").is_defined() ? lx0::lxvar_from_file( config["params_filename"] ) : lx0::lxvar::undefined(); _addMaterial("", source, parameters); } }
Let's take it for granted that _addGeometry() loads up the given model and _addMaterial() loads up a material (given a shader name and an optional list of parameters for the shader). With those assumptions, the above code should be pretty self-explanatory. Right? It's basically using lxvar's variant data to check if values exist and process them if they do.
We'll get to the implementation of _addGeometry() and _addMaterial() in a minute.
_processDocument()
Command-line options out of the way, it's time to process the Document that this View is supposed to be viewing. This means creating view-specific caches for all the geometry and materials in that document.
Let's dive in: find all elements of Material type and Geometry type and pass them on to the next worker functions (the getElementsByTagName() method should be familiar to HTML DOM users)...
void _processDocument (lx0::DocumentPtr spDocument) { // Find all the <Material> elements in the document and translate // them into runtime materials. // auto vMaterials = spDocument->getElementsByTagName("Material"); for (auto it = vMaterials.begin(); it != vMaterials.end(); ++it) _processMaterial(*it); // Do the same for <Geometry> elements // auto vGeometry = spDocument->getElementsByTagName("Geometry"); for (auto it = vGeometry.begin(); it != vGeometry.end(); ++it) _processGeometry(*it); }
_processGeometry() and BlendLoad
We'll start with Geometry because it's very simple. Read the attribute call "src", treat it as a string, and then call _addGeometry()...
void _processGeometry (lx0::ElementPtr spElem)
{
//
// Extract the data from the DOM
//
std::string sourceFilename = spElem->attr("src").as<std::string>();
_addGeometry(sourceFilename);
}And what does _addGeometry() do?
void _addGeometry (const std::string& modelFilename) { std::cout << "Loading '" << modelFilename << "'" << std::endl; lx0::GeometryPtr spModel = lx0::geometry_from_blendfile(mspRasterizer, modelFilename.c_str()); mGeometry.push_back(spModel); }
It uses the BlendLoad subsystem to do all the work. That subsystem takes a .blend file and a reference to a Rasterizer and sets up a GeometryPtr for you. It's basically doing a more generic version of the geometry creation you saw back in Tutorial 2. That's it. Nothing tricky. If you want to know more about BlendLoad, see it's documentation.
Note that we split _processGeometry() and _addGeometry() into two methods so that the if a model_filename is added on the command-line that same _addGeometry() method can be called.
_processMaterial() and ShaderBuilder
Not surprisingly, the code to add a new Material is not that different:
void _processMaterial (lx0::ElementPtr spElem)
{
std::string name = spElem->attr("id").as<std::string>();
lx0::lxvar graph = spElem->value().find("graph");
//
// Use the Shader Builder subsystem to construct a material
// (i.e. unique id, shader source code, and set of parameters)
//
lx0::ShaderBuilder::Material material;
mShaderBuilder.buildShader(material, graph);
_addMaterial(material.uniqueName, material.source, material.parameters);
}The interesting part of the above is use of the ShaderBuilder. We grab the Element's value (i.e. that chunk of JSON describing the material) and pass it off to the shader builder to create a ShaderBuilder::Material. This material has three parts:
- A unique name for the source code generated
- The GLSL source code itself that gets generated
- A set of name-value parameters to pass to the shader's uniforms to render the shader
A material in LxEngine (and most real-time systems for that matter) is composed of two parts: a shader and the set of parameters to that shader.
The ShaderBuilder therefore is smart about realizing when two materials require different shader code versus simply a different set of parameters. For example, if you define a black and white checker material and a red and blue check material, the ShaderBuilder will generate two different sets of parameters for the same shader source code.
The "unique name" output is basically an identifier which is guaranteed to be the same for two materials with the same shader source and different for any two materials for differ shader source; in other words, comparing the unique name of two materials is sufficient to see which shader they should use (rather than comparing the entire shader source text). This is all designed so the rasterizer can cache compiled shaders and reuse them rather than creating a shader for every single material (since shaders are fairly expensive objects).
LxEngine's RasterizerGL is, of course, designed to create materials via this very mechanism (note however there is no hard dependency between the ShaderBuilder and RasterizerGL - it's perfectly feasible to write an application that just uses the ShaderBuilder and virtually no other piece of LxEngine or to use RasterizerGL with your own shaders or shader generators):
void _addMaterial (std::string uniqueName, std::string source, lx0::lxvar parameters) { lx0::MaterialPtr spMaterial = mspRasterizer->createMaterial(uniqueName, source, parameters); mMaterials.push_back(spMaterial); }
We'll get into the ShaderBuilder's data definitions in a bit, but the actual creation process is pretty straightforward.
Finishing up Renderer::initialize()
Ok, so all the data is now loaded. On to the rest of Renderer::initialize():
// // Create a set of lights // lx0::LightPtr spLight0 = mspRasterizer->createLight(); spLight0->position = glgeom::point3f(10, -10, 10); spLight0->color = glgeom::color3f(1, 1, 1); lx0::LightPtr spLight1 = mspRasterizer->createLight(); spLight0->position = glgeom::point3f(10, 10, 10); spLight0->color = glgeom::color3f(.6f, .6f, 1); mspLightSet = mspRasterizer->createLightSet(); mspLightSet->mLights.push_back(spLight0); mspLightSet->mLights.push_back(spLight1); // // Build the cube renderable // mspItem.reset(new lx0::Item); mspItem->spTransform = mspRasterizer->createTransform(mRotation); mspItem->spMaterial = mMaterials[mCurrentMaterial]; mspItem->spGeometry = mGeometry[mCurrentGeometry]; // // Create the camera last since it is dependent on the bounds of the geometry // being viewed. Therefore, it needs to be created after the geometry is // loaded. // mspCamera = _createCamera(mGeometry[mCurrentGeometry]->mBBox);
We'll be a bit lazy here and just hard-code a light set since some of the materials require lights.
Then we have the familiar Item creation, but now we're reading out data out of the arrays of materials and geometry we created.
And lastly, we've moved the camera initialization so that we can set up the camera position based on the size of the model we just loaded. No new concepts in that code, just some use of the GLM and GLGeom libraries to do the math of computing the new position.
// // Creates a camera with fixed view direction and a view distance determined by // the visibility bounds. // lx0::CameraPtr _createCamera (const glgeom::abbox3f& bbox) { const glgeom::vector3f viewDirection(-1, 2, -1.5f); const float viewDistance (bbox.diagonal() * .9f); glgeom::point3f viewPoint = glgeom::point3f(0, 0, 0) - glgeom::normalize(viewDirection) * viewDistance; glm::mat4 viewMatrix = glm::lookAt(viewPoint.vec, glm::vec3(0, 0, 0), glm::vec3(0, 0, 1)); return mspRasterizer->createCamera(60.0f, 0.01f, 1000.0f, viewMatrix); }
The render() Method
No changes in the render() method other than passing in that light set we created:
virtual void render (void) { lx0::RenderAlgorithm algorithm; algorithm.mClearColor = glgeom::color4f(0.1f, 0.3f, 0.8f, 1.0f); lx0::GlobalPass pass; pass.spCamera = mspCamera; pass.spLightSet = mspLightSet; // <-- NEW algorithm.mPasses.push_back(pass); lx0::RenderList items; items.push_back(0, mspItem); mspRasterizer->beginFrame(algorithm); for (auto it = items.begin(); it != items.end(); ++it) { mspRasterizer->rasterizeList(algorithm, it->second.list); } mspRasterizer->endFrame(); }
User Interface
Let's fly through the trivial changes to add the new keyboard controls:
// // The UIBinding is not intended to do much processing itself. It should // map device-events into high-level application events. // class UIBindingImp : public lx0::UIBinding { public: virtual void onKeyDown (lx0::ViewPtr spView, int keyCode) { if (keyCode == lx0::KC_G) spView->sendEvent("change_geometry", "next"); if (keyCode == lx0::KC_F) spView->sendEvent("change_geometry", "prev"); if (keyCode == lx0::KC_M) spView->sendEvent("next_material"); if (keyCode == lx0::KC_N) spView->sendEvent("prev_material"); }
Add a new virtual method implemenation onKeyDown() and pass messages to the View.
The change of materials versus geometry is intentionally handled differently just to show two ways of accomplishing the same thing.
Then in the Renderer's handleEvent():
virtual void handleEvent (std::string evt, lx0::lxvar params) { if (evt == "change_geometry") { mCurrentGeometry = (params == "next") ? (mCurrentGeometry + 1) : (mCurrentGeometry + mGeometry.size() - 1); mCurrentGeometry %= mGeometry.size(); mspItem->spGeometry = mGeometry[mCurrentGeometry]; // // Recreate the camera after geometry changes since the camera position // is based on the bounds of the geometry being viewed. // mspCamera = _createCamera(mspItem->spGeometry->mBBox); } else if (evt == "next_material" || evt == "prev_material") { mCurrentMaterial = (evt == "next_material") ? (mCurrentMaterial + 1) : (mCurrentMaterial + mMaterials.size() - 1); mCurrentMaterial %= mMaterials.size(); mspItem->spMaterial = mMaterials[mCurrentMaterial]; } }
ShaderBuilder
We flew through the code and really glossed over how the ShaderBuilder materials are defined. The geometry was simple: a Blender file name is specified in the XML document and a utility function creates a GeometryPtr. But what about these materials?
It's a little beyond the scope of this tutorial to describe the entire ShaderBuilder interface. The ShaderBuilder documentation should be used for a full description. But for this tutorial, we'll provide a brief introduction.
JSON Shader Graphs
The materials are created via a simple shader graph (at the moment, it needs to be a DAG) specified in JSON code. The graph root is specified via a the "graph" element at the base of the JSON data. Then a shader graph node is specified.
The node itself is composed of a "_type" and a set of optional parameters, which depends on the type of node being used. All the parameters other than _type are optional as the node definition includes a default value for each parameter. In the example below, we use the solid color node to produce a very simple shader.
<Material id="solid_default"> { graph : { _type : "solid", }, } </Material> <Material id="solid_blue"> { graph : { _type : "solid", color : [ 0, 0, .7 ], }, } </Material>
The ShaderBuilder does have some convenience intelligence so that the "solid" node can actually be used where a vec3 or a vec4 output is required. But a solid colored shader isn't that interesting.
How about a Phong material with a black and white checker pattern for the diffuse channel, where the checker pattern is applied using a spherical map that scaled 4x?
<Material id="phong_checker"> { graph : { _type : "phong", diffuse : { _type : "checker", color0 : [ 0, 0, 0], color1 : [ 1, 1, 1], uv : { _type : "spherical", scale : [ 4, 4 ], }, }, }, } </Material>
Or how about a checker pattern where each tile of the checker actually has it's own Phong definition with different levels of specularity on each tile?
<Material id="phong_checker_complex"> { graph : { _type : "checker", color0 : { _type : "phong", diffuse : [ 1, 0, 0 ], specular : [ 1, 1, 0 ], }, color1 : { _type : "phong", diffuse : [ 1, 1, 1 ], specular : [ 0, 0, 0 ], }, uv : { _type : "spherical", scale : [ 4, 4 ], }, }, } </Material>
As long as the input and output types are aligned correctly, the nesting is limited only by what the runtime hardware can handle. Again, this is intended only to be a very brief intro. The ShaderBuilder documentation lists all the nodes and features of ShaderBuilder.
Conclusion
This tutorial introduced how to setup convenient command-line parameters for your program, how to load a document and process the XML elements in that document, and how to build custom materials using the ShaderBuilder subsystem.
At this point, the reader should have a fairly good idea of how to get data into their application and how render some basic geometry and materials that are a little more exciting than a simple spinning cube.
Appendix A: Feedback, Questions, & Issues
Please provide any feedback, questions, or issues on the forums.
LxEngine is in active development and in need of improvements. Suggestions for making it simpler to use are most welcome!
Appendix B: Future Improvements
These are some tentatively planned improvements for future revisions of this tutorial:
- Texture map examples
- Shadows
- Diffuse Gradient Lights
- Cel Shading
- Gooch Shading
- Wireframe Shading
- (more suggestions welcome)
