Compiling HLSL effect files for OpenGL: Part 1

The images above were rendered in different API's - the left image is DirectX 11, the right OpenGL. They were rendered with the same code in C++ - a wrapper class interprets my high-level rendering API to DX or GL interchangeably; this is not so unusual.

What's interesting here is that as well as the same C++ code, these two images were rendered with the same shaders. And that those shaders are pretty much standard HLSL .fx files. For example, here's the file to render text:

//  Copyright (c) 2015 Simul Software Ltd. All rights reserved.
#include "shader_platform.sl"
#include "../../CrossPlatform/SL/common.sl"
#include "../../CrossPlatform/SL/render_states.sl"
#include "../../CrossPlatform/SL/states.sl"
#include "../../CrossPlatform/SL/text_constants.sl"
uniform Texture2D fontTexture;

shader posTexVertexOutput FontVertexShader(idOnly IN)
{
 posTexVertexOutput OUT =VS_ScreenQuad(IN,rect);
 OUT.texCoords  =vec4(texc.xy+texc.zw*OUT.texCoords.xy,0.0,1.0).xy;
 return OUT;
}

shader vec4 FontPixelShader(posTexVertexOutput IN) : SV_TARGET
{
 vec2 tc  =IN.texCoords;

 tc.y  =1.0-tc.y;
 vec4 lookup =fontTexture.SampleLevel(samplerStateNearest,tc,0);
 lookup.a =lookup.r;
 lookup  *=colour;
 return lookup;
}

VertexShader vs = CompileShader(vs_4_0, FontVertexShader());

technique text
{
    pass p0
    {
 SetRasterizerState( RenderNoCull );
 SetDepthStencilState( DisableDepth, 0 );
 SetBlendState(AlphaBlend,vec4( 0.0, 0.0, 0.0, 0.0), 0xFFFFFFFF );
        SetGeometryShader(NULL);

 SetVertexShader(vs);
 SetPixelShader(CompileShader(ps_4_0,FontPixelShader()));
    }
}

A few things to notice: we include "shader_platform.sl" at the top; that's different for each API. For GL for example, it defines float2 as vec2, float3 as vec3, and so on. For DX, it defines out "uniform" so that's ignored. There are many aspects to the shader language differences that can be papered over with #defines. But when we get to something like:

           fontTexture.SampleLevel(samplerStateNearest,tc,0);

That's quite different syntax from the GLSL, which doesn't support separate texture and sampler objects.

We use "shader" to declare functions that will be used as shaders to distinguish them from utility functions. The DX11-style "CompileShader" commands create a compiled shader object for later use in technique definitions.

In techniques, we define passes, and in passes we can set render state: culling, depth test, blending - all that can be taken out of C++, leading to much cleaner-looking rendercode.

Quite aside from the cross-platform advantage, this is much easier to work with than standard GLSL, which would require a separate file for the vertex and fragment (pixel) shader, doesn't support #include, and has no concept of "Effect files" - which contain techniques, passes and state information: we would have to do all that setup in C++.

How's it done? I started off with glfx, an open-source project to allow Effect files for GLSL. Glfx passes-through most of the file, stopping to parse the parts that plain GLSL compilers don't understand. Glfx converts a source effect file into a set of individual shader texts (not files - it doesn't need to save them) - that can then be compiled with your local OpenGL.

 GLint effect=glfxGenEffect();
 glfxParseEffectFromFile(effect, "filename.glfx");

Having branched glfx to our own repo, it became apparent that it might actually be possible to adapt it to something that would, rather than adding Effect functionality to GLSL, simply understand DirectX 11-class HLSL fx files. To do this, rather than passing over the function/shader contents, it would be necessary to parse them completely. Glfx uses Flex and Bison, the GNU parser-generation tools. As GLSL and HLSL are C-based languages, I took the Bison-style Backus-Naur Form of C and added it to Glfx, so that

           fontTexture.SampleLevel(samplerStateNearest,tc,0);

can be automatically rewritten, not as

           textureLod(fontTexture,0);

but as:
textureLod(fontTexture_samplerStateNearest,0);

.. which in turn requires changing the sampler and texture definitions: we do all this automagically. Essentially, we've built separate samplers and textures as a feature into GLSL where it didn't exist before - we can set samplers like so:

glfxSetEffectSamplerState(effect,"samplerStateNearest", (GLuint)sampler);

or just define them in the effect files or headers, just like in HLSL:

SamplerState samplerStateNearest :register(s11)
{
 Filter = MIN_MAG_MIP_POINT;
 AddressU = Clamp;
 AddressV = Clamp;
 AddressW = Clamp;
};

It's conceivable that what we now have should really not be called glfx any more - it's quite different to the original project and I'm mainly concerned with cross-API compatibility, rather than specifically adding functionality to GLSL. The new library supports Pixel, Vertex, Geometry and Compute shaders, constant buffers and structured buffers, technique groups, multiple render targets, passing structures as shader input/output - most of the nice things that HLSL provides.

But more than any of that, it means I only need to write my shaders once.

In Part 2, I'll delve into how the translation is implemented using Flex and Bison parser generators.

Gfx.waitForPresent in Unity

Gfx.waitForPresent taking a lot of CPU time in Unity? That means the CPU is waiting for the GPU - so ignore the CPU times and profile the GPU.
http://forum.unity3d.com/threads/gfx-waitforpresent.211166/

GLSL constant buffers - alignment problems

Uniform buffers in GLSL - arrays of mats seem to destroy the alignment of the values afterwards.

layout(std140) uniform ConstantB
{
uniform mat4 views[6]; 

uniform float texCoord;  <-- this doesn't work.
uniform float gamma;
uniform float interp;
uniform float brightness;
}

So I'm working hard on make OpenGL work like DirectX 11, and come to the problem of depth.

GL expects you to use a projection matrix that transforms near z to -1.0, and far z to 1.0, which is a horrible use of numerical precision. Behind the scenes, GL will then map from [-1.0,1.0] to [0.0,1.0] to give you a nonzero depth in your buffer.

DirectX expects you to use a projection matrix that transforms near to 0 and far to 1. It doesn't need to map to [0.0,1.0] because you've already given it the range it's expecting. You can also have far as 1 and near as 0, which is actually much better for numerical precision reasons.

So if you use the same projection matrix, in the "same" vertex shader, in GL and DX, you'll get different depths. What comes out as z=0.0 in DirectX, will come out as z=0.5 in OpenGL.

glDepthRange to the rescue! This function controls the mapping from the values you actually send to GL to the values it puts in the depth buffer. If you leave it as the default, or pass glDepthRange(0.0,1.0), it will map [-1,1] to [0,1]. So if we want to leave the numbers untouched, we just pass glDepthRange(-1.0,1.0), right?

Wrong! Because as the GL spec states,

depth values are treated as though they range from 0 through 1 (like color components). Thus, the values accepted by glDepthRange are both clamped to this range before they are accepted.

Yes, another boneheaded OpenGL spec snafu means that however much you may want to map [0,1] to [0,1], OpenGL won't let you. If you pass glDepthRange(-1.0,1.0), GL will clamp the -1.0 back up to zero, and the mapping will still go [-1,1] -> [0,1].

The solution? For NVIDIA hardware, you can use their extension glDepthRangedNV, which is not nerfed like the standard function. For AMD.... ?

Font Awesome is awesome unless you want your icon anywhere but the left-hand side

Use "right", or "pull-right" to fix that.

<i class="fa fontawesome-icon right pull-right fa-times-circle size-medium medium circle-" style="color:#dd3333"></i>

Resource file access in Qt

Qt resources in qrc files have a Prefix and a Path (which is the file path relative to the qrc file).

To access a resource as if it were a file, you need to use both, with a colon at the start to indicate that it's a resource, not an object on disk:

QFile::exists(":/Prefix/rel_Path/file.ext")

ostringstream is the perfect C++ replacement for sprintf, except...

ostringstream is the perfect C++ replacement for sprintf, except for trying to debug its contents in Visual Studio.
Add this to Microsoft Visual Studio 12.0\Common7\Packages\Debugger\Visualizers\stl.natvis:

<Type Name="std::basic_ostringstream&lt;char,*&gt;">
<DisplayString>{_Mystrbuf->_Gfirst},s</DisplayString>
<StringView>_Mystrbuf->_Gfirst,s</StringView>
<Expand>
<Item Name="[state]">_Mystate</Item>
<ExpandedItem>_Mystrbuf->_Gfirst</ExpandedItem>
</Expand>
</Type>