Thursday, 4 June 2015

Compiling HLSL effect files for OpenGL: Part 1

The images above were rendered in different API's - the left image is DirectX 11, the right OpenGL. They were rendered with the same code in C++ - a wrapper class interprets my high-level rendering API to DX or GL interchangeably; this is not so unusual.

What's interesting here is that as well as the same C++ code, these two images were rendered with the same shaders. And that those shaders are pretty much standard HLSL .fx files. For example, here's the file to render text:

//  Copyright (c) 2015 Simul Software Ltd. All rights reserved.
#include ""
#include "../../CrossPlatform/SL/"
#include "../../CrossPlatform/SL/"
#include "../../CrossPlatform/SL/"
#include "../../CrossPlatform/SL/"
uniform Texture2D fontTexture;

shader posTexVertexOutput FontVertexShader(idOnly IN)
 posTexVertexOutput OUT =VS_ScreenQuad(IN,rect);
 OUT.texCoords  =vec4(*OUT.texCoords.xy,0.0,1.0).xy;
 return OUT;

shader vec4 FontPixelShader(posTexVertexOutput IN) : SV_TARGET
 vec2 tc  =IN.texCoords;
 tc.y  =1.0-tc.y;
 vec4 lookup =fontTexture.SampleLevel(samplerStateNearest,tc,0);
 lookup.a =lookup.r;
 lookup  *=colour;
 return lookup;

VertexShader vs = CompileShader(vs_4_0, FontVertexShader());

technique text
    pass p0
 SetRasterizerState( RenderNoCull );
 SetDepthStencilState( DisableDepth, 0 );
 SetBlendState(AlphaBlend,vec4( 0.0, 0.0, 0.0, 0.0), 0xFFFFFFFF );

A few things to notice: we include "" at the top; that's different for each API. For GL for example, it defines float2 as vec2, float3 as vec3, and so on. For DX, it defines out "uniform" so that's ignored. There are many aspects to the shader language differences that can be papered over with #defines. But when we get to something like:


That's quite different syntax from the GLSL, which doesn't support separate texture and sampler objects.

We use "shader" to declare functions that will be used as shaders to distinguish them from utility functions. The DX11-style "CompileShader" commands create a compiled shader object for later use in technique definitions.

In techniques, we define passes, and in passes we can set render state: culling, depth test, blending - all that can be taken out of C++, leading to much cleaner-looking rendercode.

Quite aside from the cross-platform advantage, this is much easier to work with than standard GLSL, which would require a separate file for the vertex and fragment (pixel) shader, doesn't support #include, and has no concept of "Effect files" - which contain techniques, passes and state information: we would have to do all that setup in C++.

How's it done? I started off with glfx, an open-source project to allow Effect files for GLSL. Glfx passes-through most of the file, stopping to parse the parts that plain GLSL compilers don't understand. Glfx converts a source effect file into a set of individual shader texts (not files - it doesn't need to save them) - that can then be compiled with your local OpenGL.

 GLint effect=glfxGenEffect();
 glfxParseEffectFromFile(effect, "filename.glfx");

Having branched glfx to our own repo, it became apparent that it might actually be possible to adapt it to something that would, rather than adding Effect functionality to GLSL, simply understand DirectX 11-class HLSL fx files. To do this, rather than passing over the function/shader contents, it would be necessary to parse them completely. Glfx uses Flex and Bison, the GNU parser-generation tools. As GLSL and HLSL are C-based languages, I took the Bison-style Backus-Naur Form of C and added it to Glfx, so that


can be automatically rewritten, not as


but as:

.. which in turn requires changing the sampler and texture definitions: we do all this automagically. Essentially, we've built separate samplers and textures as a feature into GLSL where it didn't exist before - we can set samplers like so:

glfxSetEffectSamplerState(effect,"samplerStateNearest", (GLuint)sampler);

or just define them in the effect files or headers, just like in HLSL:

SamplerState samplerStateNearest :register(s11)
 AddressU = Clamp;
 AddressV = Clamp;
 AddressW = Clamp;

It's conceivable that what we now have should really not be called glfx any more - it's quite different to the original project and I'm mainly concerned with cross-API compatibility, rather than specifically adding functionality to GLSL. The new library supports Pixel, Vertex, Geometry and Compute shaders, constant buffers and structured buffers, technique groups, multiple render targets, passing structures as shader input/output - most of the nice things that HLSL provides.

But more than any of that, it means I only need to write my shaders once.

In Part 2, I'll delve into how the translation is implemented using Flex and Bison parser generators.