Thursday 17 November 2016

Flex/Bison error lines compatible with Visual Studio

Flex and Bison can be built from source, to be found at github.com/AaronNGray/winflexbison. To get errors and warnings that you can double-click in Visual Studio, you can modify the code as follows:

In the file bison\src\location.c, find the function unsigned location_print (location loc, FILE *out) - and replace it with:

unsigned
location_print (location loc, FILE *out)
{
  unsigned res = 0;
  int end_col = 0 != loc.end.column ? loc.end.column - 1 : 0;
  res += fprintf (out, "%s",
                  quotearg_n_style (3, escape_quoting_style, loc.start.file));
  if(0 <= loc.start.line||loc.start.file != loc.end.file||0 <= loc.end.line)
      res += fprintf (out, "(");
  if (0 <= loc.start.line)
    {
      res += fprintf (out, "%d", loc.start.line);
      if (0 <= loc.start.column)
        res += fprintf (out, ",%d", loc.start.column);
    }
  if (loc.start.file != loc.end.file)
    {
  // Ignore: Visual Studio can't cope with this case.
    }
  if(0 <= loc.start.line||loc.start.file != loc.end.file||0 <= loc.end.line)
      res += fprintf (out, ")");
  return res;
}
That fixes it for Bison. For Flex, in flex/src/parse.c, change the function line_pinpoint( str, line ) to
void line_pinpoint( str, line )
const char *str;
int line;
 {
 fprintf( stderr, "%s(%d): %s\n", infilename, line, str );
 }

My branch incorporating these changes is here.

Tuesday 8 November 2016

Using Sublime Text with FASTBuild

Sublime text can be used to run FASTBuild, with this build script. Save this as "FASTBuild.sublime-build", in your user AppData\Roaming\Sublime Text 3\Packages\User directory (or Mac/Linux equivalent location).
{
 "cmd": ["C:/Simul/master/Simul/External/FASTBuild/FBuild.exe","-config","$file"]
 ,"file_regex": "(.*)\\((.*),(.*)\\): FASTBuild (Error .*)$"
 ,"selector": "source.bff"
}
By using the selector "source.bff", it should match up automatically with the syntax (below), but I've not quite figured this part out yet. Here's a preliminary Sublime Text syntax for FASTBuild. Call it "FASTBuild.sublime-syntax" and save it in the same directory.

%YAML 1.2
---
name: FASTBuild
file_extensions: [bff]
scope: source.bff

contexts:
  comments:
    - include: scope:source.c#comments

  instring:
    - match: "[^\"]"
      scope: string
    - match: \"
      pop: true
      scope: string
  strings:
    - match: \"
      push: instring
      scope: string

  inquotes:
    - match: "[^']"
      scope: string
    - match: "'"
      pop: true
      scope: string
  quotes:
    - match: "'"
      push: inquotes
      scope: string

  variables:
    - match: "\\.(\\w*)"
      scope: keyword
  preprocessor-includes:
    - match: "^\\s*(#\\s*\\binclude)\\b"
      captures:
        1: keyword.control.include.c++
  preprocessor-import:
    - match: "^\\s*(#)\\s*\\b(import)\\b"
      scope: keyword.control.c

  preprocessor:
    - include: scope:source.c#incomplete-inc
    - include: preprocessor-macro-define
    - include: scope:source.c#pragma-mark
    - include: preprocessor-includes
    - include: preprocessor-import
  global:
    - include: comments
    - include: preprocessor
    - include: strings
    - include: quotes
    - include: variables
  main:
    - include: global
    - match: \b(if|else|for|while)\b
      scope: keyword.control.c
      

Tuesday 9 August 2016

Qt oddness within Unity

I've noticed some very strange behaviour when launching a Qt-based UI from Unity recently. Possibly from updating Qt, but the UI would fail to respond to mouse clicks or other signals, until I dragged or resized the window, at which point, all the inputs I had made would then replay very quickly and get up to date.
The only solution I've found is to create a QTimer outside of the main window context (the QApplication is its owner), and set it to call QCoreApplication::processEvents every 100ms or so.

QTimer *timer=new QTimer(pApp);
timer->setSingleShot(false);
timer->start(100);
CONNECT_AUTO(timer,SIGNAL(timeout()),w,SLOT(Idle()));

Saturday 30 July 2016

Units for Physically Based Rendering

Physically-based rendering (PBR) should really use physical units, though many PBR engines don't.

 

Sunlight is typically described as being in watts per square metre. But that represents the energy across the entire spectrum, and has no concept of colour.

So for PBR, the units for directional light, e.g. sunlight are watts per square metre per nanometre.

Sunlight comes from so far away that the direction of the light is essentially parallel. But a local light source like a light bulb emits its energy in all directions.

So the irradiance due to a light bulb varies with distance from the bulb. If the total spectral power of the bulb, P (the spectral flux), is measured in watts per nanometre (for any given colour on the spectrum), suppose that the bulb has a radius of r metres, and thus a surface area of $4 \pi r^2$ square metres. Then the irradiance, at the surface, will be $I=P/ (4 \pi r^2) w/m^2/nm$, in the same units as sunlight.

But away from the bulb, at distance R, the same power passes through a larger surface area. So again, $I = P / (4 \pi R^2)$, where P is the same total spectral power as before.

Thus $I(R) = I(r) (\frac{r}{R})^2$ - the irradiance follows an inverse-square power law.

So we don't use irradiance to measure point lights. If the light is uniformly distributed by direction, we can use spectral flux P, watts/nm.

the radiance at any point is given in watts per steradian per nm, where there are 4pi steradians across the entire sphere.

Rendering

The challenge with rendering is to recreate (on a lcd monitor, cinema screen, or in print) the radiance that the you would perceive if you were really looking at the thing the image represents.
Imagine you want to create the experience of flying at ten thousand feet. It looks something like this:
photo at 10,000 feet altitude
trueSKY render at 10,000 feet altitude

But in rendering we don't (necessarily) want to recreate what the photo or video of something would look like: that's actually a more complex problem. The fundamental challenge of rendering is to see if we can recreate the real thing. Consider one point in the image, perhaps part of the sky. The light from that point that the eye would perceive is defined by a spectrum, like this:
and the way the human eye would perceive it is defined by the response of its three types of cone (ignoring rods for now - those are for night-vision).

The three cone types correspond only roughly to red green and blue, and they overlap considerably. So they are called X, Y and Z. These functions, mapped by the CIE, don't describe a precise physiological response in the eye - but they do allow us to match perceived colours from different sources as the eye sees them. Having these functions, we can calculate three responses:

$X= \int_{\lambda} f_x(\lambda) E(\lambda) d\lambda $
$Y= \int f_y E$
$Z= \int f_z E$

X, Y and Z are the three numbers we want to reproduce.
Now monitors have red, green, and blue elements to each pixel, and the wavelengths of these are pretty sharply concentrated. They're not single-frequency spikes like lasers, but they're clearly separated.
So in the "true" image we had one continuous spectrum, but the monitor gives out three distinct colours. You can calibrate your monitor to get its exact curves (see e.g. here) although the calibration will only be valid until you adjust the monitor's settings.

We would like to reproduce the same three X, Y, and Z values as above: that will make the colour and brightness of the point/pixel look the same as reality to the eye.

Of course, the eye response curves are meant to represent the typical human eye. If your eyes don't match the standard curves - for example, if you're colour blind - that assumption is invalid, and the two radiances won't look the same to you.

$X= \int f_x E_m$
$Y= \int f_y E_m$
$Z= \int f_z E_m$

where $E_m$ is the monitor spectral radiance, which is a combination of what the red, green, and blue elements are putting out:

$E_m(\lambda)=R_m(\lambda)+G_m(\lambda)+B_m(\lambda)$

At any given $\lambda$, we expect at most one of those values to be significant.

So for a known spectral radiance distribution, we solve for Rm, Gm and Bm:

$\int f_x E = \int f_x (R_m+G_m+B_m)$
$\int f_y E = \int f_y (R_m+G_m+B_m)$
$\int f_z E = \int f_z (R_m+G_m+B_m)$

Note we can't simply say

$\int f_x E = \int f_x B_m$

etc. X, Y and Z are not exactly blue, green and red. More like blue, yellowy-green, and greeny-yellow-violet. Our eyes know how to interpret the infinite combinations of cone responses into colour and brightness.

Let's assume that the spectral profile of our monitor is known, and that generally:

$R_m(\lambda)=R \times m_R(\lambda)$

where R is the brightness, from zero to one, of the red part of the pixel, and $m_R$ is a known function for the monitor.

\begin{align}
\int f_x E &= \int f_x (R m_R+G m_G+B m_B) \\
 &=R \int f_x m_R + G \int f_x m_G + B \int f_x m_B \\
 \end{align}

So knowing the eye functions $f_x$ etc., and the monitor functions $m_R$ etc, the right-hand-side integrations can be precalculated, leaving us with:

\begin{align}
\int f_x E &= R I_xR + G I_xG + B I_xB \\
\int f_y E &= R I_yR + G I_yG + B I_yB \\
\int f_z E &= R I_zR + G I_zG + B I_zB \\
 \end{align}

This is a 3x3 matrix equation: linear algebra.

\begin{align}
c &= M c_m
\end{align}

where c is the vector of three "ground truth" integrals, $c_m$ is the vector of three monitor rgb values (assuming a linear monitor - more on this later), and $M$ is the monitor-eye matrix, which is constant for a given monitor and viewer.

While X is kind-of blue, and Z is kind-of red, we can't really regard any of the integral constants that make up M as being close enough to zero to be negligible, except for maybe $I_xR$. We'll leave it in for now. We must get the matrix inverse of M to solve for $c_m$. If we knew the shape of $E(\lambda)$ through the whole spectrum, and assuming we have all the monitor data, and assuming we have a viewer with typical human eyes, we'd be able to calculate the exact $c_m$, the exact RGB values to send to the monitor that will reproduce the ground truth view. We would have to hope, as well, that when we've found $c_m$, none of its members are greater than 1.0. Because monitors have low dynamic range, for now, we can't represent many of the brightness values that in real life we encounter every day.

Much of the above must be taken on trust, or worked around. But what do we know about c? We probably haven't calculated the entire spectral radiance curve for the visual spectrum, for each pixel onscreen. We've probably calculated three values. Again, red, green, and blue. And we must make an assumption about how those three numbers approximate the full spectrum.

Suppose we assume that each of our three calculated values represents a range of the spectrum over which the spectral radiance is constant:

We can refine this later with a better shape. But our three columns roughly approximate the full spectrum, and they allow us to calculate $c$ as follows:


\begin{align}
c_x &= \int f_x E_d \\
c_y &= \int f_y E_d \\
c_z &= \int f_z E_d \\
 \end{align}

where $E_d$ is our rendered sr curve, which is:

\begin{align}
E_d &= R_d (r_0 \le \lambda < r_1) \\
&= G_d (g_0 \le \lambda < g_1) \\
&= B_d (b_0 \le \lambda < b_1) \\
 \end{align}

where $R_d$ etc are the rendered spectral radiances. So we can now calculate $c$, or at least $c_d$, the rendered approximation to the ground truth. And finally:

$c_m = M^{-1} $c_d


Tuesday 12 July 2016

Qt's QProcessEnvironment cannot be used to change the current environment.

http://doc.qt.io/qt-4.8/qprocessenvironment.html

Sadly, and obscurely, Qt's QProcessEnvironment can't change the running environment. If you call QProcessEnvironment::systemEnvironment(), that returns the current environment by value. Any changes you make to the returned object are discarded. You must instead use _putenv() to change environment variables.

Saturday 18 June 2016

Cubemap Texture Arrays

So in modern graphics API's: Direct3D 11 and 12, OpenGL 4.0 and so on, we can create 2D textures, textures with multiple mipmaps, arrays of textures with multiple mipmaps, and so on. Most don't yet support arrays of 3D textures.

But one little-used combination that is supported, is a texture cube array. That's a single texture, which is an array of cubemaps, which optionally have mip levels as well.

In Direct3D 11, this involves creating a 2D texture using D3D11_TEXTURE2D_DESC struct, where arraySize is six times the number of cubemaps.

Then, when you create the shaderResourceView, you'll use a D3D11_SHADER_RESOURCE_VIEW_DESC with ViewDimension equal to D3D11_SRV_DIMENSION_TEXTURECUBEARRAY.

You'll fill in the TextureCubeArray member of that struct's union, e.g.
 SRVDesc.ViewDimension    =D3D11_SRV_DIMENSION_TEXTURECUBEARRAY;
 SRVDesc.TextureCubeArray.MipLevels  =numMips;
 SRVDesc.TextureCubeArray.MostDetailedMip =0;
 SRVDesc.TextureCubeArray.First2DArrayFace =0;
 SRVDesc.TextureCubeArray.NumCubes  =numLevels;

Thursday 5 May 2016

Versioning SDK's with Git for builds with Jenkins

Alright I think I've finally figured out a way to build SDK's with versioning.

Git for source control, obviously.

Development happens on the "master" branch. There's a "dev" branch for experimental stuff.
About every other month, I'll create a numbered branch. I've started with 4.0 because it continues from the old versioning system.

Each numbered branch receives only bug-fixes. All feature changes go in the master branch.

Jenkins gets a list of versions to build, defined in the main Jenkins Config as an environment variable. At the moment is looks like this:


Using Jenkins' Dynamic Axis Plugin and Matrix Projects, I use this as an axis for the matrix builds. Over time, new project numbers will be added and older ones will drop off. Jenkins will add build numbers to the version numbers, so I'll end up with installers numbered 4.0.125 etc.

Because only bug-fixes go in numbered branches, higher build numbers will always be more stable. And once you've chosen a numbered branch for your project, you can stick with that number, report to us any bugs, and be confident that the fixes will go in without any new breaking changes.

And I can develop freely on the master branch without worrying about breaking the versions in use by customers.

This should work. I'm probably the last person to figure this out.

Tuesday 26 April 2016

Using curl in Windows to download files

This downloads a file if it's newer than the one that's on your local drive:

curl -z "path\filename.lib" ftp://ftp.yoursite.com//ftppath/filename.lib -o "path\filename.lib"

The "-z" means only copy if it's newer than a specified date. But then you just put the filename, meaning that curl should use the date of that file.
If the file doesn't exist locally, you'll get a warning about "invalid date or not a file". This can be ignored.
The "-o" means download it to the local file specified (same filename as for -z), instead of just outputting it to the console.

Monday 29 February 2016

Problem with handling HTML form checkboxes in php

php doesn't handle checkboxes properly in html forms. If you use

My Checkbox <input type="checkbox" name="my_checkbox" id="my_checkbox"'.($current_value?'checked':'').' />

You'll only get a result for isset($_REQUEST['my_checkbox']) if the box is *checked*, not if it's cleared. So you can't test for whether the box was unchecked by the user, because using !isset would be the same whether the page was just loaded, or if the form was submitted with the box unchecked by user input.

The solution from Stack Overflow:

Every checkbox generated is associated with a hidden field of the same name, placed just before the checkbox, and with a value of "0".

<input name="my_checkbox" type="hidden" value="0" />
<input name="my_checkbox" type="checkbox" value="1" />


Then isset($_REQUEST['my_checkbox']) always returns true if the box was modified, and false if the page was just loaded. And you'll always get the correct '0' or '1' value in $_REQUEST['my_checkbox'].

Thursday 25 February 2016

How to create a plugin for Lumberyard

Lumberyard/CryEngine uses waf, a build configuration system similar to CMake.

To add a plugin project, create a subdirectory NewPlugin in dev/Code (a plugin might go in dev/Code/Sandbox/Plugins for example).

Add a wscript file to this directory, looking like this:

 
def build(bld): 
 
 bld.CryPlugin(
  target = 'NewPlugin',
   vs_filter = 'Sandbox/Plugins',
        file_list   = 'newplugin.waf_files',
  features =  ['qt'],
  includes = [
            '.',
            '..'
        ],
  use='EditorCommon'
 )
And add a file newplugin.waf_files, looking like:
{
 "NoUberFile": 
 {
  "Root":
  [
   "NewPlugin.cpp",
   "NewPlugin.h"
  ]
 }
}
Then go to dev/_WAF_/specs, and edit all.json. Add "NewPlugin" to the win_profile_modules and win_debug_modules list. Finally, recreate the solution by opening a command prompt in dev/ and running "lmbr_waf configure"

Sunday 14 February 2016

The Three Hierarchies of C++

Lest anyone ever tell you programming isn't complicated, it occurred to me that a C++ program has three interlocking, but independent hierarchies.

  • Namespace: The name of the function, within its parent namespace and/or the class that owns it.
  • Call Stack: The parent is the calling function, the child is the callee.
  • Inheritance: The class/struct is a child of the base class.

Could we stand to lose one or two of these?

Friday 12 February 2016

Reverting all deletes in Git, leaving file modifications in place

I asked how to do this on StackOverflow, and the answer is:

git diff --diff-filter=D --name-only | xargs git checkout

You need the "xargs": just piping the output of "diff" to "checkout" doesn't work.

Sunday 24 January 2016

Minimal Hello World for FASTBuild

FASTBuild is an open source build program with amazing potential. It apparently allows distributed compilation out-of-the-box, which is something Incredibuild charge a lot of money for. But the documentation is minimal so far, and it lacks a functional example. So I put together a "Hello World" for FASTBuild. The HelloWorld.cpp is:

#include <iostream>

int main(int , char * [])
{
 std::cout << "hello world!\n";
 return 0;
}

And the project config file is build.bff:

;-------------------------------------------------------------------------------
; Windows Platform
;-------------------------------------------------------------------------------
.VSBasePath  = 'C:/Program Files (x86)/Microsoft Visual Studio 12.0'
.WindowsSDKBasePath = 'C:/Program Files (x86)/Windows Kits/8.1'
.ClangBasePath  = 'C:/Program Files/LLVM'
.WindowsLibPaths = '$WindowsSDKBasePath$/lib/winv6.3/um'


.BaseIncludePaths  = ' /I"./"'
    + ' /I"$VSBasePath$/VC/include/"'
    + ' /I"$WindowsSDKBasePath$/include/um"'
    + ' /I"$WindowsSDKBasePath$/include/shared"'

Compiler( 'Compiler-x86' )
{
 .Root  = '$VSBasePath$/VC/bin'
 .Executable = '$Root$/cl.exe'
}
; MSVC Toolchain
;---------------
.ToolsBasePath  = '$VSBasePath$/VC/bin'
.Compiler  = 'Compiler-x86'
.Librarian  = '$ToolsBasePath$/lib.exe'
.Linker   = '$ToolsBasePath$/link.exe'
.CompilerOptions = '%1 /Fo%2 /c /Z7'
.CompilerOptions + .BaseIncludePaths
.LibrarianOptions = '/NODEFAULTLIB /OUT:%2 %1'
.LinkerOptions  = ' /OUT:%2 %1 /SUBSYSTEM:CONSOLE'
.LinkerOptions  +  ' /LIBPATH:"$WindowsLibPaths$/x86" /LIBPATH:"$VSBasePath$/VC/lib"'

; Libraries
;----------
Library( 'exelib' )
{
    .CompilerInputPath = '.\\'
    .CompilerOutputPath= 'Out\'
    .LibrarianOutput   = 'Out\exelib.lib'
}
; Executables
;------------
Executable( 'myExe' )
{
 .CompilerInputPath = '.\'
 .CompilerInputPattern    ='*.cpp'
 .CompilerOutputPath= 'Out/'
 .Libraries    = { 'exelib' }
 .LinkerOutput = 'HelloWorld.exe'
 .LinkerOptions + ' /SUBSYSTEM:CONSOLE'
   + ' LIBCMT.LIB'
}

; 'all'
;-------
Alias( 'all' )
{
   .Targets = { 'myExe' }
}


To build it, go to the directory in a command prompt and run FBuild.exe.
This is for Visual Studio 2013 (12.0) - for different versions just change the VSBasePath variable.

One really interesting thing is that you can't seem to compile source files into Executables directly: you have to specify a library (e.g. "exelib" above), and use .CompilerInputPath to specify a directory. It will scan the path for all files of interest (.cpp by default) - you can specify them individually if you want. But the Executable command doesn't compile cpp's and if you don't include a library, it will complain of having no Object files to link.

Update 27/1/2016

FASTBuild author Franta Fulin has posted an official Hello World example: you can find it here.

Tuesday 19 January 2016

$(VCTargetsPath) in MSBuild

MSBuild may fail to build, complaining that
"C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V110\Microsoft.Cpp.Default.props" was not found
i.e. trying to load the v110 toolset even though you may not have the corresponding Visual Studio version. To fix it, set the environment variable VisualStudioVersion=12.0 for example, or pass it to msbuild on the command line as /p:VisualStudioVersion=12.0.

Friday 8 January 2016

Visual Studio-style mouse controls for Sublime Text

Create C:\Users\(YOURNAME)\AppData\Roaming\Sublime Text 3\Packages\User\Default (Windows).sublime-mousemap, and fill it with:

[
  // Ctrl-click word select
    {
        "button": "button1", "count": 1, "modifiers": ["ctrl"],
        "press_command": "drag_select",
        "press_args": {"by": "words"}
    }
  // Column select
  ,{
        "button": "button1", 
        "count": 1, 
        "modifiers": ["alt"],
        "press_command": "drag_select",
        "press_args": {"by": "columns"}
  }
]

Tuesday 5 January 2016

If your Windows build server is failing to find executables...

... your PATH environment variable is probably too long. Cut it shorter than 2048 characters and things should start working again.