If your Windows build server is failing to find executables...

... your PATH environment variable is probably too long. Cut it shorter than 2048 characters and things should start working again.

Piping Doxygen output to Github Pages for automatic navigation

Github Pages is a wonderful system for building a website. It works using Git. You edit your files locally, push them to Github, and the site automatically updates. docs.simul.co uses it for Simul's documentation. One great thing about gh-pages is that because you edit it locally, you can use something like Sublime Text to edit the files, which still, in 2015, is so much more responsive than editing online with all the latency that entails.

Gh-pages uses a system called Jekyll, which is also great. While the site can take straight-up html files and reproduce them on the site, it also understands Markdown syntax. While html is a text format, it's hard to write - you often want to use a specialized editor - but these end up filling your file with unnecessary cruft - repeated format tags and so on. Markdown is much simpler - it's plain text with a few conventions, for instance:

Heading
--------

when compiled to html by Jekyll, becomes:

Heading

So if you put a file in your repo called test.md, that becomes a page on your website called test.html. It will only be used if it has Jekyll Front Matter at the top, like so:

---
title: Introduction
layout: reference
url: Introduction
---

Jekyll will format it, and apply a theme if you have one. The sites it generates are generally good-looking.

So that's all great

but most of my interesting stuff comes from the Simul source code - I use Doxygen to generate documentation from source. Based on formatted comments, Doxygen creates html (and various other formats if desired) docs.

So what I wanted was to pipe the doxygen output into my Jekyll site, but keep the Jekyll formatting, and generate a consistent nav bar that looked the same whether you were in the Doxygen-created automatic docs, or the static markdown-based site.

Now Doxygen can't generate markdown. It can use markdown as an input - and create html from it just as Jekyll does. Fortunately, Jekyll doesn't insist on having markdown as an input. If you put a plain html file in a Jekyll site, it will be ignored, but if you add some Jekyll front matter to the top of your html as plain text, it will be included in the Jekyll site. The front matter makes it a wrong html file, so it will only work properly within Jekyll.

So the first step to get the Doxygen content into the Pages site was to create a special jekyll_header.html that Doxygen would use, which has the front matter at the top, like so:

---
title: $title
layout: reference
---

<!-- HTML header for doxygen 1.8.9.1-->

... more typical doxygen stuff here...

Doxygen replaces $title with the page title. We specify "reference" as the layout so that we can create a special Jekyll layout if needed for the docs.

So now Doxygen will build html files which, if copied into the Pages repository, will appear in the website with Jekyll styling.

The whole point of this is automation. We use Jenkins, with a project set up to run doxygen on the source code. It's arranged to that the Doxygen configuration file has these settings, which read environment variables:

HTML_OUTPUT            =$(HTML_OUTPUT)
HTML_HEADER            =$(HTML_HEADER)
HTML_FOOTER            =$(HTML_FOOTER)

Then we call doxygen twice with output to two different folders - once to generate the standard compiled html (chm) help file, and once with the jekyll header to prepare the docs for Pages.

A separate Jenkins project then copies the output into the "reference" directory in the Pages repo. Before this, we delete the entire contents of this directory, because any files that were not regenerated by Doxygen should be removed from the folder - otherwise they'll still show up on the site. Then we git commit, git push, and the site should update.

This much will get the generated docs into Jekyll, and the Jekyll styling will be applied, if you've created the "reference" layout (any good Jekyll tutorial can help you with this).

Doxygen is probably the best docs-generator of its kind, but there's a lack of control. The html generated can have a navigation tree on the side, but the generated tree uses frames, a relic of the 90's which doesn't play well with modern site design, much less with the mobile web. Jekyll on the other hand, can render very nicely on mobile, and with a bit of work, can display a nice modern html navigation bar.

So the next step was to try to fix up the left-hand-side navigation tree for the Doxygen content.

Our existing method relies on the directory structure of the website: each folder is a level of the tree, with the filename being the bottom level. That doesn't work for Doxygen content because Doxygen doesn't generate a directory tree structure, but a single flat directory, full of files with names like "classsimul_1_1base_1_1EnvironmentVariables.html".

In the Jekyll _includes directory, create an html file called something like "ref_sidebar.html", and fill it with:

<div class="sidebar">
  <div class="container sidebar-sticky">
    <div class="sidebar-about">
      <h1>
        <a href="{{ site.baseurl }}">
          {{ site.title }}
        </a>
      </h1>
      <p class="lead">{{ site.description }}</p>
    </div>
    <nav class="sidebar-nav">
    </nav>
  </div>
</div>

Now inside the "sidebar-nav", we will put the Jekyll/liquid code.
{% assign sorted_pages = site.pages | sort: "url" %}

This is the first step - we create a sorted list of all the url's in the site. It's pretty fast. I don't know if it's possible to create an array of structs in Liquid, so we create some blank arrays that we'll fill up with properties of the various pages.

{% assign this_path = "" %}

{% assign pages = site.array %}

{% assign paths = site.array %}

{% assign titles = site.array %}

{% assign path_sizes = site.array %}

Then we're going to go through all the pages (again, not slow), and from the url's we will create a "path" for each page with "/" as a separator. For example, the page
{% for node in sorted_pages %} {% assign path = node.url | remove: "/index.html" %}
So the path is formed initially from the url, knocking off the trailing "/index.html" that Jekyll may have generated. This is so that the main page of any tree branch will look like the root of the branch. Then we discard some pages that we don't want:

{% if path contains "/dir_" or path contains "/struct" or path contains "/functions" or path contains "/namespacemembers" or path contains "graph_legend" or path contains "google" %} {% continue %} {% endif %}
If a page refers to a source file, doxygen gives it a filename ending in _source.html, for example, "ChunkInputOutput_8h_source.html" would be the page generated for the source file "ChunkInputOutput.h" . In this case, if it's in the "reference" folder, we make it a child of "/source/", so all the source pages get grouped together.

{% if path contains "_source.html" %} {% assign path = path | replace: "reference/","reference/source/" %} {% endif %}

Similarly, the page generated for the class simul::base::BaseProfilingInterface would be called "classsimul_1_1base_1_1BaseProfilingInterface.html". We want this to be listed under the page "classes.html", which is doxygen's class list. But we also want it to be in a tree structure, so its full path should read "reference/classes/simul/base/BaseProfilingInterface". The following code achieves this. It replaces "_1_1" (which represents "::") with "/". It replaces "reference/class" with "reference/classes", so that the class definitions will be children of "classes". But that would turn the actual "classes.html" into "classeses.html". So we fix that too. We remove any ".html" extension so that tree parents are properly recognized. Various other functions are performed: "_8" goes back to ".".

Note also that we replace "_ooo" with "/". I've chosen "_ooo" as a separator in doxygen source filenames (not source code, just text documents that doxygen should interpret). This is so we can get proper parent-child relationships in such documents.

{% assign path = path | replace: "reference/class","reference/classes/" | replace: "reference/classeses","reference/classes" | remove: ".html" | replace: "_1_1","/" | replace: "_01"," " | replace: "_8","." | remove_first: "/" | replace: "_ooo","/" %}

Now we'll store some useful information: how many parts (separated by "/") are in the path?

{% assign psz = path | size %} {% assign path_parts = path | split: '/' %} {% assign path_parts_size = path_parts | size %}

And we're ready to push our path onto the list.

{% assign paths = paths | push: path %} {% assign pages = pages | push: node %} {% assign path_sizes = path_sizes | push: path_parts_size %} We fix up the title to look friendlier. We're going to use this in the nav tree.

{% assign title = node.title | remove: " Source File" | remove: " Class Reference" %} {% assign titles = titles | push: title %}

And finally we check if this is the current page.

{% if node.url == page.url %} {% assign this_path = path %} {% assign this_node = node %} {% assign this_index = forloop.index0 %} {% endif %} {% endfor %}

Having prepared

all that, we can now construct the nav tree using nested <ul> and <li> tags:
<ul>

{% assign menu_level = 0 %}

We iterate through the list "pages" that we've just created. The forloop index corresponds for "pages", "paths", "titles" and so on. Remember to use forloop.index0. Just "index" would start from 1 and give you the wrong page.

{% for node in pages %} {% assign node_path = paths[forloop.index0] %} {% assign node_title = titles[forloop.index0] %} {% assign node_path_parts = node_path | split: '/' %} {% assign node_path_parts_size = path_sizes[forloop.index0] %} {% if node_title == "404" %} {% continue %} {% endif %}

At the first level of the nav tree, we only want single-part paths:

{% unless node_path_parts_size == 1 %} {% continue %} {% endunless %} <li> <a class="sidebar-nav-item{% if page.url == node.url %} active{% endif %} sidebar-1" href="{{ node.url }}">{{ node_title }}</a></li>

NOW,

if the path of the page you're viewing contains the path of the page in the list, we'll expand its children!

{% if this_path contains node_path %}

We'll build new lists containing only the children of this parent:

{% assign pages2 = site.array %} {% assign titles2 = site.array %} {% for node2 in pages %}

and only two-level paths at this stage:

{% unless path_sizes[forloop.index0] == 2 %} {% continue %} {% endunless %} {% assign path2 = paths[forloop.index0] %} {% if path2 contains node_path and node2.url != node.url %} {% assign pages2 = pages2 | push: node2 %} {% assign titles2 = titles2 | push: titles[forloop.index0] %} {% endif %} {% endfor %}

{% if pages2.size > 0 %} {% assign menu_level = 1 %} <ul> {% for node2 in pages2 %} <li><a class="sidebar-nav-item{% if page.url == node2.url %} active{% endif %} sidebar-2" href='{{node2.url}}'>{{ titles2[forloop.index0] }}</a></li> {% endfor %}

Now this is the point where you might want to add more levels to the tree. Otherwise, close out the <ul> and the rest is just tidy-up.

</ul> {% endif%}

{% endif %}

{% endfor %} <<!-- node in pages -->

{% for num in (1...menu_level) %} </ul> {% endfor %} </ul>

Generating mipmaps for 3D textures

Mip-mapping is very useful, and sometimes we want to generate the mips of a texture in a shader. In DirectX 11 hlsl this is tricky because you can't read from the resource that you're writing to.

For example, you might have a 3D texture that you've filled in with a compute shader. You then want to generate its mipmaps from the original texture. But the ShaderResourceView you created can't be read while you're writing to. It will silently fail - unless you have Direct3D debug output enabled, in which case you'll get a warning.

You probably created the SRV like this:

  D3D11_SHADER_RESOURCE_VIEW_DESC srv_desc;
  ZeroMemory(&srv_desc, sizeof(D3D11_SHADER_RESOURCE_VIEW_DESC));
  srv_desc.Format                     = f;
  srv_desc.ViewDimension              = D3D11_SRV_DIMENSION_TEXTURE3D;
  srv_desc.Texture3D.MipLevels        = m;
  srv_desc.Texture3D.MostDetailedMip  = 0;
  d3D11Device->CreateShaderResourceView(texture, &srv_desc,&shaderResourceView);

which means that when you pass it to the hlsl shader, it contains all of the mips from 0 to m-1. What you want instead (or rather, as well), is a SRV for each individual mip:

D3D11_SHADER_RESOURCE_VIEW_DESC mip_srv_desc;
ZeroMemory(&mip_srv_desc, sizeof(D3D11_SHADER_RESOURCE_VIEW_DESC));
mip_srv_desc.Format                    = f;
mip_srv_desc.ViewDimension             = D3D11_SRV_DIMENSION_TEXTURE3D;
mip_srv_desc.Texture3D.MipLevels       = 1;
mip_srv_desc.Texture3D.MostDetailedMip = i;
d3D11Device->CreateShaderResourceView(texture, &mip_srv_desc, &mipShaderResourceViews[i]);

Then use these as input to the compute shader.

Unity fails to open a project, spuriously reports that it is open in another instance

Unity refuses to open the same project in two instances - this much is sensible. But when Unity crashes, sometimes it leaves behind a "Temp" folder in the project, and if this exists, you won't be able to reopen the project. So delete "Temp", and problem solved.

Automatically Generated Nav Bar for Jekyll and Github pages

Github pages, which I use for the Simul documentation, uses Jekyll, which is great for building static websites and blogs.

I wanted to create an automatic two-level nav bar for my site, and after much searching, I found this approach, using the url's of the pages. The implementation didn't quite work for me, and I wanted multi-level navigation, so this is what I came up with:

      
      {% assign url_parts = page.url | split: '/' %}
      {% assign url_parts_size = url_parts | size %}
      {% if url_parts_size != 0 %}
        {% assign rm = url_parts | last %}
        {% assign base_url = page.url | replace: rm %}
      {% else %}
        {% assign base_url = page.url %}
      {% endif %}
      <ul>
      {% for node in site.pages %}
        {% assign node_url_parts = node.url | split: '/' %}
        {% assign node_url_parts_size = node_url_parts | size %}
        {% assign filename = node_url_parts | last %}
       
        {% if node_url_parts_size == 2  %}
            <li>
            <a class="sidebar-nav-item{% if page.url == node.url %} active{% endif %}" href="{{ node.url }}">{{ node.title }}</a></li>
            {% if page.url contains node.url %}
              <ul>
              {% for node2 in site.pages %}
                {% if node2.url contains node.url %}
                  <li><a href='{{node2.url}}'>{{node2.title}}</a></li>
                {% endif %}
              {% endfor %}
              </ul>
            {% endif %}
        {% endif %}
      {% endfor %}
      {% for num in (1...menu_level) %}
        </ul>
      {% endfor %}
      </ul>
      

I think this works. It just looks at all the site's pages, and prints up links for those with two parts to the URL: these are the first sub-level. If that URL is part of the URL of the current page, it does another loop, to find the current page's siblings.

UPDATE:

You could add further nav levels by putting extra inner for loops. Here's a more complete example from my site, that goes to three levels. (See also http://dbgdiary.blogspot.com/2015/11/piping-doxygen-output-to-github-pages.html Done)

{% assign sorted_pages = site.pages | sort: "url" %}

    {% assign this_path = "" %}

    {% assign pages = site.array %}
    {% assign paths = site.array %}
    {% assign titles = site.array %}
    {% assign path_sizes = site.array %}
    {% assign this_index = 100000000 %}

We've initialized some empty arrays here using site.array, which is defined in _config.yml at the site root as:

array: []

Next we build the arrays from the page list, tweaking Doxygen's filenames to produce a tree-like structure (doxygen's file structure is flat, but we want a tree of URI's):

    {% for node in sorted_pages %}
        {% assign path = node.url | remove: "/index.html" %}
        {% if path contains "/dir_" or path contains "/struct" or path contains "/functions" or path contains "/namespacemembers" or path contains "graph_legend" or path contains "google" or path contains "-members.html" or path contains "/classes.html" %}
          {% continue %}
        {% endif %}
        {% if path contains "_source.html" %}
          {% assign path = path | replace: "reference/","reference/source/" %}
        {% endif %}
        {% assign path = path | replace: "reference/class","reference/classes/" | replace: "reference/classes/es","reference/classes" | replace: "reference/namespaces","reference/classes" | remove: ".html" | replace: "_1_1","/" | replace: "_01"," " | replace: "_8","." | remove_first: "/" | replace: "_ooo","/" %}
        {% assign psz = path | size %}

        {% assign title = node.title | replace "Namespace List","Reference" | remove: "Files (x86)/Jenkins/jobs/Simul Installers/workspace/Simul/PRODUCTS/TrueSky/Help/" | remove: " Source File" | remove: " Class Reference" %}
        {% if title == "404" or path contains "search-results" %}
          {% continue %}
        {% endif %}
        {% assign path_parts = path | split: '/' %}
        {% assign path_parts_size = path_parts | size %}

        {% assign paths = paths | push: path %}
        {% assign pages = pages | push: node %}
        {% assign path_sizes = path_sizes | push: path_parts_size %}

        {% assign titles = titles | push: title %}
        {% if node.url == page.url %}
          {% assign this_path = path %}
          {% assign this_node = node %}
          {% assign this_index = forloop.index0 %}
        {% endif %}
      {% endfor %}

Having build the arrays, we now iterate through the page URI's from the outermost level to as many levels as we want. The first loop:

      {% assign menu_level = 0 %}
      {% for node in pages %}
        {% assign node_path = paths[forloop.index0] %}
        {% assign node_title = titles[forloop.index0] %}
        {% assign node_path_parts = node_path | split: '/' %}
        {% assign node_path_parts_size = path_sizes[forloop.index0] %}
        {% unless node_path_parts_size == 1 %}
          {% continue %}
        {% endunless %}
<li>
            <a class="sidebar-nav-item{% if page.url == node.url %} active{% endif %} sidebar-1" href="https://www.blogger.com/%7B%7B%20node.url%20%7D%7D">{{ node_title }}</a></li>
{% if this_path contains node_path %}
              {% assign paths2 = site.array %}
              {% assign pages2 = site.array %}
              {% assign titles2 = site.array %}

For the second loop we will iterate through all the pages, but only pick out the ones with two elements in the path, and only the ones that contain the path to the current parent in loop 1. First we build the array, pages2:
{% for node2 in pages %}
                {% unless path_sizes[forloop.index0] == 2 %}
                  {% continue %}
                {% endunless %}
                {% assign path2 = paths[forloop.index0] %}
                {% if path2 contains node_path and node2.url != node.url  %}
                  {% assign paths2 = paths2 | push: path2 %}
                  {% assign pages2 = pages2 | push: node2 %}
                  {% assign titles2 = titles2 | push: titles[forloop.index0] %}
                {% endif %}
              {% endfor %}

Now iterate through pages2:

              {% if pages2.size > 0 %}
                {% assign menu_level = 1 %}
                <ul>
                {% for node2 in pages2 %}
<li><a class="sidebar-nav-item{% if page.url == node2.url %} active{% endif %} sidebar-2" href="https://www.blogger.com/%7B%7Bnode2.url%7D%7D">{{ titles2[forloop.index0] }}</a></li>
{% assign path2 = paths2[forloop.index0] %}

                  {% if this_path contains path2 %}

Now the same process for pages3, build the array:
                    {% assign pages3 = site.array %}
                    {% assign titles3 = site.array %}

                    {% for node3 in pages %}
                      {% unless path_sizes[forloop.index0] == 3 %}
                        {% continue %}
                      {% endunless %}
                      {% assign path3 = paths[forloop.index0] %}
                      {% if path3 contains path2 and node3.url != node.url  %}
                        {% assign paths3 = paths3 | push: path3 %}
                        {% assign pages3 = pages3 | push: node3 %}
                        {% assign titles3 = titles3 | push: titles[forloop.index0] %}
                      {% endif %}
                    {% endfor %}

And iterate through the array for the third level:
{% if pages3.size > 0 %}
                      {% assign menu_level = 1 %}
                      <ul>
                      {% for node3 in pages3 %}
<li><a class="sidebar-nav-item{% if page.url == node3.url %} active{% endif %} sidebar-3" href="https://www.blogger.com/%7B%7Bnode3.url%7D%7D">{{ titles3[forloop.index0] }}</a></li>
{% endfor %}
                      </ul>
{% endif%}
                  {% endif %}

                {% endfor %}
                </ul>
{% endif%}
            {% endif %}

      {% endfor %}

That's it for a set of Doxygen html pages - if your output is not from Doxygen the process may be simpler.

Building icu statically for linkage to Qt

It's bad enough having to link to all the Qt dll's without having to distribute its various dependencies as well - OpenSSL and  ICU chiefly among them. Then if you write a dll that plugs into someone else's exe you need to watch out that you could be trying to load a different version of a dependency dll that's used elsewhere in the exe.

So using static libs of the dependencies to link when you build Qt should help a lot. OpenSSL has some nice well-maintained static windows binaries (https://www.openssl.org/related/binaries.html), but icu does not.

There is this: http://www.npcglib.org/~stathis/blog/precompiled-icu/

But it uses static runtimes. So if you want a static build with dynamic (MD) runtimes, you're out of luck.

Building with Visual Studio

The Visual Studio solution all_in_one.sln that comes with ICU only builds dynamic libs. So you can modify the projects in the solution to link statically. Make sure that the output debug libraries have a suffix 'd' to their filenames, because they'll be built in the same directory as the release libraries.

Building with make

The recipe to build static ICU with MSVC is:

cd "%WORKSPACE%\source\"
set PATH=C:\cygwin\bin;%PATH%
call "C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\vcvarsall.bat" x86
dos2unix -f *
dos2unix -f configure
dos2unix -f mkinstalldirs

bash runConfigureICU Cygwin/MSVC -prefix=/cygdrive/d/Jarvis/icu_static/dist -enable-static -disable-shared
dos2unix -f install-sh

make && make install

Dos2unix prevents errors of the form '\r unknown command' in the make script.

Path order is crucial. If you call vcvarsall.bat before adding cygwin\bin to the path, you'll get something like:

icupkg: unable to open input file
/cygdrive/d/icu/win32-x86/data/out/build/icudt55l\coll\root.res"

But if you add cygwin/bin after %PATH%, you'll get something like:

link.exe is not a valid executable.

So only the above order works.

Output

You'll get these files in lib (for Win32) or lib64 (for x64):

icudt.lib icuin.lib
icuio.lib
icule.lib
iculx.lib
icutest.lib
icutu.lib
icuuc.lib
testplug.lib

and for debug:

icudtd.lib icuind.lib
icuiod.lib
iculed.lib
iculxd.lib icutestd.lib
icutud.lib
icuucd.lib testplugd.lib

When you come to build Qt, set an ICU environment variable to the parent icu directory, and in the Qt make command add the options:

-icu -I "%ICU%\include" -L "%ICU%\%LIB_OR_LIB64%"

where LIB_OR_LIB64 is "lib" or "lib64" depending on the platform.

Help Qt Designer to find plugins

Qt Designer can't find plugins without QT_PLUGIN_PATH being set. You should also set QT_QPA_PLATFORM_PLUGIN_PATH for the Qt gui applications. If you're writing your own Qt exe, you can set these paths internally.

set QTDIR=C:/Qt/qt5_msvc2012_x64_build/qtbase
set QT_PLUGIN_PATH=%QTDIR%/plugins
set QT_QPA_PLATFORM_PLUGIN_PATH=%QTDIR%/plugins/platforms

Note also that Qt Designer will only load plugins built with the same libraries. And when you do a debug build of Qt, although it appends "d.dll" to its dll's, it does not append "d" to its executables. So a debug build of Qt Designer, in the same Qt distribution as release, will overwrite the release build, thus preventing your plugins from loading! So always build debug first, then release.