Thursday, 2 August 2018

Finding and removing files added to git by accident

If for example, you've added lib files by mistake to a large git repo, and want to remove them, but don't know the exact paths, use this:

git ls-files *.lib>lib.bat

Then in lib.bat you may have e.g.:

Add git rm --cached to the front of each line, then run the batch file and commit the result.

Thursday, 21 June 2018

How to make a custom Wizard for Unreal Editor

I wanted to create a wizard in the trueSKY Unreal plugin that would make it easier for users to add trueSKY to UE scenes. I was following this video where Epic's Michael Noland describes various ways to modify the Editor. So I made a custom Property Editor window with settings to select a sky sequence, create a TrueSkyLight etc.

But it didn't look very friendly. And implementing a wizard-style Apply button just put a button in amongst the other settings - not great. After some searching in the UE codebase, I discovered the SWizard class that Unreal Editor uses for its own wizards. Here's what you do:

1. Create a class derived from SCompoundWidget containing a TSharedPtr<SWizard>. Mine looks like this:

DECLARE_DELEGATE_FourParams( FOnTrueSkySetup, bool, ADirectionalLight* ,bool, UTrueSkySequenceAsset *);
#define S_DECLARE_CHECKBOX(name) \
 bool name; \
 ECheckBoxState Is##name##Checked() const { return name ? ECheckBoxState::Checked:ECheckBoxState::Unchecked;} \
 void On##name##Changed(ECheckBoxState InCheckedState) {name=(InCheckedState==ECheckBoxState::Checked);}

class STrueSkySetupTool : public SCompoundWidget
 SLATE_BEGIN_ARGS( STrueSkySetupTool )
 /** A TrueSkyLight actor performs real-time ambient lighting.*/
 /** TrueSKY can drive a directional light to provide sunlight and moonlight.*/
 /** If there's no directional light in the scene, you can create one with this checkbox.*/
 /** The TrueSKY Sequence provides the weather state to render.*/
 SLATE_ARGUMENT(UTrueSkySequenceAsset *,Sequence)
 /** Event called when code is successfully added to the project */
 SLATE_EVENT( FOnTrueSkySetup, OnTrueSkySetup )
 /** Constructs this widget with InArgs */
 void Construct( const FArguments& InArgs );
 /** Handler for when cancel is clicked */
 void CancelClicked();

 /** Returns true if Finish is allowed */
 bool CanFinish() const;

 /** Handler for when finish is clicked */
 void FinishClicked();


 void SetupSequenceAssetItems();
 void CloseContainingWindow();
 /** The wizard widget */
 TSharedPtr<SWizard> MainWizard;
 FOnTrueSkySetup OnTrueSkySetup;

The SLATE_ARGUMENT macros allow initialization of named parameters in this style:

TSharedRef<STrueSkySetupTool> TrueSkySetupTool = SNew(STrueSkySetupTool).OnTrueSkySetup(OnTrueSkySetup1).CreateTrueSkyLight(true);

etc. This is super-useful.

2. Create a callback for the wizard to execute:
FOnTrueSkySetup OnTrueSkySetupDelegate;

3. Create a window for the widget. This function is called when the menu option to start the wizard is selected:

void FTrueSkyEditorPlugin::OnAddSequence()
 TrueSkySetupWindow = SNew(SWindow)
   .Title( NSLOCTEXT("InitializeTrueSky", "WindowTitle", "Initialize trueSKY") )
   .ClientSize( FVector2D(600, 550) )
   .SizingRule( ESizingRule::FixedSize )
 TSharedRef TrueSkySetupTool = SNew(STrueSkySetupTool).OnTrueSkySetup(OnTrueSkySetupDelegate);
 TrueSkySetupWindow->SetContent( TrueSkySetupTool );

If the main frame exists parent the window to it. The main frame should always exist...

 TSharedPtr< SWindow > ParentWindow;
 if( FModuleManager::Get().IsModuleLoaded( "MainFrame" ) )
  IMainFrameModule& MainFrame = FModuleManager::GetModuleChecked<imainframemodule>( "MainFrame" );
  ParentWindow = MainFrame.GetParentWindow();
 bool modal=false;
 if (modal)
  FSlateApplication::Get().AddModalWindow(TrueSkySetupWindow.ToSharedRef(), ParentWindow);
 else if (ParentWindow.IsValid())
  FSlateApplication::Get().AddWindowAsNativeChild(TrueSkySetupWindow.ToSharedRef(), ParentWindow.ToSharedRef());
4. Implement the setup tool:
void STrueSkySetupTool::Construct( const FArguments& InArgs )
 OnTrueSkySetup = InArgs._OnTrueSkySetup;

The interface to build the actual UI is really interesting. By overloading the [] and + operators, Epic lets you specify the widget structure like so:

  .BorderImage( FEditorStyle::GetBrush("Docking.Tab.ContentAreaBrush") )
    SAssignNew( MainWizard, SWizard)
    .CanFinish(this, &STrueSkySetupTool::CanFinish)
    .FinishButtonText(  LOCTEXT("TrueSkyFinishButtonText", "Initialize") )
    .OnCanceled(this, &STrueSkySetupTool::CancelClicked)
    .OnFinished(this, &STrueSkySetupTool::FinishClicked)
    .InitialPageIndex( 0)
      .TextStyle( FEditorStyle::Get(), "NewClassDialog.PageTitle" )
      .Text( LOCTEXT( "WeatherStateTitle", "Choose a Sequence Asset" ) )
       .Text(LOCTEXT("TrueSkySetupToolDesc", "Choose which weather sequence to use initially.") )
       .TextStyle(FEditorStyle::Get(), "NewClassDialog.ParentClassItemTitle")


So by adding new +SWizard::Page() elements we add pages to the wizard.

5. Finally, implement the callback that the delegate calls when you click "Finish":

void FTrueSkyEditorPlugin::OnTrueSkySetup(bool CreateDirectionalLight, ADirectionalLight* DirectionalLight,bool CreateTrueSkyLight,UTrueSkySequenceAsset *Sequence)

The end result looks like this:

Full source for this is at our UE branch, (register at Simul to access).

Tuesday, 10 April 2018

Signing installers with certificates

Windows Defender has recently decided to falsely mark all of our installers as containing some virus or other.

It'll be a long long while before they get around to questioning whether their algorithms are in fact, "full of it", as they say, so let's see what happens if we sign our executables using a root certificate.

First, get a certificate, from Comodo. This takes weeks while they check whether an arbitrary non-governmental organization, Dun and Bradstreet, regards your company as genuine. Just check with Companies House? Way too simple!

So you need to get a DUNS number from D&B, then buy a certificate from tucows/Comodo.

After jumping through their hoops (which don't seem to be very secure to me, just cumbersome), you'll get a .crt file.

Then will tell you how to convert your crt to a pfx.

Finally, use the pfx and signtool.exe (in the Windows SDK) to sign your executable.

Friday, 17 November 2017

Building Unreal Engine projects from the solution using MSBuild

If you want to use MSBuild to build UE4 projects, but need to build them from within the solution instead of specifying the vcxproj file (which doesn't always work correctly), you need to use the "Target" /t: command line parameter, like so:

"path to MSBuild.exe" /t:Engine\UE4 /p:Configuration="Development Editor" /p:Platform=Win64 UE4.sln

Key things to note:
  • the configuration and platform specifiers are Solution-style, with spaces instead of underscores, and Win64 instead of x64 etc.
  • The solution folder path must be specified in the target parameter, otherwise MSBuild will not recognize the project name.

Saturday, 3 June 2017

Advanced custom Qt Container Widgets and Qt Designer

Qt has a nice UI editor called Designer, and you can create custom widgets that go in Designer's toolkit. But the only example I've ever found is this one in the Qt docs, which doesn't explain how to create container widgets.

The problem is to create a widget that contains some decoration or controls, but also has a sub-window where people can put their own widgets.

For example, I wanted an "accordion" control that had a checkbox at the top to open and close it, then to be able to put any other control inside this.

The way it will be structured is a QAccordion, with a VBoxLayout, will contain a QCheckbox and a QWidget called the content widget. This content widget will have its own VBoxLayout, where the controls will go.

You will create two classes: one called (say) QAccordion, which implements the widget, and one called QAccordionInterface, which tells Designer that it's available.

<#include <qwidget>
#include "GeneratedFiles/ui_QAccordion.h"
#include "Export.h"

class SIMUL_QT_WIDGETS_EXPORT QAccordion : public QWidget
 Q_PROPERTY(QString title READ title WRITE setTitle DESIGNABLE true)
 Q_PROPERTY(bool open READ isOpen WRITE setOpen DESIGNABLE true)
 QAccordion(QWidget *parent = 0);
 void setTitle(QString f);
 QString title() const;
 void setOpen(bool o);
 bool isOpen() const;
public slots:
 void on_accordionCheckBox_toggled();
 void setSearchText(const QString &);
 void childEvent ( QChildEvent * event ) override;
 void paintEvent(QPaintEvent *) override;
 Ui::Accordion ui;
 bool setup_complete;
 QWidget *contentsWidget;
 void hookupContentsWidget();

The subclass Ui::Accordion shows that I created the basic class in Designer itself. This is optional, but the QAccordion.ui file is just:

<?xml version="1.0" encoding="UTF-8"?>
<ui version="4.0">
 <widget class="QWidget" name="Accordion">
  <property name="geometry">
  <property name="sizePolicy">
   <sizepolicy hsizetype="Preferred" vsizetype="Preferred">
  <property name="windowTitle">
  <layout class="QVBoxLayout" name="verticalLayout">
    <widget class="QCheckBox" name="accordionCheckBox">
     <property name="text">
     <property name="checked">

By putting the layout and checkbox in the ui file, they will be created in code, in Ui::Accordion.

But: if we were to create the whole thing, including the contents widget in here, after we built the class the contents widget would NOT be accessible in Designer, and neither would its layout be recognized. So instead we put these in a function called domXml in QAccordionInterface.

 QString QAccordionInterface::domXml() const
     return "<ui language=\"c++\">\n"
            " <widget class=\"QAccordion\" name=\"accordion\">\n"
            "  <property name=\"geometry\">\n"
            "   <rect>\n"
            "    <x>0</x>\n"
            "    <y>0</y>\n"
            "    <width>100</width>\n"
            "    <height>24</height>\n"
            "   </rect>\n"
            "  </property>\n"
            "  <property name=\"toolTip\" >\n"
            "   <string></string>\n"
            "  </property>\n"
            "  <property name=\"whatsThis\" >\n"
            "   <string>.</string>\n"
            "  </property>\n"
      "  <widget class=\"QWidget\" name=\"contentsWidget\" native=\"true\" >\n"
      "   <layout class=\"QVBoxLayout\" name=\"accContentsVLayout\">\n"
      "    <property name=\"spacing\">\n"
      "     <number>2</number>\n"
      "    </property>\n"
      "    <property name=\"leftMargin\">\n"
      "     <number>2</number>\n"
      "    </property>\n"
      "    <property name=\"topMargin\">\n"
      "     <number>2</number>\n"
      "    </property>\n"
      "    <property name=\"rightMargin\">\n"
      "     <number>2</number>\n"
      "    </property>\n"
      "    <property name=\"bottomMargin\">\n"
      "     <number>2</number>\n"
      "    </property>\n"
       "    </layout>\n"
      "  </widget>\n"
            " </widget>\n"

By specifying the contents widget and its layout here, Designer will know to dynamically create them when you add a QAccordion, so they'll appear in the editor. You can then drag any control into the contents widget, and it will be correctly positioned. Be careful that you drag it to the contents widget and not the QAccordion itself or a subcontrol. Designer doesn't properly obey its "isContainer" function, so it sees any custom control as a container, not just the ones you indicate.

So now in designer, we can add QAccordions. Without styling they just look like checkboxes with a space below where you can drag controls:

After applying some styling, the final result looks like this:

The accordion elements - Cloud Window, Precipitation etc are inside a searchable property panel, implemented on the same principles.

And here are the files for the final class:

Saturday, 20 May 2017

Sfx: a generic effect compiler for shaders

So Microsoft, Nvidia and the rest used to support effect files: a text source file that contained multiple shaders, but also "techniques" and "passes", where each pass can have state specified: blending, rasterization etc.

Some time ago, for reasons I guess of supply and demand, fx fell out of fashion. Microsoft still provides the D3D11 version of its Effects library as open source here, and this reads binary output that the fxc tool can create. But Fxc is being replaced by by this, which doesn't support effects. Nvidia's Tristan Lorach proposed a new framework, nvFX (pdf), but I don't think it's under active development.

D3D12 has no effect support. So we need to add it.

My approach at Simul is called Sfx. The idea is to take an initial file that's compatible with Microsoft's HLSL effect format, with the code written in HLSL. Sfx will extract the passes and compile all the relevant shaders by building smaller individual shader source files and calling an external compiler. A small json file will specify which compiler to use, and various other parameters to allow translation from HLSL to whatever language the compiler expects.

For example, HLSL.json looks like:

  "compiler": "C:/Program Files (x86)/Windows Kits/10/bin/x64/fxc.exe",
  "defaultOptions": "/T {shader_model} /nologo",
  "sourceExtension": "hlsl",
  "outputExtension": "cso",
 "outputOption": "/Fo",
 "entryPointOption":  "/E{name}",
  "multiplePixelOutputFormats": false

So Sfx should be compiler-independent. It outputs two things: a text file with the .sfxo extension, and a number of platform-specific shader binaries. The sfxo looks like this:

texture fontTexture 2d read_only 0 single
SamplerState clampSamplerState 9,LINEAR,CLAMP,CLAMP,CLAMP,
SamplerState cmcNearestSamplerState 13,POINT,CLAMP,MIRROR,CLAMP,
RasterizerState RenderNoCull (false,CULL_NONE,0,0,false,FILL_SOLID,true,false,false,0)
RasterizerState wireframeRasterizer (true,CULL_NONE,0,0,false,FILL_WIREFRAME,false,false,false,0)
BlendState AlphaBlendRGB false,(true),1,1,4,5,0,0,(7)
DepthStencilState DisableDepth false,0,4
 technique backg
  pass p0
   rasterizer: RenderNoCull
   depthstencil: DisableDepth 0
   blend: AlphaBlendRGB (0,0,0,0) 4294967295
   vertex: font_FontVertexShader_vv.cso,(),(),()
   pixel: font_FontPixelShader.cso,(),(),()

This stands to improve over time. In this case, we've taken an sfx file containing this definition:

VertexShader vs = CompileShader(vs_4_0, FontVertexShader());
technique text
    pass p0
  SetRasterizerState( RenderNoCull );
  SetDepthStencilState( DisableDepth, 0 );
  SetBlendState(AddBlendRGB,vec4( 0.0, 0.0, 0.0, 0.0), 0xFFFFFFFF );

so we've compiled FontPixelShader() and FontVertexShader() into the cso files: this example is for HLSL. As we extend Sfx to other languages, the json definition in particular will become more complex - possibly using regexes to specify how HLSL is translated into other C-style shader languages.

Thursday, 18 May 2017

HDR output in Unreal Engine

To get Unreal Engine to output from consoles in HDR format (i.e. to HDR TV's), there are a few settings. according to this post, there are variables that can go in the .ini files. But in my experience, setting EnableHDROutput to 1 in Engine.ini, causes UE to crash on initialization. As of May 2017, the solution seems to be to either enter the hdr settings every time via the console, or use the Blueprint function EnableHDRDisplayOutput:

This seems to also cover the r.HDR.Display.OutputDevice and r.HDR.Display,ColorGamut settings, so one call will do it.