Programming notebooks and online IDEs

Today many tools exist to write and test some code without actually opening any traditional programming IDE or even simple text editor. Many of these tools are in fact web applications. I decided to sum up some of them:

Jupyter Lab, jupyter.org – Powerful notebook with support for many languages. Although Python is the default, built-in programming language, users can install support for many other languages (kernels). Demo is available online. Local instances can be installed with pip or conda. Supported programing languages: Python (built-in), C, C++, R, Ruby, SQL, etc (see https://github.com/jupyter/jupyter/wiki/Jupyter-kernels)

Colaboratory, colab.research.google.com – google’s Python notebook with access to GPUs and TPUs. Based on the Jupyter project. Notebooks can be saved to google drive and shared.

CodeSandbox, codesandbox.io – full blown online IDE focused on web development. It offers creating, testing, and deploying apps written in HTML, JavaScript, TypeScript, Node.JS together with popular frameworks and libraries.

Franchise, franchise.cloud – web SQL notebook with support for many different database backends. Users can easily display data as charts, maps and export results to several file types. Franchise is an open source initiative with a repository hosted on GitHub.

SQL Notebook, sqlnotebook.com – desktop SQL notebook for Windows. Can connect to external databases as well as to built-in SQLite db. Data can be imported and exported.

CodeChef , www.codechef.com/ide – CodeChef site is all about programming challenges. Their simple “IDE” support a few dozen languages including C#, Java, Rust or even the NASM assembler.

OnlineGDB, www.onlinegdb.com – online compiler with debugging. Many languages supported: C++, Java, Objective C, C#, SQL, etc. Code can be shared and stored in an online account.

C++ shell, cpp.sh – simple online c++ compiler with support for capturing input. Rather old, currently up to C++14. GCC is used underneath.

ShaderToy, www.shadertoy.com – show me your shader site. Popular among game developers. Do not launch this page on weak machines: the main page often features heavy shaders. Supported programming language: GLSL (only fragment shader, and custom “sound shader”).

Automatic translation reloading in the Qt

Professional desktop applications require translations. The Qt framework has great support for translation like special CMake functions and Qt Linguist tool. Despite this fact, after translation is updated in the Qt Linguist, the person who is editing the translation must take some steps to view results of his/her work in the application. Below I will demonstrate how to make the work of the translator easier.

First of all we want to detect file changes. Qt has a special class called QFileSystemWatcher for doing this kind of job. We pass files (or directories) we want to monitor to the constructor of QFileSystemWatcher. Then we set the handler.

#include <QFileSystemWatcher>
...
#define COMMON_DIR "../common/"
#define SOURCE_DIR "../../src/myapp/"
...
// Some place of application initialization, in my case it was
// constructor of CApplication (derived form QApllication)
...
QStringList paths =
{
   COMMON_DIR "myapp_en_US.qm",
   COMMON_DIR "myapp_pl_PL.qm",
   SOURCE_DIR "myapp_en_US.ts",
   SOURCE_DIR "myapp_pl_PL.ts",
};

m_watcher = new QFileSystemWatcher(paths, m_mainWindow);
    connect(m_watcher, SIGNAL(fileChanged(QString)), m_mainWindow, SLOT( HandleFileChanged(QString) ));

After the file system watcher is in place we must write our handler. Below you can see the contents of this method.

void CMainWindow::HandleFileChanged(const QString& filename )
{
    if(filename.endsWith(".qm"))
    {
        //code for loading the translation
    }
    else if(filename.endsWith(".ts"))
    {
        QProcess cmdProcess;
        cmdProcess.setWorkingDirectory("../..");
        cmdProcess.start("cmd.exe", {"/c release_translations.bat"});
        cmdProcess.waitForFinished(-1);
    }
}

As you can see after *.ts file is modified we start a batch file. This batch file calls lrelease executable, which converts xml based ts files into binary qm files. Normally this step is done during the build. Below you can see the contents of release_translations.bat file.

SET PATH=%PATH%;C:\Qt\5.15.2\msvc2019_64\bin

lrelease src\myapp\myapp_en_US.ts -qm bin\common\myapp_en_US.qm
lrelease src\myapp\myapp_pl_PL.ts -qm bin\common\myapp_pl_PL.qm

As an alternative to using batch file one could call lrelease application directly from Qt application. In my case I decided to use batch file as proxy, where I can modify lrelease settings, without modifying the code.

After code is up and running user can just press Ctrl+S in the Qt Linguist and the new translation will jump right into the application he/she is working on.

Counting memory in UE4

UE4 has many stats and memory counting mechanisms with great tools to display it. Unfortunately useful tools, like Statistics window or Size Map, display not really useful data. Major problem is that in-editor memory is reported and not memory for a cooked game. Another problem is that even this memory is not always properly counted. Under the hood UE4 has two memory size counting mechanisms. One is serialization based and uses FArchiveCountMem serializer. Another is based on calling a special GetResourceSizeEx virtual function defined in UObject. You can see both used in functions like UEngine::HandleListStaticMeshesCommand. Digging deeper you can see that in fact GetResourceSizeEx internally uses FArchiveCountMem to count memory of a current object.

During my work on UE4 based games (Evil West and Shadow Warrior 3) counting memory properly was a top priority. Therefore I modified memory counting mechanisms to count memory for specific cooked platforms. Using this corrected counting mechanism I was able to create easy to use level streaming budgets (displayed in the viewport) and assets’ budgets based on Data Validation interface (https://docs.unrealengine.com/en-US/ProgrammingAndScripting/ProgrammingWithCPP/Assets/DataValidation/index.html).

Creating GI Multiplier: useful custom output material node

Our team was creating cutscenes for the game and they had problems setting up proper lighting. We generally use stationary lights, because we want precomputed GI and shadows for the game. Unfortunately the cutscene team was unable to get rid of GI on characters, without modifying lighting on the environment. Custom code was needed. At first I thought I would have to add another output to the main material node, but while looking inside the code I realised that I can create a custom output node. I used UMaterialExpressionBentNormalCustomOutput as my template. Work on C++ side consisted of creating the node and registering it in FMaterialAttributeDefinitionMap::InitializeAttributeMap.

AddCustomAttribute(FGuid(0x2E5E2BF3, 0x646F4531, 0x8E36B5C0, 0xA8055AFB), "GIMultiplier", "GIMultiplier", MCT_Float1, FVector4(1, 0, 0, 0));

The engine created a shader define and function for me. On HLSL side I just used them in GetPrecomputedIndirectLightingAndSkyLight function in BasePassPixelShader.ush

#if NUM_MATERIAL_OUTPUTS_GIMULTIPLIER > 0
  float GIMultiplier = saturate(GIMultiplier0(MaterialParameters));
  OutDiffuseLighting *= GIMultiplier;
  OutSubsurfaceLighting *= GIMultiplier;
#endif

When my node was ready I placed it together with a parameter from Material Parameter Collection in base materials for characters. Now our cutscene lighting artist is able to use it inside a sequencer to scale GI to his wish.

Debugging texture streaming in UE4

Today I was working on a problem with impostor texture streaming in UE4. I was not satisfied with the possibility of disabling the streaming for those textures, because they are quite large. I also found out that generating material streaming information does not work properly, because it reads GameThreadShaderMap that, as the comment states, is loaded from cooked assets and not available in the editor. This is a topic for further investigation, though. Meanwhile my solution was to modify FStreamingTexture::GetExtraBoost function, adding a branch for impostor texture groups with a boost value of 16.

I found some useful console variables and a command during the process. Here they are:

r.Streaming.DropMips 2 - disables mip map caching, not needed mip levels will be unstreamed immediately

r.Streaming.FullyLoadUsedTextures 1 – forces loading of an entire mip chain

r.Streaming.Boost <value> - global wanted texture size multiplier

InvestigateTexture <texture name> - displays streaming information  for a texture in a console window, most notably the highest loaded mip  level 

Hello again!

Five years ago I decided that in the age of Twitter, SlideShare and DropBox my own page is not needed any more. Now jpleskot.info is back in the form of a blog.