Loksim3D Developer Stories – Part 2 – C++ template magic

In Loksim a rail track can have a lot of associated properties. Properties like maximum speed, signals, illumination characteristics, name, stops and so on, which can be seen in the following screenshot:
props

So far the internal implementation of this system heavily depends on constants. For example, if you want to get the maximum speed on a track ‘t’ at position ‘pos’ you would use a function call similar to this:

auto& propSpeed = t.GetProperty(PROPERTY_MAXIMUM_SPEED, pos);

However, ‘p’ does not contain the actual maximum speed: A maximum speed is not a single value, it e.g. contains information if this maximum speed is different for trains with tilt technology or if it is shown in the timetable of the train operator. So a second query – again using a constant – is needed:

float maxSpeed = propSpeed.GetPropertyFloat(PROPERTY_MAXIMUM_SPEED_NORMAL_TRAIN);

This also shows, that one has to know the data type of the actual property when retrieving it. In fact, at the moment all these properties are stored as strings and converted to the requested type when they are queried by the GetPropertyType functions.

This system has two main advantages:

  • It can be extended quite easily: One just has to introduce a new constant PROPERTY_XXX
  • It can be serialized and deserialized to XML in a straightforward way: The values are just written out using the PROPERTY_XXX constants as attribute names

However, working with this system for some years also showed some disadvantages:

  • There does not exist any static check if the property that is queried really makes sense. E.g. in the example above, one could easily write the line propSpeed.GetPropertyString(PROPERTY_STOP_NAME). The compiler can never recognize this error and you would just get an empty property (string) back
  • It can be hard to find the appropriate constant for a property of a rail track. In fact I often end up checking the name in the GUI, searching for it in GUI resource file, than searching for the resource identifier in the code of the editor. There you find the connection between the constant and the GUI name, since there exists some kind of data binding between the GUI and the properties store
  • There exists no IntelliSense that could help to find the existing properties.
  • The data values are converted when they are retrieved and not stored in the original format

One additional characteristic of the system is, that there exist properties which are the same for both directions (e.g. illumination of the rail track) and other ones which are different depending if you are driving forwards or backwards on the track (e.g. stops and names). So far the developer has to know which properties are valid for both directions and which ones depend on the direction on the track. The GetProperty funktion takes an additional argument which determines if the property for the forward or for the backward direction is retrieved.

In the process of rewriting the system for generation of 3D models for the rail tracks I need to access some of these properties. I decided this would be a good occasion to experiment with a new system for such properties. My main design goals were the following:

  • Design system in a way, that the compiler can check for invalid usage of properties and allow the IDE to offer IntelliSense
  • Available properties and the types of the properties should be represented in some way in the source code
  • Static differentation between properties that are the same in both directions and ones where different properties for both directions exist
  • Values should be stored in a typesafe way and not needed to be converted each time properties are accessed
  • Allow fast access and filtering of properties, which means the property lists should store the properties by value and not allocate space on the heap for each single property

First of all I decided to use structures to store the different types of properties. Since every property has at least a position (relative to the track it is associated with), I created a base class for all different kind of properties:

template <bool TBothDirections>
struct RailProperty
{
	static const bool BothDirections = TBothDirections;

	RailProperty() : RailProperty(0.0f * units::meter) { }
	explicit RailProperty(units::distance_m position) :
		position_(position)
	{
	}

	units::distance_m position_;
};

The template parameter TBothDirections is used to differentiate between properties that are valid for both directions and one where different properties for forward and backward exist.

Each type of property is a subclass of the RailProperty struct. In the following two examplary properties are shown:

struct PropLimit : public RailProperty<true>
{
	PropLimit(units::distance_m position, l3d::units::speed_kph limitNormal = units::speed_kph::from_value(100.0f));
	PropLimit(units::distance_m position, l3d::units::speed_kph limitNormal, l3d::units::speed_kph limitNeigetechnik);

	units::speed_kph limitNormal_;
	units::speed_kph limitNeigetechnik_;
};

struct PropUeberhoehung : public RailProperty<false>
{
	PropUeberhoehung(/* ctor parameters */);

	// Properties
};

So how can these properties now be accessed? The user of the interface accesses them through a RailDescription class, which stores information about one rail. Each type of property is stored in an own custom list class. The lists itself are organized in a map[1], where std::type_index is used as a key. The type_index class is new in C++11, can be constructed from std::type_info (which is returned by the typeid operator) and can be used in (hash) maps.
The RailDescription class contains a Properties() function which provides access to a list of properties of the specified type. Since the type of the property is given as a template parameter, the concrete type of the return value is determined at compile time and can be used by IDEs for IntelliSense.

class RailDescription
{
public:
	/* other stuff */

	template <class Prop>
	railprops::PositionedPropertyList<Prop>& Properties()
	{
		auto& l = propLists_[typeid(Prop)];
		if (l.get() == nullptr) {
			l = std::make_unique<railprops::PositionedPropertyList<Prop>>();
		}
		return static_cast<railprops::PositionedPropertyList<Prop>&>(*l);
	}

private:
	std::map<std::type_index, std::unique_ptr<railprops::PositionedPropertyListBase>> propLists_;
	/* other stuff */
};

So what about the PositionedPropertyList class?

template <class Prop>
class PositionedPropertyList : 
	public std::conditional<Prop::BothDirections, PositionedPropertyListWithDir<Prop>, PositionedPropertyListWithoutDir<Prop>>::type
{
};

This class has only one purpose: If the properties that are stored in the list are handled differently for the two directions, PositionedPropertyList is a subclass of PositionedPropertyListWithDir, otherwise of PositionedPropertyListWithoutDir

The PositionedPropertyListWithDir class is responsible to provide two lists itself: One for the forward and one for the backward direction:

template <class Prop>
class PositionedPropertyListWithDir : public PositionedPropertyListBase
{
public:
	PositionedPropertyListWithDir();

	// forward
	PositionedPropertyListWithoutDir<Prop>& Fwd() { return *forwards_; }
	// backwards
	PositionedPropertyListWithoutDir<Prop>& Bwd() { return *backwards_; }
private:
	std::unique_ptr<PositionedPropertyListWithoutDir<Prop>> forwards_;
	std::unique_ptr<PositionedPropertyListWithoutDir<Prop>> backwards_;
};

The class in which the properties are actually stored is the PositionedPropertyListWithoutDir class:

template <class Prop>
class PositionedPropertyListWithoutDir : public PositionedPropertyListBase
{
public:
	// Property with position_ <= position
	// Returns nullptr if there does not exist such property
	const Prop* At(units::distance_m position) const
	{
		auto it = std::upper_bound(cbegin(orderedProps_), cend(orderedProps_), position);
		if (it != cbegin(orderedProps_)) {
			return &(*(it - 1));
		}
		else {
			return nullptr;
		}
	}

	/* Implementation of functions omitted */
	Prop* At(units::distance_m position);
	Prop& Add(const Prop& v);
	Prop& Emplace(units::distance_m position);		
	template<class... Args>
	Prop& Emplace(units::distance_m position, Args&&... args);
	bool DeleteAt(units::distance_m position);
	
private:
	std::vector<Prop> orderedProps_;
};

In this class the properties are stored in a std::vector and a handful of methods to retrieve, add new and delete properties exist. For some use cases probably more advanced methods to retrieve a range of properties are needed, which are not yet implemented.

In the end, the user can access the properties in the following way:

RailDescription rd;

rd.Properties<PropLimit>().Fwd().Add(PropLimit(10.0f * l3d::units::meter, 100.0f * units::kilometer_per_hour));
rd.Properties<PropLimit>().Fwd().Emplace(50.6f * l3d::units::meter, 100.0f * units::kilometer_per_hour);
auto& p1 = rd.Properties<PropLimit>().Fwd().At(4.0f * l3d::units::meter)

rd.Properties<PropUeberhoehung>().Emplace(100.0f * units::meter, UeberhoehungType::FesterWert, 45.0f * units::degrees);
auto& a = rps.Properties<PropUeberhoehung>().At(100.0f * l3d::units::meter)->angle_;

Everything is type checked at compile time and the IDE can provide full IntelliSense. I always knew that C++ templates are a real nice technique to work with. However, I really had a lot of fun creating this system for storing properties. I still think that it offers a nice interface forusers of this part of the code and can be easily used even by people who do not know anything about the way it is implemented. The main reason for this is, that it is statically typed and therefore prevents a lot of careless mistakes.

So far I did not think a lot about serialization of this new property store. I have a few options in mind, but at the moment I do not need it and writing code that is not needed at the moment is… unnecessary 😉

[1] Accoring to a talk by Chandler Carruth both std::map and std::unordered_map are not really optimal data structures. Maybe I will exchange the usage of std::map with a better map sometime, but this is not really relevant for this post.

Posted in Development | Tagged , | Leave a comment

Loksim3D Developer Stories – Part 1

If you read the “About” section of this blog, you may know that I am one of the developers of Loksim3D. To be honest, at the moment I am kind of the project’s lead developer. Never really wanted to be it, but – well – it just happened…

Loksim3D is an old – very old – project: The current version relies on a code base, where development was started in 2000. At least the graphics part of the code is that old, could be that some other parts are even older. Loksim currently uses Direct3D7 (yes seven) for 3D graphics. There does no longer exist any useful documentation for DirectX7. At least I cannot find any, if you know one please let me know. So yes, that’s not really up-to-date and therefore one – or *the* – goal for me is to update Loksim’s graphics engine to a more recent version. However, this is not an easy task.

I really learnt some interesting things analysing the existing 3D engine of Loksim, but a lot of things are just outdated and one would write things completely different for today’s hardware and with today’s C++. Additionally the code that interacts with DirectX7 is tightly coupled with the rest of the code.

Therefore I decided to rewrite the complete graphics stack to be able to upgrade the underlying graphics API. Before I started working on this task, the code of Loksim consisted of roughly 240.000 lines of code. It is just an approximation, because there are some libraries in source code included, which are not maintained by ourselves.

I started this task in the late summer of 2013. The first decision I made was to upgrade the whole project from VisualStudio 2010 to VisualStudio 2013 – I enjoy programming C++11 and VisualStudio 2013 offers much more features of C++11. The second decision was to implement all new graphics stuff in a separate VisualStudio project, an own static library. The main reason for this was my intention to set the warning level of the compiler as high as possible. However, in the existing codebase lots of warnings were generated when I tried to set the warning level higher. Therefore I decided to use a new project.

After all this more or less ‘organisational stuff’ I tried to find a strategy for the implementation (and integration) of a completely new graphics engine for Loksim. I identified the following different areas that interact with Direct3D7:
– Rendering of 3D-Objects. These objects are stored in a special XML-based file format (.3dobj and .l3dgrp)
– Rendering of rails. Rails can be seen as 3D objects that are generated by code
– Rendering of scenery/terrain. The terrain in Loksim is defined using an own file format (again XML-based) and heavily depends on the layout of the rail tracks
– Rendering of 2D content. The main application of 2D-rendering is the display of the driver’s cabin, which is only rendered in 2D

After identifying these four areas, I thought about how a migration might work the best. Of course one idea was to implement all areas at once and make a ‘big-bang integration’ afterwards. Mainly because I knew this would take a lot of time, I decided that this is not a desirable solution for me. I just want to release ‘my code’ regularly (in Loksim this means roughly twice a year), because otherwise I would just write code alone at home and would not get any feedback from users.

So I decided to start with 3D-objects only (I named this task ‘objectsystem’) and try to get a Direct3D7 version running again after the implementation of the required code parts of this area. I should add, that I did not want to change the rendering of the objects alone, I wanted to change almost everything in code that deals with 3D-objects. I already mentioned that Loksim objects are stored in XML files. These XML files were all parsed using a handwritten parser. I think this decision was made 14 years ago, because no usable (free) XML-parser existed at the time. However, this handwritten parser of course does not work ideally and nowadays nobody would think about writing his own XML parser. So using this nice Stackoverflow flowchart I decided to implement the I/O of 3D objects using PugiXML

Additionally I wanted to make life for 3D object authors easier:
– I wanted to extend the special Loksim file formats: ‘Loksim objects’ (.l3dobj) that consist of points and areas and ‘Loksim group objects’ (.l3dgrp) that consist of ‘Loksim objects’. However, a group object itself could not contain another group object. The new objectsystem should be able to deal with group objects that contain other group objects
– In Loksim objects the normal vectors (required for proper lighting) needed to be set for each point. If the same point (coordinate) is used in two or more areas, most of the time a separate point for each area had to be created. Otherwise the lighting of the object did not work very well. The new objectsystem should be able to store normal vectors for each area, which makes object autor’s life easier most of the time. Additionally this allows the automatic generation of normal vectors. Of course, defining normal vectors per points should still be supported if the object requires it (e.g. round objects)
– I wanted to support other 3D object file formats then the simple Loksim formats. I decided to use Assimp to support a bunch of different 3D model formats. It is really easy to use and supports much more features than Loksim itself can handle at the moment. So of course only a subset of all possibilities in these ‘external’ 3D model formats is currently supported, nevertheless object authors can use these formats as long as they think about what Loksim’s graphics engine currently supports.

Based on all these design goals I had to rewrite just everything that somehow deals with 3D objects: I/O, editing in the editor, converting to an internal mesh format, organisation in a tree structure, rendering, detection of visibility and so on

I published the first (somehow) working version of these changes internally (just for beta testers) in the end of April 2014. It took six further months of polishing, improvements, bug fixing, lots of cursing until a new Loksim version containing these complete revision of so an important part of the simulation could be published in the beginning of November 2014. Of course time is more than relative in this case: I do all this work in my free time: Sometimes I have more of it, sometimes not so much. Sometimes I am very motivated coding for Loksim, other times I prefer doing other stuff. And some times it is summer where a beer with friends is much more enjoyable than sitting in front of the computer in the evening 😉

Loksim3D 2.9

However, now the new version is out and hopefully users enjoy the now possibilities. Some stats of the new version: Loksim 2.9 now consists of around 265.000 lines of code. Around 20.000 lines of the new code is mainly focused on the 3D objectsystem, the rest of the roughly 5.000 other new lines is mostly ‘glue code’ to connect the existing code parts with the new objectsystem and some minor other new features.

In the future I will hopefully have some time to share more insights on the changes I made for version 2.9. In the meantime I am already working on a new ‘rail track system’, which will be my next step to achieve the goal of new graphics engine for Loksim

Posted in Development | Tagged , | Leave a comment

Qt with Visual Studio Express – 1

As a student I have access to the standard editions of VisualStudio for free (MSDNAA). However, I am working with some people that just have the Express versions of Visual Studio.
When we decided to start using Qt, we faced the challenge that Qt plugin cannot be used with the Express editions of Visual Studio. So I tried to get Qt working for Visual Studio Express 2013 as good as possible.

To get things running we used a step-by-step guide written by “Kabilen”. It was written for Qt4, but it also works with Qt5.
Using this guide, you can now compile projects using Qt in Visual Studio Express.

We added one additional step to setup our project: In the linked guide the QT directory is accessed using an environment variable QTDIR. We also setup this variable on our machines, since from time to time such variable is quite helpful. However, when this environment variable is accessed in the VisualStudio project, one looses the ability to use different versions of Qt in different projects on the same machine. Luckily VisualStudio offers the perfect solution for this problem: We already used a property sheet to store all machine/user-dependent paths. So we extended the property sheet to include a new user macro ‘QTDIR’ which can be used during the build process.

However, it is still very complex to add new Qt resources to the Visual Studio project.

The first thing we wanted to be able to do, is to add new Qt dialogs to our project. So I analyzed what the Qt plugin was doing when adding a new dialog:
– A .ui, .h and a .cpp file for the dialog is created and added to the project
– For the .h and .ui files, custom build events are created.
— For the .h file of the dialog, a custom build event creates a moc_dialogname.cpp file in the “GeneratedFiles/ConfigurationName” folder with the moc tool. ConfigurationName is replaced with the active configuration of the build, e.g. Release or Debug. It is used, because the moc tool destinguishes between a debug and a release mode.
— For the .ui file a custom build event is generated that creates a ui_dialogname.h file in a “GeneratedFiles” folder using the uic tool from Qt
– All mentioned files are added to the build process and into the VisualStudio solution. The files in the “GeneratedFiles/ConfigurationName” folder, are just used in build of the respective configuration. E.g. files in GeneratedFiles/Debug are excluded from Release builds.

All the steps can also be done using VisualStudio Express without any further utilities. However, it is a very cumbersome and error-prone process. Therefor I decided to write a powershell script that automates all this steps

The powershell script expects the name of the new dialog as parameter -dialogname, otherwise it asks for it. Additionally one can also specify a -project parameter. In our case we do not really use this parameter, the default value set in the QtProject.ps1 script is just set to the needed value.

The script uses 3 template files for the .ui, .h and .cpp files to create. These files can contain arbitrary code, the string ‘TEMPLATE_DIALOG_NAME’ is replaced with the name of the dialog to create. The content of the template files with the replaced dialog name is then written to a dialog directory ($dialogDir) in the project directory.

Afterwards the $project.vxproj and $project.vcproj.filters file are modified: In the .vcxproj file all the necessary build events and files to include to the build are added. In the .filters file all generated files that should not be modified by hand are added into one filter. This filter contains children for all different build configurations. The other three files (.cpp, .h and .ui) are added into three other filters.

We add the ui_dialogname.h file to the filter ‘Generated Files’, the moc_dialogname.cpp for the debug version into ‘Generated Files/Debug’ and the moc_dialogname.cpp for the release version into ‘Generated Files/Release’. For the source files we use ‘Qt\Source Files’, ‘Qt\Header Files’ and ‘Qt\Forms’

The script is available as initial version at GitHub. There is also a README file available which contains instructions how to setup and use the script for your own project.

As we are moving on using Qt, I plan to publish other utilities as soon as I see the need for it. I think the next step will be a utility for translation files.

Posted in Development | Tagged , , , | Leave a comment

qtwinmigrate – VisualStudio 2013

A week ago I faced the challenge to start the migration of a legacy project that was built using MFC to Qt. The “weapon of choice” was qtwinmigrate
There is an INSTALL.txt file included, that gives detailed instructed how to compile the project. However, following these instructions using the VisualStudio 2013 compiler I ran into a problem: When building the project the linker complained about a missing mfc120.lib file
I searched my computer for this file and only found a mfc120u.lib. So it seemed like I only had the Unicode version of the library. I remembered reading something about Non-Unicode MFC projects quite some time ago, but I found the blog post again.
In this post there is a download link for the MBCS libraries for VisualStudio 2013, that include the required mfc120.lib file. After installing these libraries, the build process of qtwinmigrate succeeded.

For the first steps with qtwinmigrate there exists a pretty good Walkthrough. However, adding a new dialog like described in the “A new Qt Dialog” section, did not exactly worked as described in the walkthrough.

I had to replace the code for the OnAppOptions function with the following:

void WindowsApp::OnAppOptions()
 {
     QWinWidget *win = new QWinWidget(theApp.m_pMainWnd);
     win->showCentered();

     OptionsDialog *dialog = new OptionsDialog(win);
     dialog->setAttribute(Qt::WA_DeleteOnClose);
     dialog->show();
 }

Maybe this is because the walkthrough was written for an older Qt version.

However with this small tweak, so far qtwinmigrate works very well!

Posted in Development | Tagged , | Leave a comment

C++ – Image Dimension

Recently I needed to determine the image dimension of graphics files in C++. I was required to support BMP, PNG and TGA files. I thought that must be something many devs need to do now and then. However, I did not find a suitable solution on the net, because I did not want to load the whole image into memory just to get the image size.
Of course the next step was to ask on Stackoverflow. With the answers provided there, I could write my own simple piece of code:

bool GetImageDimensionFromFile(const std::string&amp; fileName, 
    int& width, int& height)
{
	std::ifstream in(fileName, std::ios::binary);
	std::uint32_t temp = 0;
	bool suc = false;

	if (!in.is_open()) {
		return false;
	}

	auto fmt = GetImageFormatFromFilename(fileName);
	if (FORMAT_BMP != fmt) {
		width = height = 0;
		in.read(reinterpret_cast<char*>(&width), 1);
		in.read(reinterpret_cast<char*>(&height), 1);
		if (width == 0x42 && height == 0x4D) {
			fmt = CXIMAGE_FORMAT_BMP;
		}
	}
	width = height = 0;

	switch (fmt) {
	case FORMAT_PNG:
		in.seekg(16);
		in.read(reinterpret_cast<char>(&width), 4);
		in.read(reinterpret_cast<char>(&height), 4);
		width = ntohl(width);
		height = ntohl(height);
		suc = true;
		break;
	case FORMAT_BMP:
		in.seekg(14);
		in.read(reinterpret_cast<char*>(&temp), 4);
		if (temp == 40) {
			// Windows Format
			in.read(reinterpret_cast<char>(&width), 4);
		        in.read(reinterpret_cast<char>(&height), 4);
			suc = true;
		}
		else if (temp == 20) {
			// MAC Format
			in.read(reinterpret_cast<char>(&width), 2);
		        in.read(reinterpret_cast<char>(&height), 2);
			suc = true;
		}
		break;
	case FORMAT_TGA:
		in.seekg(12);
		in.read(reinterpret_cast<char>(&width), 2);
		in.read(reinterpret_cast<char>(&height), 2);
		suc = true;
		break;
	}

	if (in.eof()) {
		return false;
	}
	return suc;
}

For the ntohl function I used a function that is defined in the boost header boost/asio.hpp. The function is not in the public documentation of boost, so you may run into problems with new Boost versions. However, it is also possible to use a platform specific ntohl function or a function from another library

I have used this piece of code for quite a time for many images and so far I did not run into any problems. However, there may be some exoctic variations of an image format, where this simple solution is not sufficient…

Update 22.09.2014
Recently I came across a .PNG file which in reality is a BMP file. This file is of size 8×4 pixels, so its a pretty small image. The behavior, that imaging software like Paint.NET produces such files is reproducable. So maybe the format detection using just the extension is not that perfect. For now a made of workaround to check each file if it starts with the magic bytes of BMP (0x42 0x4D). I should probably use the magic bytes in all cases later on. However, so far this was the first problem I encountered using the function above.

Posted in Development | Leave a comment

Hello (Blog-) World!

Hi there,

here I am, after I decided to start a small blog.

I have no idea how regularly I will be blogging about stuff, if anyone will every read what I am posting and for sure I do not know if anyone finds the stuff interesting or helpful.

So I will just explain the reasoning about this blog:

What is the content of this blog?

I will blog about tech stuff, mainly about development stuff
It just occurs now and then that Google, Bing or whatever search engine does not have the answer. Doesn’t happen that often, but you know, sometimes you just have to find out yourself. Exactly this kind of stuff I find out myself and what could be interesting for another developers will be the content of my blog.
Things were you think: ‘What the hack, why is there no description how to configure my system/library/IDE in this and that way?’ Or ‘Why is there no available solution/library/code for this simple problem? There most be tons of people facing the same problem!’

Why are you writing in English and why is your English so terrible?

I just think that English is the language when it comes to development. Every time I search the web to find information about development or tech stuff in general, I search in English. And all my friends I know do it in the same way. It is just expected to understand English when you are working as a developer and it is the only possibility to share your information around the whole world. So this decision was rather easy for me.
So what about my English? I am from Austria, English is not my native language. I hope everyone understands what I am writing. However, it may not be my best English. This blog is not about improving my language skills and I will not spend half of the time checking spelling or grammar of my entries. I will do my best, I hope that’s fine for the most of you.

Maybe at same later time I will add an “About” page or something similar. But for now, I just want to get things running…

Posted in General | Leave a comment