Merging autogen configurations

Now that I can use scenProc to easily create new autogen, the urge to add custom autogen models increases as well. To make the autogen even more realistic I want to add custom roofs, custom vegetation models, etcetera. But all these modifications to the autogen configuration files give developers some headache when trying to deploy their scenery. How to make sure that the end user gets these modifications as well?

The problem is that there is one autogen configuration that all developers modify. So if not done correctly a developer might wipe out the modifications done by another developer. There are a few approaches that are used nowadays:

  1. Some developers just copy their modifications over the existing files, whiping out previous modifications
  2. Some developers merge their modifications into the autogen configurations files
  3. Some developers merge their modifications and the modifications of sceneries often used together with their product into the autogen configuration files

Obviously one approach 2 is the right way! Approach 1 is bad because it just erases any modification done by other developers. Your scenery will work fine, but others will stop working. Approach 3 might sound like a nice service to the end user, but what happens if the other scenery you include the modifications from update their custom autogen configuration? In that case your product would still include the old one and in that case the order of install will determine the result the end user gets. That’s not good.

So I would say the only way developers should distribute their custom autogen configurations is to only distribute their own modifications (not the default MS configurations, not those of other developers). And developers should properly merge their configurations into the main configuration file.

But of course we will have the issue of developers who don’t play it nice and remove other developers modifications, how do we deal with that? It would be annoying that end users have to reinstall your scenery again just to restore the autogen configuration files.

So I would propose that we make a tool that mimics the behaviour the Microsoft should have added out of the box already. A tool that allows each scenery to have their own local autogen configuration files. Each developer will put his own modifications in a folder within his scenery and the tool will scan all these custom autogen configurations at startup of FS and merge them into one main autogen configuration file. And if this is done at each startup of the sim, developers who wipe out your configurations don’t do any hurt anymore.

I plan to try to create such a tool soon. But I would love to hear back from other developers if this sounds like a good approach. Do you have other use cases in mind that such a tool should cover? Or would you like to help me develop and test this tool? Just let me know!

Support for FSX:SE

fsxsteamAbout two months ago a new edition of FSX was released on Steam, which is basically the same as the FSX Gold Edition. My different tools have been updated by now to be able to support this new edition as well.

When you have both FSX and FSX:SE installed the tools will try to detect FSX first and if found they will use that path to load FS specific items from. If you only have FSX:SE installed that version will be used as FSX install by the tools.

The main difference is that FSX:SE does not include the SDK tools, which are used by most of my tools. And this week we finally got an answer from Dovetail Games as well that they will not include these tools either. This means that developers with only FSX:SE will need to get the SDK tools in a different way.

Installing the FSX SDK only works if you have the boxed version of FSX, since only SDK updates can be downloaded. But you need to install the original SDK from the disc first. So that’s not an option in general for FSX:SE only users. Luckily you can download the P3D 1.4 SDK, which contains equivalent tools and can be downloaded for free. So if you want to use my tools to make scenery and only have FSX:SE installed, be sure to grab the P3D 1.4 SDK and install that. The tools will find the location of the SDK automatically based on the registry settings.

Crashes at startup

Since I moved to .NET 4.0 a few weeks ago, some of the tools had random crashes, usually at startup. It turned out this was an issue in the text editor control that is used in the tools. Mainly scenProc and FXEditor were affected, because the text editor is a main part of the GUI there, but ModelConverterX also uses it in some editors.

Yesterday I have fixed these issues, by disabling the buggy part of the control (it was a part we don’t use in the tool anyway). So please make sure to grab the latest development release.

Recent changes

Just a quick note, I am aware that after the recent changes to my build server that the recent changes page is not always very useful. The change log message are sometimes truncated, meaning that not all information is visible. I will try to fix this, so that a good overview of changes is available.

Different .NET version

From today the development releases of my tools target a different version of the .NET framework. You’ll need .NET 4.0 from now on for all my tools. So be sure that you have it installed.

For scenProc I have used some of the new functionally of .NET 4.0 directly. As a result of the better parallel processing the image analysis of the feature detection step is now a lot faster. On my PC almost three times faster, but that depends on how many CPU cores you have as well of course.

Build server changes

Related to the move of this website to the new server, I also had to make a few changes to the build server that I use to make the development releases. I’ll just start with an overview of the main changes, below I will explain in a little more detail how my build process works.

The main changes are:

  • I have moved from SVN to Git as a version control system, so the development releases won’t be identified by a revision number anymore, but with a Git commit ID. This means you will no longer have nicely increasing numbers, but each release will still have an unique identification number.
  • The previous development releases are no longer accessible on the scenerydesign.org site, only the last one is available. I  haven’t been able to figure out how to get this working in the new build system and also the amount of disk space on the new server probably wouldn’t be big enough.
  • The update notifications of old versions of the tool are broken now. That’s because the filetype (a file without extension) that I used to download the latest release date from is not allowed on the new server. So I had to change the filename and I have updated the current development release to check for this new name.

And now some more background information for those that are interested:

Old situation

For quite some time the daily development release is build automatically each night if there are changes. I did this to not have to manually publish a new release every time. And by using a build server, I also have an extra check if everything is correct in version control. Because it builds the applications from the sources in version control, so if I forget to commit a file I will notice that quicker.

Before all components of my build system were running on a small Linux based webserver I have at my house. So the SVN version control, the Jenkins CI build server and the web server where you can download the files were all running on the same machine.

You might have read Linux based server and wonder how I am building Windows tool on it? I have been using Mono for that all those years. So the tools you have been using have been build using the open source .NET alternative Mono.

But this small home server is not very powerfull (so building the whole tool takes like 15 minutes) and worse it is running on an older Linux distribution, which I can’t upgrade easily due to the specific hardware to make the server energy friendly. This old Linux distribution also meant I can’t install recent Mono versions and therefore I am stuck with .NET 3.5 or lower.

So now with the move to a new house in a few weeks, which would give downtime of time home server anyway, I thought it would be a good idea to change my build system a bit and get rid of some of the limitations.

New situation

Not really related to the build system, but I also decided to move to a different version control system, I’m now using Git instead of SVN. And I have my Git repository in the cloud as well, so that there is a backup outside of my house as well.

The Jenkins build server is now temporarily running on my development machine, so it is running on Windows instead of Linux now. This means I can upgrade to more recent .NET versions. Especially for scenProc the better support for parallel computing in .NET 4.0 could be useful.

The results of the build server are now uploaded to the webhost where the SceneryDesign.org website is now hosted. You have probably already noticed it is no longer hosted at my house, since the site got a bit quicker now.

Once we have moved into our new house, I will probably setup the Jenkins build server on a spare PC, sot that it doesn’t have to run on my development machine anymore. But I don’t have the space (or time) at the moment to do so.

Server move – part 2

In my previous post I talked about an upcoming server now. This evening the domain will be switched from the old to the new server. I have already uploaded all files to the new server and just before the move I’ll do a last sync of all databases. So hopefully it will be a smooth move. But if you notice some issues you know why.

Server move

As you might know this website is running on a small home server in my living room. But since the server is getting a bit old and slower, I’ll be moving this site to another server somewhere in the next days. This might give a little downtime, so you have been warned now.

The new server should be quicker, so the site will become more responsive. And it’s no longer located at my house, so that prevents downtime in a few months when we’ll be moving to our new house.

Scrubs

For the Nantucket project I have used the scenProc feature detection function to get the autogen trees from the imagery. But would a similar approach also work for scrubs?

First I tried with polygonal autogen vegetation. The detection of the polygons went OK, but the density of the resulting autogen in FSX was much too sparse to be realistic.

So the next attempt was to use the PLACEPOINTSINPOLYGON step to generate point features within the detected polygons and then create rectangular vegetation autogen from those points. That did work, but it was slooooooow. So also not a feasible way to generate scrubs for the entire island.

Therefore I have modified the feature detection step now so that it can generate point features directly, instead of polygons. This saves the step to generate them later. How does it work? Here are the main steps of the point feature detection algorithm:

  1. The first step is similar to the polygon detection algorithm. The texture filter is ran over the imagery and the result of this is a new raster image that indicates areas that match the criteria of the filter.
  2. In the polygon detection mode this raster image would be polygonized into polygons, but in the point detection mode the raster image is used differently. Using the user provided spacing in pixels the raster image is tested to see if that location matches the filter or not. If it matches a point feature is made at the location.

For Nantucket I used a spacing of 20 meters for my scrubs and that gives a much denser scrub autogen than I got with the polygonal autogen.

This update of the feature detection step is available in the next development release. Be aware that the changes to the texture filter can influence your saved texture filter configuration files. The filter now also takes a minimum and maximum value of each pixel channel and if you load an previously saved filter this maximum will be zero, resulting in no matches. So please check your existing filters with this new update.

The only problem I have found so far with the scrub detection is that it’s much slower than trees. Not really because the algorithm, just because there are potentially more areas with scrubs than trees in many images. So there are more pixels to examine. I have some ideas to speed up the feature detection by moving part of the logic from the CPU to the GPU. It will be interesting to see how much that can speed up things in the future.

Status update

After spending almost 10 weeks in hospital, our son could finally come home this week. It’s very good to have the family all at home again, not having to travel to the hospital continuously.

The only small downside is that our sleep is now interrupted in the night, but that’s normal with a small baby at home of course.

So as things return more to normal at home, I can hopefully pick up some FS related things again. I would really like to do some debugging on my tools as a way to relax after all these things.

But on the other hand we are also still busy with our new house, so time will still be sparse in the near future. So hopefully you’ll see me on the forum now and then again, but at least you now know I’m doing ok.