Recent changes

In the last three weeks I have been a bit busy, so I haven’t posted about the changes I made to my tools recently. Here is a quick overview of the most important changes:

  • There was a memory leak in scenProc when using the SplitGrid step, this has been fixed. But if you still have a version build between the 2nd and 11th of March, I would urge you to update to the latest version.
  • You can now highlight the selected part in the hierarchy viewer in the preview. This can be useful to identify with node of the scenegraph you have selected.
  • There was a bug that objects with empty LOD did not export correctly anymore, that has been fixed now.
  • I have added a new batch operator to apply a material template. This means that you can now apply a certain material template to a selection of objects using the batch convert wizard. Given that applying a night texture is also a material template, the special batch operator to apply a night texture has been removed now.

Visibility tags

I have added a new feature to ModelConverterX, it now also uses the visibility tags that are used often in FSX aircraft models. They are read from either the MDL or the X file. And when exporting to FSX MDL they are written as well. So this should mean that such visibility tags no longer get lost when converting a FSX MDL with ModelConverterX.

At the moment there is no way to edit the visibility tags yet, but in the hierarchy viewer you can see which tag is added to a node. I guess for the future I should add some functionality in the preview that allows users to switch the visibility tags on or off.

Batch convert changes

In the next development release a couple of changes have been made in the batch convert wizard of ModelConverterX. The main changes are:

  • You can now select if you want to save objects or scenery. When you select objects each model in the selected files is saved as a separate file. If you select scenery, one output file will be made for each input file.
  • There are two button to save and load the batch convert settings. This will save the files you have selected to process, the operators and their settings and the output settings. So this should make it easier to run the same or similar batch runs in the future.

I have added some new batch operators as well, this is an overview of them:

  • Add placement: adds a placement (latitude/longitude) to all objects
  • Center object: makes sure the reference point is in the centre of the object. If the object has a placement, this placement is also updated to match the shift of the object.
  • Rename object: renames all objects to the given name, if multiple objects in one file they are extended with a number
  • Rename textures: rename all textures of an object. They are renamed with the object name, followed by a number

I plan to write another blog post soon explaining how I used these new operators, so that it makes more sense why I added them.

Importing 3DS files

I have just added a new importer to ModelConverterX. From the next development release you will also be able to import files in the 3DS format. Below is a quick overview of what is supported and what the limitations are:

  • Geometry and texture mapping is read
  • The 3DS format does not store the normals of the object, so after import they might be slightly different. Although the library I use to import has a feature to reconstruct the normals for smooth objects. That seems to work quite well.
  • Animations are supported. But the special FSX tagging for animations is stored as a user property and those are not written to the 3DS file. So that information can not be imported either.
  • I have tried to map the different material settings that the 3DS format supports to the FSX material as well as possible. But the 3DS file can not store the FSX specific settings.
  • Another limitation of the 3DS format is that texture names are stored in the old DOS way. So that means only 8.3 characters are available. Any longer texture name will be truncated. So when importing from 3DS you might want to double check your texture names.

By implementing this new reader and especially the animation part, I have learned a lot about how animations are stored in the 3DS format. So my next update will be for the 3DS export functionality of ModelConverterX. I want to add support to export animations to that as well.

7 days, 7 tools

7-Tools-I-use4

Most of you know that I make all those flight simulation development tools for fun, it is my hobby. This means I have no tight schedule for when I work on certain features, I just work on the tool or the feature that I most enjoy at the moment.

So that means that sometimes I jump around between development of different tools quite a bit. Usually this is a lot of fun, but sometimes the distraction means that a certain feature does not get finished as quickly as I would like.

Most of the time I don’t even notice that I work like this. But about a week of two ago I noticed that I had worked on about 7 different tools or projects in one week. Just for the fun of it, let me share with you how my distraction can work during a week… (and luckily for myself it is not always like this, that would probably drive me crazy :))

  1. Sunday, triggered by some discussion on the forum about the ModelConverterX object report I decide to start working on a new feature that should make it easier to save screenshots in any resolution and without rendering the object on screen first. This feature is on my wishlist for a while already and it gives me the chance to try some new OpenGL functions as well.
  2. Monday, while browsing through some geo data on the internet I find interesting vector data that contains population density data of the Netherlands. This is exactly the kind of data I have been looking for, since it will allow me to let scenProc set the distribution of the building heights based on this data. So I quickly jump to scenProc to implement this new feature.
  3. Tuesday, still working on the new scenProc feature. I actually got it implemented and working. Great!
  4. Wednesday, while browsing the same website as Monday I also find some new elevation data for the Netherlands. For the NL2000 project we have been looking for better elevation data for quite a while already. Using GDAL and resample and I put the data into FSX quite quickly. Only problem is that the data has not been filtered for the cities, so that buildings are visible in the elevation data.
  5. Thursday, I remember that I have some vector data laying around that has contour lines for the Netherlands. Using GDAL (once again) I create raster elevation data from these contour lines. I then use this data to replace the data I found on Wednesday in the city areas. Luckily some vector data was supplied with the elevation data that tells which data has not been filtered. It is really amazing what you can do with gdal_rasterize and gdal_calc! So I have much better elevation data now.
    Within the NL2000 project we get questions quite often if we can make a version for X-Plane as well. Triggered by one of those discussions again I decide to check how the tool that converts FS scenery to X-Plane works. ModelConverterX can already read X-Plane OBJ files and adding support to write them is on my wishlist for a while already. But to make a useful conversion I should figure out how the placement of objects works in X-Plane. This is also something I was planing to do for a while already. So let’s print the X-Plane DSF file format description and read it before going to sleep.
  6. Friday, the DSF file format is well documented and should not be that difficult to read. I am looking forward to experiment with reading and writing that format a bit. But to be able if I read the format correctly it would be best to be able to display the objects defined in it on a map. That’s exactly what the ObPlacerX² tool I made a prototype of last year does. So let’s dust off that prototype again!
  7. Saturday, to get the ObPlacerX² tool working with the most recent version of my other tools, I would need to make some changes to the way that object placements are represented internally. This is also something that I have been thinking about before, as it would also make ModelConverterX more efficient and better structured. So let’s start and make those changes first.

7-Tools-I-use4 (1)

Recent changes

Over the last weekend I made a couple of changes to the development releases of my tools. Here is a quick overview of the changes:

  • Last Friday I introduce the big changes to ModelConverterX that allow multiple object placements. Unfortunately some bugs slipped into these changes. These caused crashes while loading certain BGL files and also the drawcall minimizer was not working correctly anymore. All of these bugs should be fixed by tomorrows development release. But let me know if you find some new issues.
  • Last week I added an extra filter to the scenProc step that sets the height of the buildings in a tile. Now a similar filter has been added to the steps that set the building texture, row house texture and vegetation settings. This means you can now set them differently for each tile.
  • There was a bug in scenProc as well for the way it wrote library objects to autogen files. Due to this bug these library objects did not show correctly in Annotator and that also meant that when saving in Annotator they would be lost. This bug has been fixed now. So if you made autogen files with library objects in a prior scenProc version it is highly recommended that you recreate them with the latest build.
  • Last change is that the ground polygon wizard of ModelConverterX now has two additional checkboxes that allow you to select and deselect all textures at once to enable night textures and/or seasonal textures. Especially for models with many textures this should save quite some mouse clicks.

Multiple object placements

The next development release of ModelConverterX contains a number of changes related to object placement.  The main change is that you can now place multiple instances of the same object from within ModelConverterX.

The placement is no longer done from the Object Information form, instead a new Object Placement form has been added. You still have the familiar reference map that can show maps from different providers.

Another big change is that there are two export buttons now. One to export objects and one to export scenery. Let me explain the differences.

  • Export object will export only the selected object to the format of your choice. You can choose any of the supported 3D object formats. No placement information is exported in this case.
  • Export scenery will export all loaded objects and their placement information at once. For the moment the supported formats are FSX BGL, FS2004 BGL and BGLComp XML.

I have made a video tutorial explaining the changes.

On the surface this might seem a relatively small change, but underwater I had to make quite some changes to make everything more flexible and support multiple placements. I have tried to test all features, but if you have trouble with the latest development release please let me know.

With the changes I have made now it will become quite easy to add support to scenProc to export XML object placement as well. So that is something I will be working on soon. And these changes also allow me to finish the ObPlacerX² object placement tool that I have been making a prototype of secretly :).

Creating elevation data

This week I did some work on creating elevation data for our Dutch NL2000 scenery. I would like to share the challenges and solutions I encountered.

For the NL2000 scenery we have been looking for replacement elevation data for quite a while already. The default FSX elevation data is OK, but it is not really detailed. Although the Netherlands is not mountainous, it would be nice to see more detail in the hills and dunes.

Our first challenge was to find source elevation data that we could use. Given that it is a freeware project we we looking for free or contributed data sources. Just this week I found at that the Dutch government has made 25 meter resolution elevation data available for free. So this sounded like good news. However an examining the data a bit more, it turned out that certain parts of the data were not filtered. This means that the data does not only contain the terrain there, but also elevation points that belong to buildings and other structures. When I tried this elevation data in FSX, especially the cities turned out to have a lot of noise in the data.

So this did not look good. But fortunately a vector file was provided that contained the areas were the data had not been filtered. So using this vector data it should be possible to replace the elevation data in those areas with another source. But where to find another source?

In the end I decided to use the vector data set of the Netherlands that the government has also made available. This vector data set does contain contour lines and elevation points. So how to turn this data into raster data so I could merge it with the other set?

In the end it turned out that with GDAL this was not so hard to do. Using gdal_rasterize I could make a raster burning in the elevation data of the contour lines and elevation points. And with gdal_fillnodata I could fill the parts in between. Here are the scripts I used:

gdal_rasterize -a hoogte -a_nodata -32768 -ts 10000 10000 IsoHoogte.shp test.tif
gdal_rasterize -a hoogte HoogteOfDieptePunt.shp test.tif
gdalwarp -t_srs "+proj=latlong +datum=WGS84" -srcnodata -32768 -dstnodata -32768 test.tif test_wgs84.tif
gdal_fillnodata test_wgs84.tif test_wgs84_fill.tif

The results of gdal_fillnodata are not perfect, sometimes you can see some artifacts in the elevation data. But since I mainly wanted to use this data to replace elevation data in cities, it was not a big issue. Most cities in the Netherlands are relatively flat.

So how did I merge the two? With GDAL again. First I burned a big negetive value into the elevation data based on the vectors of the cities. And then I replaced those parts with the other dataset. Here is the script again:

gdal_rasterize -b 1 -burn -1000000 stadspolygonen.shp ahn_stad.tif
gdal_calc -A ahn_stad_wgs84.tif -B ..\top10\test_wgs84_fill.tif --outfile ahn_merge.tif --calc '(A==-1000000.0)*B + (A!=-1000000.0)*A'

The final GeoTIFF could be put in FSX very easily with resample. So that resulted in nicer elevation data for the Netherlands. As NL2000 team we are still testing the new data, but it might end up in a future update of the scenery.

ElevationStPietersberg

Two scenProc changes

In the next development release there are two important changes in scenProc. These will mean that your old configuration file will no longer work directly, you will have to make some small modifications to them.

The first change is that the SHP and KML importer have an additional argument. Just like the OSM importer you can now specify a bounding box (minimum and maximum longitude and latitude) and only the features that are within this bounding box will be loaded from the file. Other features are skipped. If you just want all the features in the file you specify a * for the bounding box coordinates.

This can be useful if you only want to import part of a file. Dropping the features at import means you don’t have to process them with the other steps as well, so that keeps the processing speed high.

The second change is related with the post I made yesterday. The SETAGNBUILDINGHEIGHT step now takes an additional argument as well. This is a filter and only if the centre of the autogen tile is within one of the polygons specified by the filter the building heights will be updated to the provided values. This means you can use vector data to vary the building height distribution. Once again * means that the height values will be set on all tiles.

So how does this work? Below is a sample configuration file where I load buildings from a OSM file and I load a SHP file that contains statistical data that also includes information about the population density for each area. Most of the steps are ones that you are familiar with by now. But in SETAGNBUILDINGHEIGHT you see that I use a filter. First I set all autogen tiles to only use low buildings. Then I update the tiles that fall within polygons where the population density is greater than 2500 to have bigger buildings. And finally I assign tall buildings to areas with a density of over 5000.

The big question of course is, does it work? I guess the answer to that is yes and no. Yes, you see more taller buildings in the city centres with those settings. But no, it does not yet realistically. Why, I see a number of reasons for that.

  1. The density values I used to select the different heights might not be realistically. It would probably need some fine tuning to find out what is a good density for more taller buildings.
  2. The actual buidlings that FSX makes also depend on the size of the footprint. So a small footprint will still get a small buidling. This is realistic of course, but that means you might not always see the height distribution you specified.
  3. I probably did not use the right building GUID. I had gabled buidlings and these look funny when they get bigger. You just see a very big roof, while you would expect a taller buidling. So a lesson from that is that some more work is needed to select the right buidling type for each area.

So enough food for thought and more experimentation.

IMPORTOSM|adam.osm|*|building
IMPORTSHP|..\cbs\CBSvierkant500m201207.shp|4.4;5.1;52.28;52.54|+proj=sterea +lat_0=52.15616055555555 +lon_0=5.38763888888889 +k=0.999908 +x_0=155000 +y_0=463000 +ellps=bessel +units=m +towgs84=565.04,49.91,465.84,-0.409394387,0.35970519561,-1.868491,4.0772 +no_defs
#
SPLITGRID|AGN|building=*
#
SETAGNBUILDINGHEIGHT|*|1.0;0.0;0.0;0.0
SETAGNBUILDINGHEIGHT|FROMFILE=CBSvierkant500m201207.shp;OAD2011>2500|0.0;0.5;0.5;0.0
SETAGNBUILDINGHEIGHT|FROMFILE=CBSvierkant500m201207.shp;OAD2011>5000|0.0;0.0;0.0;1.0
#
CREATEAGNGENBUILD|FTYPE=POLYGON;building=*|{5ae04eb6-934c-4f63-bb48-5e7dee601212}
#
WRITEAGNFILES|FSX|C:\flightsim\nl2000\nl2k_v4_resample_NH\texture