7 days, 7 tools

7-Tools-I-use4

Most of you know that I make all those flight simulation development tools for fun, it is my hobby. This means I have no tight schedule for when I work on certain features, I just work on the tool or the feature that I most enjoy at the moment.

So that means that sometimes I jump around between development of different tools quite a bit. Usually this is a lot of fun, but sometimes the distraction means that a certain feature does not get finished as quickly as I would like.

Most of the time I don’t even notice that I work like this. But about a week of two ago I noticed that I had worked on about 7 different tools or projects in one week. Just for the fun of it, let me share with you how my distraction can work during a week… (and luckily for myself it is not always like this, that would probably drive me crazy :))

  1. Sunday, triggered by some discussion on the forum about the ModelConverterX object report I decide to start working on a new feature that should make it easier to save screenshots in any resolution and without rendering the object on screen first. This feature is on my wishlist for a while already and it gives me the chance to try some new OpenGL functions as well.
  2. Monday, while browsing through some geo data on the internet I find interesting vector data that contains population density data of the Netherlands. This is exactly the kind of data I have been looking for, since it will allow me to let scenProc set the distribution of the building heights based on this data. So I quickly jump to scenProc to implement this new feature.
  3. Tuesday, still working on the new scenProc feature. I actually got it implemented and working. Great!
  4. Wednesday, while browsing the same website as Monday I also find some new elevation data for the Netherlands. For the NL2000 project we have been looking for better elevation data for quite a while already. Using GDAL and resample and I put the data into FSX quite quickly. Only problem is that the data has not been filtered for the cities, so that buildings are visible in the elevation data.
  5. Thursday, I remember that I have some vector data laying around that has contour lines for the Netherlands. Using GDAL (once again) I create raster elevation data from these contour lines. I then use this data to replace the data I found on Wednesday in the city areas. Luckily some vector data was supplied with the elevation data that tells which data has not been filtered. It is really amazing what you can do with gdal_rasterize and gdal_calc! So I have much better elevation data now.
    Within the NL2000 project we get questions quite often if we can make a version for X-Plane as well. Triggered by one of those discussions again I decide to check how the tool that converts FS scenery to X-Plane works. ModelConverterX can already read X-Plane OBJ files and adding support to write them is on my wishlist for a while already. But to make a useful conversion I should figure out how the placement of objects works in X-Plane. This is also something I was planing to do for a while already. So let’s print the X-Plane DSF file format description and read it before going to sleep.
  6. Friday, the DSF file format is well documented and should not be that difficult to read. I am looking forward to experiment with reading and writing that format a bit. But to be able if I read the format correctly it would be best to be able to display the objects defined in it on a map. That’s exactly what the ObPlacerX² tool I made a prototype of last year does. So let’s dust off that prototype again!
  7. Saturday, to get the ObPlacerX² tool working with the most recent version of my other tools, I would need to make some changes to the way that object placements are represented internally. This is also something that I have been thinking about before, as it would also make ModelConverterX more efficient and better structured. So let’s start and make those changes first.

7-Tools-I-use4 (1)

Recent changes

Over the last weekend I made a couple of changes to the development releases of my tools. Here is a quick overview of the changes:

  • Last Friday I introduce the big changes to ModelConverterX that allow multiple object placements. Unfortunately some bugs slipped into these changes. These caused crashes while loading certain BGL files and also the drawcall minimizer was not working correctly anymore. All of these bugs should be fixed by tomorrows development release. But let me know if you find some new issues.
  • Last week I added an extra filter to the scenProc step that sets the height of the buildings in a tile. Now a similar filter has been added to the steps that set the building texture, row house texture and vegetation settings. This means you can now set them differently for each tile.
  • There was a bug in scenProc as well for the way it wrote library objects to autogen files. Due to this bug these library objects did not show correctly in Annotator and that also meant that when saving in Annotator they would be lost. This bug has been fixed now. So if you made autogen files with library objects in a prior scenProc version it is highly recommended that you recreate them with the latest build.
  • Last change is that the ground polygon wizard of ModelConverterX now has two additional checkboxes that allow you to select and deselect all textures at once to enable night textures and/or seasonal textures. Especially for models with many textures this should save quite some mouse clicks.

Multiple object placements

The next development release of ModelConverterX contains a number of changes related to object placement.  The main change is that you can now place multiple instances of the same object from within ModelConverterX.

The placement is no longer done from the Object Information form, instead a new Object Placement form has been added. You still have the familiar reference map that can show maps from different providers.

Another big change is that there are two export buttons now. One to export objects and one to export scenery. Let me explain the differences.

  • Export object will export only the selected object to the format of your choice. You can choose any of the supported 3D object formats. No placement information is exported in this case.
  • Export scenery will export all loaded objects and their placement information at once. For the moment the supported formats are FSX BGL, FS2004 BGL and BGLComp XML.

I have made a video tutorial explaining the changes.

On the surface this might seem a relatively small change, but underwater I had to make quite some changes to make everything more flexible and support multiple placements. I have tried to test all features, but if you have trouble with the latest development release please let me know.

With the changes I have made now it will become quite easy to add support to scenProc to export XML object placement as well. So that is something I will be working on soon. And these changes also allow me to finish the ObPlacerX² object placement tool that I have been making a prototype of secretly :).

Creating elevation data

This week I did some work on creating elevation data for our Dutch NL2000 scenery. I would like to share the challenges and solutions I encountered.

For the NL2000 scenery we have been looking for replacement elevation data for quite a while already. The default FSX elevation data is OK, but it is not really detailed. Although the Netherlands is not mountainous, it would be nice to see more detail in the hills and dunes.

Our first challenge was to find source elevation data that we could use. Given that it is a freeware project we we looking for free or contributed data sources. Just this week I found at that the Dutch government has made 25 meter resolution elevation data available for free. So this sounded like good news. However an examining the data a bit more, it turned out that certain parts of the data were not filtered. This means that the data does not only contain the terrain there, but also elevation points that belong to buildings and other structures. When I tried this elevation data in FSX, especially the cities turned out to have a lot of noise in the data.

So this did not look good. But fortunately a vector file was provided that contained the areas were the data had not been filtered. So using this vector data it should be possible to replace the elevation data in those areas with another source. But where to find another source?

In the end I decided to use the vector data set of the Netherlands that the government has also made available. This vector data set does contain contour lines and elevation points. So how to turn this data into raster data so I could merge it with the other set?

In the end it turned out that with GDAL this was not so hard to do. Using gdal_rasterize I could make a raster burning in the elevation data of the contour lines and elevation points. And with gdal_fillnodata I could fill the parts in between. Here are the scripts I used:

gdal_rasterize -a hoogte -a_nodata -32768 -ts 10000 10000 IsoHoogte.shp test.tif
gdal_rasterize -a hoogte HoogteOfDieptePunt.shp test.tif
gdalwarp -t_srs "+proj=latlong +datum=WGS84" -srcnodata -32768 -dstnodata -32768 test.tif test_wgs84.tif
gdal_fillnodata test_wgs84.tif test_wgs84_fill.tif

The results of gdal_fillnodata are not perfect, sometimes you can see some artifacts in the elevation data. But since I mainly wanted to use this data to replace elevation data in cities, it was not a big issue. Most cities in the Netherlands are relatively flat.

So how did I merge the two? With GDAL again. First I burned a big negetive value into the elevation data based on the vectors of the cities. And then I replaced those parts with the other dataset. Here is the script again:

gdal_rasterize -b 1 -burn -1000000 stadspolygonen.shp ahn_stad.tif
gdal_calc -A ahn_stad_wgs84.tif -B ..\top10\test_wgs84_fill.tif --outfile ahn_merge.tif --calc '(A==-1000000.0)*B + (A!=-1000000.0)*A'

The final GeoTIFF could be put in FSX very easily with resample. So that resulted in nicer elevation data for the Netherlands. As NL2000 team we are still testing the new data, but it might end up in a future update of the scenery.

ElevationStPietersberg

Two scenProc changes

In the next development release there are two important changes in scenProc. These will mean that your old configuration file will no longer work directly, you will have to make some small modifications to them.

The first change is that the SHP and KML importer have an additional argument. Just like the OSM importer you can now specify a bounding box (minimum and maximum longitude and latitude) and only the features that are within this bounding box will be loaded from the file. Other features are skipped. If you just want all the features in the file you specify a * for the bounding box coordinates.

This can be useful if you only want to import part of a file. Dropping the features at import means you don’t have to process them with the other steps as well, so that keeps the processing speed high.

The second change is related with the post I made yesterday. The SETAGNBUILDINGHEIGHT step now takes an additional argument as well. This is a filter and only if the centre of the autogen tile is within one of the polygons specified by the filter the building heights will be updated to the provided values. This means you can use vector data to vary the building height distribution. Once again * means that the height values will be set on all tiles.

So how does this work? Below is a sample configuration file where I load buildings from a OSM file and I load a SHP file that contains statistical data that also includes information about the population density for each area. Most of the steps are ones that you are familiar with by now. But in SETAGNBUILDINGHEIGHT you see that I use a filter. First I set all autogen tiles to only use low buildings. Then I update the tiles that fall within polygons where the population density is greater than 2500 to have bigger buildings. And finally I assign tall buildings to areas with a density of over 5000.

The big question of course is, does it work? I guess the answer to that is yes and no. Yes, you see more taller buildings in the city centres with those settings. But no, it does not yet realistically. Why, I see a number of reasons for that.

  1. The density values I used to select the different heights might not be realistically. It would probably need some fine tuning to find out what is a good density for more taller buildings.
  2. The actual buidlings that FSX makes also depend on the size of the footprint. So a small footprint will still get a small buidling. This is realistic of course, but that means you might not always see the height distribution you specified.
  3. I probably did not use the right building GUID. I had gabled buidlings and these look funny when they get bigger. You just see a very big roof, while you would expect a taller buidling. So a lesson from that is that some more work is needed to select the right buidling type for each area.

So enough food for thought and more experimentation.

IMPORTOSM|adam.osm|*|building
IMPORTSHP|..\cbs\CBSvierkant500m201207.shp|4.4;5.1;52.28;52.54|+proj=sterea +lat_0=52.15616055555555 +lon_0=5.38763888888889 +k=0.999908 +x_0=155000 +y_0=463000 +ellps=bessel +units=m +towgs84=565.04,49.91,465.84,-0.409394387,0.35970519561,-1.868491,4.0772 +no_defs
#
SPLITGRID|AGN|building=*
#
SETAGNBUILDINGHEIGHT|*|1.0;0.0;0.0;0.0
SETAGNBUILDINGHEIGHT|FROMFILE=CBSvierkant500m201207.shp;OAD2011>2500|0.0;0.5;0.5;0.0
SETAGNBUILDINGHEIGHT|FROMFILE=CBSvierkant500m201207.shp;OAD2011>5000|0.0;0.0;0.0;1.0
#
CREATEAGNGENBUILD|FTYPE=POLYGON;building=*|{5ae04eb6-934c-4f63-bb48-5e7dee601212}
#
WRITEAGNFILES|FSX|C:\flightsim\nl2000\nl2k_v4_resample_NH\texture

How to vary autogen building height?

One of the items still on my scenProc wishlist is adding a feature that allows you to vary the heights of the autogen buildings  In the current version you have to set the same height distribution for all autogen tiles. But how to do that? I started a thread on the forum a while ago to discuss the possibilities and at the moment I see a couple of ways this can be done.

  1. First option is to let the user draw some polygons manually where he wants more hi-rise buildings. All areas not covered by these polygons would get the low buildings only. Since KML file can now be read, these polygons could even be drawn in Google Earth quickly.
  2. vector_pop_densityA second option is to find and use some vector data that contains information about population density, amount of houses in an area or something similar. For the Netherlands I found some (free) data from the national statistics bureau that gives information about amount of people, population density and even amount of houses for the entire country. This information is given for cells of 500×500 meter. To the right is a screenshot of this data.
  3. raster_pop_densityA third option would be to use a raster image with population density data. I did not yet find a good source for this however. I found some population density data at the EEA website that covers most of Europe. But it only showed the population density per municipality. Which means that the city centre and the suburbs have the same density. While the aim is to vary the autogen between those parts of town of course. I have included a screenshot of this data on the right again. But there might be better data available and then using raster data would be a good idea.

So with those 3 ideas in mind, how am I going to implement it in scenProc? In the end it all comes down to the same idea. Taking the centre coordinate of the autogen tile the available data should be checked for the population density and than based on the value a selection is made for the distribution of the buildings heights in the autogen. So let’s see if I can get that coded…

Reading KML files

kml-685I have added support for another file format to scenProc, it is now also possible to read KML files. This means that you can now use Google Earth as well to draw your points, lines and polygons. And then use scenProc to convert this data into autogen for FSX.

I have only tested with some KML files made with Google Earth and QGIS. So if you have KML files made with another tool and have problems with importing them, please let me know.

Down time

Just a quick reminder that this website will be offline for an hour or so tomorrow morning. This means that also the development releases of my tools are not accessible for that time.

As some of you might know the development releases and this site our hosted on a small server at my home. And tomorrow I will be making some small adjustments to the electrical wiring in the house to make it safer for our baby that can almost crawl now. So therefore the server will be offline for a while.

Auto completion for attributes

From the next development release scenProc has a new feature, it is now possible to get auto completion of possible attributes and their values. All you have to do is scan the file you will be processing by pressing the “Scan attributes” button. Be aware that this might take a while for big files.

After that you will see that the auto completion when constructing a filter also shows the attributes and their possible values (as found in the OSM or Shapefile). This makes it a lot easier to construct the filters. And it is also a great way to explore which possible values there are for you to use.

Below is a quite video I made to demonstrate the feature.