Is it easy to waste performance?

Let me start with the answer to the question I posed in the title: YES.

Earlier this week I introduced a scenProc update that could handles holes in shapefile polygons. This new features used the functions I have to do boolean operations on polygons. Things like intersect or subtract two polygons. This functionality is also used in the ground polygon wizard of ModelConverterX or in the SPLITGRID step of scenProc. Initially I noticed that loading the shapefiles with holes took quite a bit longer than loaded the shapefiles before.

So time to have a look where that extra time was spend. Not really surprisingly it was inside my boolean functions. After I used a profiler to see in which functions most time was spend (it is always in the functions that you don’t expect), I was able to optimize the algorithm quite a bit more.

So compared to the beginning of the week, my boolean functions have become at least 6 times quicker. That will not only be useful when importing shapefiles with holes, but also when performing the SPLITGRID step. So I hope you enjoy the extra speed.

Looking for presenters

As FSDeveloper we are trying to organize an online conference for FS developers, with the aim to share techniques, knowledge and just have fun. We are now one step closer to organizing this conference, as we have our call for presenters ready. So do you have a topic that you would like to talk about for other developers, let us know!

Features with holes

Holes are often used in shapefiles, for example to mark an island in an water polygon or to mark a clearing in a forest polygon. Until now scenProc did not support holes, so they were ignored when importing shapefiles. As a result of this the autogen generated would not be correct.

The scenProc code that handles the splitting of features into the autogen grid does not know about holes and also the autogen polygons in the AGN file don’t know the concept of holes. So therefore I have added functionality to the SHP importer to remove them. Instead a cut is made in the polygon, so that the entire shape can still be represented as one polygon. After that they can be processed like any other shapefile.

Interestingly you can have island within holes again (and that island can have a hole of itself). scenProc should be able to handle such situations as well, as you can see in the picture below. Although I haven’t tested it to the extreme.

The new functionality will come at a small performance cost. When importing features with holes scenProc has to do additional work to correctly represent them. So for complex files this can increase the loading time a bit. But of course you get support for the holes in return.

If you still have a file with holes that doesn’t work correctly please let me know.

holes

Be careful for narrow houses

narrow house 6Two users recently reported the same problem to me with autogen files made with scenProc. In Annotator they could not see all the buildings and after saving in Annotator many of the buildings were gone.

It turned out this problem was caused by very narrow buildings. So I want to warn all scenProc users of this problem. There are two causes for narrow buildings:

  1. Your data already contains them
  2. The SPLITGRID step made them

For the first case there is not much I can do, so just check your data. But to be honest the second case is most likely the cause of this issue in most cases. Normally the SPLITGRID step does make sure all features are divided along the borders of the autogen tiles. So if a polygon crosses the border it is split into two polygons.

Now what would happen if you house just crossed the border a tiny little bit? That would result in that tiny bit being sliced away from the house and being represented as a separate polygon (and thus seen as another house). This can result in very narrow houses in some cases.

The obvious solution for this is to not slice the houses, but instead just put the entire footprint polygon of the house in the terrain tile where the centre is. This option is already possible with scenProc, it is the second argument you can give the step.

Let me show you two examples of how to use it. The first case assumes that you load all your building footprints from a separate file, let’s assume it is called buildings.shp. In that case you can tell scenProc not to slice the buildings by using the following command:

SPLITGRID|AGN|FROMFILE=buildings.shp

Another case would be where you loaded OpenStreetMap data. In that case the buildings are not in a separate file. So you would need to filter on some attribute. For example:

SPLITGRID|AGN|building=*

Tree detection – part 4

It’s a while ago that I posted about the progress of my tree detection efforts. So here is a quick update. Some other things kept me busy, so there hasn’t been too much progress recently.

I was able to run the tree detection algorithm on the entire island of Nantucket now. It took around 3.5 hours. Below you see a screenshot of the result. I have to be honest that I took the screenshot from a good angle. There are areas where there are quite a few false detections, which results in trees on locations where they should not be. Mainly in the water. So my efforts will be focussed on finding a way to eliminate those.

In one of my earlier posts I mentioned that polygonal autogen vegetation did not work well, since trees not always show up. But that was wrong. I ended up added a buffer to my detected areas to make them a few pixels bigger and that was enough to get the trees displaying. I now I conclude that the polygonal vegetation is probably still the best way to go.

Image2013-06-16 1715.10.962

SPLITGRID? Check!

check-markIn my bug reporting system I can see that it relatively often happens that people run into trouble (crashes) because they remove the SPLITGRID step from their configuration file. If you then try to create autogen features scenProc will crash. This is because I assumed that the SPLITGRID step will be done first. This step will make sure all features are divided correctly in the terrain grid used by FS.

So to prevent such crashes from happening, I have now added an extra check to the validation of your configuration file. If the SPLITGRID step is not present before using the steps to create the autogen you will get an error.

I hope this will help to prevent this common mistake from happening.

I flipped it

Yesterday I fixed a bug in the scenProc step that imports AGN files. There was an issue in this step that resulted in part of the autogen features being displayed wrong. They were vertically flipped.

So if you are using the import AGN step in your workflow, please update to the latest version and run your configuration file again. Else you can have wrong results.

BGLComp SceneryObject flags

A few days ago I added support to ModelConverterX to read the additional options like <NoShadow /> or <NoAutogenSuppresion /> from the object placement code in the BGL file. Initially I had some trouble to find out where these options ended up in the binary code, since it was not documented in the otherwise excelent description that Winfried Orthmann made a while ago.

After the latitude, longtide and altitude the description says there is a short that stores if the altitude is AGL or not. But actually this short can store much more than that. It more acts like a set of flags for multiple options. Here is the list of possible flags:

0x01 AltIsAgl
0x02 NoAutogenSuppression
0x04 NoCrash
0x08 NoFog
0x10 NoShadow
0x20 NoZWrite
0x40 NoZTest
Just wanted to share this, in case other tool developers are struggling with the same in the future.

Tree detection – part 3

I have made some more progress on the tree detection algorithm in the last days, but I am not there yet. So some more testing is needed before the functionality can be put in scenProc.

In my previous post I showed some results from the algorithm running in a test application. What I have done now is put it in scenProc itself. This means that scenProc can now read raster files through GDAL, run the histogram based filtering on the image and create vectors from the results (thanks to GDAL and OGR again).

Here are my first findings of putting the algorithm in scenProc:

  • It is still very slow. Running the whole island of Nantucket through the tree detection algorithm takes more than half a day. So I guess it would be a good moment to get the profiler tool again and find out where performance gains can be made.
  • The screenshots I showed last time were from images at a lower resolution. I had down sided them from the 30 cm resolution to make testing easier. It seems the filter that worked well on those, does not necessary work well on the full resolution as well. It seems resizing them made the trees slightly darker. So I will have to do some more testing to see which filter settings work well with the full resolution.
  • With the algorithm you get a lot of relatively small polygons. Most polygons cover a few trees. Only in a dense forest you would get bigger polygons. Creating polygonal autogen vegetation from these small polygons does not work well, quite often no trees show at all in FSX. Using rectangular autogen vegetation seems to work much better. Even for a small rectangle at least one tree will show up. So it seems rectangular vegetation would be the way to go with this algorithm.

And now back to some more testing…

Tree detection – part 2

I am travelling for my work this week, but when I had an few hours in the evening to spend in my hotel room I did some more experimentation with the tree detection algorithm.

I have now modified my test tool so that you can click on the trees in the image to determine the rules that are used for the detection. Just click on a dozen different trees or so, and the tool will figure out which rule should be used to find the trees in this kind of image.

From the tests I have done it seems to work quite well. Below are two screenshots of the Nantucket imagery where I used to same rules to find the trees in both images. I am quite happy with the result till now.

tree_detect1

tree_detect2

Next step will be to add this algorithm to scenProc, so that I can test it on a bigger area and see if the results still look convincing. I am curious to see how this autogen will turn out. But that will have to wait till I am back home, I don’t have FSX here at the moment.