This work sheet is intended for professionals in surveying and geodesy as well for those with an interest in coordinate transformations and reference system transitions.
Basic geodetic knowledge is assumed.
NTv2 files are usually modeled by external companies with the required expertise and equipment. Experience shows that the necessary software and personal costs are very expensive. As a result, many administrations, surveying and cadastre offices are hesitant to take the step of creating a highly accurate NTv2 file from their existing base data. But this could be useful for a wide range of surveying tasks where high accuracy is required. A method is presented on how to model an NTv2 file even with limited budget and personal resources. The software used in this example costs less than 1000 US Dollars in total.
The German Independent City of Wuppertal has published on its website base point files with high-precision identical points of the urban area in the coordinate reference systems DHDN/Gauß-Krüger and ETRS89/UTM. The base point files are available there for downloading and free use within the scope of copyright. From these data the City of Wuppertal has developed a set of Helmert transformation parameters for the transition from the reference system DHDN to ETRS89, but not an equivalent NTv2 file. Later we will come back to this Helmert set in context of the quality analysis.
This article explains the creation of an NTv2 file using the geodetic program NTv2Creator from KilletSoft for the even more accurate transition from DHDN to ETRS89 in the urban area. Just as for the creation of the Helmert transformation parameters, the same base point files of the City of Wuppertal are used for the creation of the NTv2 file.
Reference system transition with the NTv2 method
NTv2 means "National Transformation Version 2" and describes a method for the transfer of georeferenced data from one Reference System to another. Originally developed in Canada, it has grown to a worldwide standard.
The NTv2 method is based on a regular grid of geographic coordinates for which shift values are stored in a file as the difference between two reference systems. For a coordinate transformation between the Reference Systems, the grid mesh first is determined, in which a point to be transformed lies. From the shift values of the four grid points in the corners of this mesh the point lying within the grid mesh is interpolated. The interpolated shift values of the longitude and latitude are then added to the coordinate of the source reference system to obtain an equivalent coordinate in the target reference system.
Modeling an NTv2 file
So much about using already existing NTv2 files. But how is an NTv2 file created from a cloud of irregularly distributed geodetic points in two base point files? First of all, all points of the base point field must be available as so-called identical points, which have been precisely measured and calculated in two different reference systems. From the identical points, the shift values of the grid points for a defined working area are calculated in a one-time process and stored in an NTv2 file. Since modeling an NTv2 file is a very time-consuming process, high computational time expenditure is to be expected depending on the size of the working area, the number of identical points in the base point files and the fineness of the grid meshes.
As a result, the program NTv2Creator creates two variants of NTv2 files. One variant is a file in readable ASCII format, which has the file name suffix gsa. The other variant is a binary NTv2 file with the file name suffix gsb, which can be used directly by coordinate transformation software and GIS and allows a high execution speed.
To create an NTv2 file, first some characteristics concerning the dimensions of the grid, the mesh size, the ellipsoids of the source and target reference systems and some designations for the reference systems and for the grid have to be defined. The program inserts these data into a main header and into a grid header of the NTv2 file. In the ASCII version this can be easily viewed in plain text. Following the headers, the shift values of the grid points and quality characteristics are written. For each grid point there are four decimal numbers. These are the shift values for the geographic latitude and longitude of the grid points and the quality attributes for the latitude and longitude of the grid points.
At the beginning of the processing, the coordinates of the identical points are tested for outliers and deviations using statistical methods. The determined outliers can be excluded from further calculation. Subsequently, all grid points are calculated. To calculate a grid point, the program collects the identical points from the grid point files of the source reference system and the target reference system, which are located in a certain radius around the grid point. These identical points are used to perform a small-scale three-dimensional Helmert Seven Parameter Transformation. The Helmert parameters determine during the subsequent coordinate transformation of the grid point in the source reference system the coordinate of the same grid point in the target reference system. To increase the accuracy the Natural Neighbour Interpolation is used for the distribution of the residual gaps. The difference from the coordinates of the grid point in both reference systems is written into the NTv2 file as two shift values. The calculated shift values include the compensation of regional and local tensions and they consider the different dimensions and bearings of the ellipsoids used in both reference systems.
Using an NTv2 file is the better option
The advantage of the NTv2 method is that small-scale transitions of the reference syste,s are stored in many grid meshes over an area. In contrast, when using a Helmert parameter set, a single large-scale transition is used for the entire area. The grid-based small-scale transitions result in significantly greater accuracy. This is especially beneficial when the underlying reference frames are inhomogeneous or highly distorted.
The base point field of the City of Wuppertal
First, the two base point files with identical points of the source and target reference systems have to be downloaded from the website of the City of Wuppertal. The Google search for Wuppertal Bezugssystem ETRS89/UTM leads to a page where under the link "Transformation" the two base point files can be downloaded. The file STP177_final.txt contains Gauss-Krueger coordinates in the reference system DHDN of about 16800 grid points. The file STP489_final.txt contains UTM coordinates in the reference system ETRS89 for the same grid points.
Fig. 1 click! The figure shows the area of the City of Wuppertal. It is taken from the free encyclopedia WIKIPEDIA. The dark areas show the hight-density built-up urban areas. The light areas are predominantly forested and have less build-up development.
Fig. 2 click! This graphic shows the distribution of the identical points of the base point files in the city area of Wuppertal. The graphic has been generated with the program TRANSDATpro as a shape file. It can be observed that the points are distributed reasonably evenly. In the northwestern part, the point density is slightly higher than in the southeastern part. Along the city boundary, the point density is very high because of the many boundary stones there. Some points are obviously located outside the real city boundary.
Preparation of the base point files
Fig. 3 click! With the program NTv2Creator a highly exact NTv2 file shall be created from the base point files. Since the program cannot process Gauss-Krueger coordinates directly, the two files are converted first with the program TRANSDATpro by KilletSoft into geographic coordinates. It is to be noted that the reference systems respectively their ellipsoids are preserved.
In TRANSDATpro therefore first so-called user-defined ellipsoids are set. For the base point file stp177_final.txt with Gauss-Krueger coordinates in the reference system DHDN the Bessel ellipsoid is set as source and target ellipsoid. Equivalently, for the stp489_final.txt grid point file with UTM coordinates in the ETRS89 reference system, the GRS80 ellipsoid must be set as source and target ellipsoid.
Fig. 4 click! In the main window of the program TRANSDATpro the coordinate system Gauss-Krueger is set for the source coordinates. Geographical coordinates in degree notation are set for the target coordinates. For the reference systems, the previously defined user-defined ellipsoids must be set in the source and target systems respectively. Since the source coordinates in the base point file STP177_final.txt are available as Gauss-Krueger coordinates in the reference system DHDN on the Bessel ellipsoid, all coordinates of the output file are given as geographical coordinates in the reference system DHDN on the Bessel ellipsoid.
For the file STP489_final.txt with UTM coordinates in the reference system ETRS89 on the ellipsoid GRS80 equivalent settings are necessary. For this purpose, the GRS80 ellipsoid is set as user-defined ellipsoid on both sides. For the source coordinates the coordinate system UTM is set. All other settings are identical.
Fig. 5 click! In this window of the program TRANSDATpro he input and output files are configured. The input file STP177_final.txt of the City of Wuppertal contains Gauss-Krueger coordinates in the reference system DHDN. It is available as a formatted text file. In it a point number, a right value and a high value are contained line by line formatted in columns. The first character of the easting value of the coordinates is in each line at position 17.
The transformed coordinates are to be written into a formatted text file named geo_dhdn_wuppertal.txt. To get high accuracy the maximum possible number of decimal places is set. To transfer the point number into the file, the program must copy the leading text in front of the coordinates into the output file. The east and north components of the coordinates are exchanged, to output first the latitude, then the longitude of the geographic coordinates.
The input file STP489_final.txt of the City of Wuppertal contains UTM coordinates in the reference system ETRS89. It contains a point number, an easting value and a northing value formatted in columns line by line. For the output file the name geo_etrs_wuppertal.txt has been chosen. All other settings are identical.
After clicking on the "Calculate" button, the two base point files are prepared so far that they can be used to read the identical points into the program NTv2Creator.
The geodetic program NTv2Creator
Fig. 6 click! The program NTv2Creator by KilletSoft creates an NTv2 file from the two base point files. This allows converting any coordinates in the urban area of Wuppertal from the reference system DHDN to the reference system ETRS89 or backwards with high accuracy using suitable transformation software or GIS. But first some settings are necessary in the program NTv2Creator.
The figure shows a general view of the user interface of the program NTv2Creator. In the upper left area, the base point files and the parameters of the reference systems are inserted. In the middle lower area the file name and further parameters for the NTv2 file to be created are filled in. In the right upper area further settings are inserted, which are needed during the calculation of the NTv2 file. These three areas are explained in detail in the following paragraphs.
Defining the base point files and the reference systems
An NTv2 file for the conversion of any coordinates in the urban area of Wuppertal is to be generated. For this the two base point files prepared before with the program TRANSDATpro are entered as data sources in the source reference system and in the target reference system. In the text fields "Designat." can be entered significant designations for the source and target reference systems for later recognition.
Fig. 7 click! The format of the point names and coordinates contained in the base point files is selected from the list box "Format". The list box allows settings of many different configurations of the data stored in base point files. In our base point files, a point designation, a latitude and a longitude are included in order per data record. Consequently, the option "Geographic Coordinates (No,lat,lon)" is selected. The geodetic ellipsoid Bessel (1841) on which the source reference system DHDN is based is selected from the list field "Ellipsoid". For the target reference system ETRS89 the ellipsoid GRS80 (1980) is defined. The respective semi-axes and the flattening are set automatically by the program.
Parameter of the NTv2 file to be created
Here the filename and the directory of the NTv2 file to be created are specified. The program ensures that the file name always ends with the file suffix gsa. This suffix denotes an NTv2 file in readable ASCII format. The output of a binary NTv2 file with the file suffix gsb is set later.
Fig. 8 click! Three models for creating NTv2 files are available in the NTv2Creator program. Detailed descriptions of the models can be found in the help file, which is available after installing the program on the computer. Here the model "Grid-wide transformation" is set for creating the NTv2 file.
In this model, first of all a area-wide Helmert Seven Parameter Set from all identical points of the base point files is generated and stored. During the calculation of the NTv2 file, so-called missing grid points can occur, which are then filled with results from the area-wide Helmert set. This will be explained in more detail in the further course of the text.
Significant short names must be entered in the text fields for the designations of the source and target systems and for the designation of the NTv2 grid, which are transferred to the NTv2 file to be generated as SYSTEM_F, SYSTEM_T and SUB_NAME.
Further settings for creating the NTv2 file
Most of the settings are self-explanatory. Only the options checked in this example are discussed here. The descriptions of all other settings can be found in the help file, which is available after installing the program.
Fig. 9 click! In the outlier test, identical points clearly deviating from the mean value are detected in the base point files. These may have been caused by measurement errors or transfer errors. If outliers have been found, they can be checked and, if necessary, corrected in the base point files. Alternatively, the program offers the possibility to exclude the outliers from the calculation.
In an NTv2 file, there are four entries per grid point: two shift values and two quality values. The quality values vary in numbers between zero and one. A zero indicates grid points that could not be determined and are therefore called missing grid points. In the selected calculation model, the result from a region-wide Helmert transformation is entered for missing grid points. The larger the quality value, the more accurate are the shift values. A one is entered for the highest possible quality.
As mentioned before, during the calculation of the NTv2 file with the identical points found around each grid point, a small-scale Helmert Seven Parameter Set is generated. By back-calculating with these Helmert parameters for all identical points found, the deviations (gaps) from the actual point coordinates can be determined. The gaps in turn, weighted by the squared distance to the grid points, can be used as correction factors for the shift values of the grid points. This procedure is called "Residual Gaps Distribution by Neighbour Interpolation".
First an NTv2 file in readable ASCII format with the file name suffix gsa is calculated. To additionally generate the binary variant with the file name suffix gsb, the corresponding option must be checked. The gsb variant can be used directly by coordinate transformation software and GIS.
When the program has generated the NTv2 binary file (gsb), a quality analysis can be performed. For this purpose, the program converts all identical points of the source system into the target system using the new NTv2 binary file. The target coordinates calculated in this way are compared with the coordinates of the identical points in the target system. The program splits the deviations between the calculated coordinates and the original coordinates of the identical points of the target system into clusters. Badly matching identical points are determined suggested for correction.
Determining the boundaries and mesh sizes
When all settings required by the program have been made, the identical points in the base point files can be analyzed. The analysis is started by clicking the "Analyze point files" button. If there are errors in the base point files, the program will display them and suggest them for correction.
Fig. 10 click! Based on the spatial distribution of the identical points, the program calculates the boundary of the quadrangular effective area of the NTv2 file. The number of identical points is decisive for the default setting of the number of vertical and horizontal grid meshes. The more identical points there are, the more grid meshes can be calculated. From the numbers of vertical and horizontal grid meshes and the southern and northern as well as the western and eastern boundaries, the program calculates the vertical and horizontal mesh sizes.
The southern, northern, western and eastern boundaries of the effective area are entered by the program in the data fields S_LAT, N_LAT, W_LON and E_LON. The vertical and horizontal mesh sizes are entered in the data fields LAT_INC and LON_INC.
Fig. 11 click! It is possible to manually adjust the values calculated by the program. The mesh widths must be corrected first. The program adjusts the boundaries automatically. Afterwards it is possible to correct the boundaries. The changes should be limited to the southern and western boundaries, so that the previously set mesh sizes are not automatically adjusted by the program. It is best to round off both values to a few decimal places.
Here the boundaries and mesh sizes for the area of the City of Wuppertal have already been adjusted manually. Resulting from the adjustments, the program has recalculated the numbers of grid points and grid meshes.
It makes no sense to change the boundaries strongly outward, because there are no identical points in the enlarged parts of the effective area. On the other hand, it is possible to reduce the effective area by moving the boundaries inwards. Also the identical points, which are then outside the effective area, are used by the program for the calculation of the NTv2 file.
The boundaries of the effective area can be displayed on the surface of the Earth using the "Google Earth" button in the program of the same name.
Modeling the NTv2 file
Clicking the "Create NTv2 file" button starts the modeling of the NTv2 file. Depending on the size of the effective area, the number of identical points and the number of grid points, the modeling of the NTv2 file takes some time. Meanwhile, the processing status is continuously displayed. The modeling of the NTv2 file for the area of the City of Wuppertal takes only a few minutes on a modern computer, because here only 16788 identical points have to be processed to 18626 grid points. This is a small task for the program Ntv2Creator. In fact, the program can process up to one million identical points and create up to one million grid points from them.
Fig. 12 click! All settings, processing steps and results are written to a log file during the modeling of an NTv2 file. After each processing step, this can be viewed in the file editor by clicking on the "View log file" button. In this example only some important details from the log file are mentioned.
Since the distribution of identical points over the effective area can be very different, it is recommended to get an optimal result by trying different mesh sizes. When the modeling is completed, the following information can be found in the log file:
A total of 18626 grid points have been calculated.
Of these, 8132 missing grid points have been transformed with
a grid-wide Helmert seven-parameter set.
There are not enough identical points for these grid points
or the identical points around the grid point are inhomogeneous.
A high number of missing grid points can be quite
normal due to missing identical points in the border areas.
In fact, there are too few identical points in the base point field of the City of Wuppertal for the positions of the missing grid points. This is obvious if you look at the distribution of identical points in Figure 2. An NTv2 file always covers a quadrangular effective area. However, the identical points are, with a few exceptions, located within the polygonal city area. Thus, in the peripheral regions of the effective area covered by the NTv2 file, there are no identical points for the small-scale calculations of the grid points there.
The program has created an NTv2 ASCII file with the filename ntv2_wuppertal.gsa and an NTv2 binary file named ntv2_wuppertal.gsb. These files can be used with modern coordinate transformation software and GIS to convert any coordinates in the Wuppertal urban area in either direction from DHDN to ETRS89 and from ETRS89 to DHDN with high accuracy. Here are some excerpts from the NTV2 ASCII file ntv2_wuppertal.gsa. For better readability the excerpts are shown here slightly formatted. The NTv2 ASCII file contains the main header generated by the program:
Directly below follows the grid header generated by the program:
Some of the previously entered values can be found again in the headers, others have been converted into seconds by the program. It is remarkable that the eastern and western longitudes in our example are provided with negative signs. Since the NTv2 format was developed in Canada, which is located west of Greenwich, the normally negative signed longitudes were simply used as positive values in the formula set of the NTv2 method. As a result, the longitudes east of Greenwich are included in the grid header with a negative sign.
Below the two headers the 18626 grid points with two shift values and two quality values per line are written. The shift values represent the shifts of the latitude and longitude in seconds between the two reference systems at the grid point. The quality values are both the same and provide information about the quality of each grid point. The quality is determined from the number of identical points used for the grid point and their distribution and distance from the grid point. The higher the quality value, the more accurate are the shift values. An One is assigned for the highest possible quality. A Zero assigns a missing grid point.
-4.632984 2.850686 1.000000 1.000000
-4.632793 2.849199 1.000000 1.000000
-4.632920 2.848079 1.000000 1.000000
-4.632865 2.847290 1.000000 1.000000
-4.632808 2.846101 0.500000 0.500000
-4.632714 2.845069 1.000000 1.000000
-4.632815 2.843382 0.333333 0.333333
-4.632828 2.842144 0.333333 0.333333
-4.632743 2.840873 0.200000 0.200000
-4.632734 2.839520 0.142857 0.142857
-4.632625 2.838639 0.125000 0.125000
-4.632717 2.837211 0.125000 0.125000
-4.632665 2.835261 0.100000 0.100000
-4.632533 2.833570 0.100000 0.100000
-4.632290 2.832986 0.000000 0.000000
-4.632240 2.830917 0.100000 0.100000
Quality analysis of the NTv2 file
To determine the overall quality of the NTv2 file, the 16788 identical points of the source reference system have been transformed with the new NTv2 binary file and compared with the original points of the target reference system. The differences between the calculated points in the target reference system and the original points in the target system represent the criterion for the quality of the NTv2 file. The distribution of the deviations is listed in clusters:
5 with deviations > 0.01 <= 0.05 millimeter = 0.03%
8 with deviations > 0.05 <= 0.1 millimeter = 0.05%
329 with deviations > 0.1 <= 0.5 millimeter = 1.96%
899 with deviations > 0.5 <= 1.0 millimeter = 5.36%
10801 with deviations > 1.0 <= 5.0 millimeter = 64.34%
3507 with deviations > 5.0 <= 10.0 millimeter = 20.89%
1231 with deviations > 10.0 <= 50.0 millimeter = 7.33%
8 with deviations > 50.0 <= 77.32 millimeter = 0.05%
Mean position error: 4.50 millimeter
The mean position error for all 16788 points is only 4.5 millimeters. Around 93% of the deviations are smaller than one centimeter. No deviation is larger than 7.8 centimeter. The result has a high quality regarding the accuracy and shows that the City of Wuppertal has developed a homogeneous and error-free base point field. The larger inaccuracies of more than one centimeter result from the use of the area-wide Helmert parameter set in the point-poor peripheral areas of the quadrangular effective area.
If further high-quality identical points from the neighbouring communities were available for the point-free border areas, the quality of the NTv2 file could be increased considerably. For the modeling only identical points without specification of the ellipsoidal height were available. With ellipsoidal heights, the result would be even slightly better.
The quality analysis does not assess the quality of the calculations performed by the program NTv2Creator, but the quality of the identical points used for modeling. By correcting erroneous or inaccurate points or by filling gaps, the quality can be further improved.
A comparative calculation with the Helmert Seven Parameter Set offered on the website of the City of Wuppertal gives the following result:
6 with deviations > 0.1 <= 0.5 millimeter = 0.04%
12 with deviations > 0.5 <= 1.0 millimeter = 0.07%
557 with deviations > 1.0 <= 5.0 millimeter = 3.32%
1938 with deviations > 5.0 <= 10.0 millimeter = 11.54%
12753 with deviations > 10.0 <= 50.0 millimeter = 75.96%
1521 with deviations > 50.0 <= 100.0 millimeter = 9.06%
1 with deviations > 100.0 <= 107.36 millimeter = 0.01%
Mean position error: 27.93 millimeter
The mean position error for all 16788 points is 27,93 millimeters. The mean position error determined with the NTv2 file is only 4.5 millimeters, which is 6 times better than the mean position error determined with the Helmert parameter set. Only about 15% of the deviations are smaller than one centimeter. The maximum deviation is still almost 11 cm. The comparative calculation proves that the use of the much more accurate NTv2 file makes sense.
Visualization of the modeled NTv2 file
The figure shows the shift ranges and shift directions in the effective area of the NTv2 file of the City of Wuppertal. The graphic has been created with the program NTv2Tools by KilletSoft.
Fig. 13 click! The shift width is the difference between the source reference system and the target reference system at a coordinate point. The shift width is calculated from the shifts of the latitude and the longitude as absolute value in geographic seconds. For the graphical representation, the shift widths are calculated for all grid meshes present in the NTv2 file and are assigned to the rainbow colours.
The shift direction is the geographic angle that occurs between a coordinate point of the source reference system and the same point in the target reference system. The shift directions are displayed in a matrix over the effective range of the NTv2 grid as vectors. Each vector represents the direction and quantity of the shift at a coordinate point shown as a small circle.
The shift ranges and shift directions hardly show any obvious anomalies. The graph confirms a largely homogeneous and error-free base point field in the urban area of the City of Wuppertal. There are only three small areas where the color gradient is inhomogeneous. The support point field can be further improved by checking and correcting the identical points in the marked areas.
Search for the optimal NTv2 file
Of course, the author was not able to make an improvement by adjusting and adding identical points in the base point field. However, a second calculation with different mesh sizes resulted in a slightly better quality than the previous exemplary calculation. The mesh sizes were reduced to three vertical seconds and five horizontal seconds. This has increased the number of grid points to 45507. The clusters of the improved deviations can be found in the log file:
11 with deviations > 0.01 <= 0.05 millimeter = 0.07%
17 with deviations > 0.05 <= 0.1 millimeter = 0.10%
634 with deviations > 0.1 <= 0.5 millimeter = 3.78%
1689 with deviations > 0.5 <= 1.0 millimeter = 10.06%
11800 with deviations > 1.0 <= 5.0 millimeter = 70.29%
1835 with deviations > 5.0 <= 10.0 millimeter = 10.93%
784 with deviations > 10.0 <= 50.0 millimeter = 4.67%
16 with deviations > 50.0 <= 100.0 millimeter = 0.10%
2 with deviations > 100.0 <= 111.79 millimeter = 0.01%
Mean position error: 3.39 millimeter
The mean position error for all 16788 points is around 1.1 millimetres better with 3.39 millimetres compared to 4.5 millimetres. The number of deviations greater than one centimeter has also improved slightly with 95% compared to 93%. However, the largest deviation of 11.2 centimeters is slightly worse than the previously determined 7.8 centimeters.
By trying various boundaries and mesh sizes, you can gradually find the optimum result. This is where the disadvantage of modeling an NTv2 file can be seen: It is time-consuming.
For the modeling of the NTv2 file from a number of identical points in two reference systems in the area of the City of Wuppertal three geodetic programs by KilletSoft have been used:
Detailed information about the software and the possibility to download trial versions can be found on the KilletSoft websites listed above.
The installation of the software is very simple and can be done with just a few clicks.
After installing the free trial version of the program NTv2Creator, all files described in this article are available for a test run with the identical points of the city of Wuppertal. A configuration file that can be read into the program causes the settings required by the program. During the tests with the three possible transformation models the program generates NTv2 files in ASCII and binary format.
The three geodetic programs are based on the Geodetic Development Kit GeoDLL from KilletSoft. All geodetic formulas and calculations used by the programs are included in the Dynamic Link Library GeoDLL.