RasterTreeBuilder - Generating image pyramids from georeferenced image collections
1. General Information
First of all, please make sure to read the chapter on General Information for all deegreeTools.
The main utility program coming with deegree WCS is the RasterTreeBuilder. It is located in org.deegree.tools.raster package and offers several configuration parameters to optimize data structure for fast access to coverage source(s)*.
The basic idea is that especially for large coverages in most cases a GetCoverage request targets just a small subset of the coverage's spatial extent with a high resolution or a large subset with a low resolution. So it will be useful to have a mechanism that avoids reading and processing data that is not required.
The deegree approach to solve this problem is similar to mechanisms used by several other software packages handling large amounts of raster data:
- In step of a pre-processing, the original, large data sources will first be resampled into several levels of lower resolution.
- After that each resolution level will be splitted into tiles each having a moderate size.
The optimal size strongly depends on the desired use of a WCS. E.g. if a WCS shall be used for online image visualization within a standard web browser a GetCoverage request hardly will use WIDTH and HEIGHT exceeding 800x800px. So an optimal size of a tile will be ~500x500px guaranteeing no more than nine tiles are needed to create the result to a GetCoverage request and in many cases even less tiles are needed.
3. Library Dependancies
deegree2.jar acme.jar batik-awt-util.jar commons-beanutils-1.5.jar commons-codec-1.3.jar commons-collections-3.1.jar commons-digester-1.7.jar commons-discovery-0.2.jar commons-httpclient-2.0.2-deegreeversion.jar commons-logging.jar ehcache-1.2.0_03.jar jai_codec.jar jai_core.jar mlibwrapper_jai.jar j3dcore.jar j3dutils.jar vecmath.jar jts-1.8.jar log4j-1.2.9.jar axis.jar jaxen-1.1-beta-8.jar
To resample and tile one or more source images the RasterTreeBuilder application needs definition of several parameters passed via commandline options. For each phase of the program – input, processing, output – there are several parameters needed.
An image file passed to the RTB must have a world file assigned to it by having the same name but with extension *.tfw, *.wld, *.jgw, *.jpgw, *.gfw, *.gifw, *.pgw or *.pngw. GeoTIFF files are also accepted.
There are three alternative ways/parameters to define which input files shall be used for creating a raster tree:
- defines a list of image file names (including full path information) seperated by ',', ';' or '|'
- defines a directory that shall be parsed for files in a known image format. Each file found will be used as input.
- conditional parameter used with -rootDir. It defines if all sub directories of -rootDir shall be parsed too (true|false - default false)
- name of a dBase file that contains a column, listing all files to be considered by the program
- name of the column containing the file names (mandatory if -dbaseFile is defined)
- name of the directory where the files are stored. If this parameter will not be set the program assumes the
- contains completely referenced file names (optional)
- If -dbaseFile is defined one can define a column that shall be used for sorting the files referenced by the -fileColumn (optional)
- If -sortColumn is defined this parameter will be used for definition of sorting direction (UP|DOWN default UP)
name of the spatial reference system used for the coverage (default EPSG:4326) Defines the EPSG-Code of the spatial reference system (SRS) that the coordinates of the input data are in. Just this native SRS is valid in the moment here. Be aware that the RTB is not able to project/transform your raster data at the moment.
- two types of are common:
- a) the boundingbox is defined on the center of the corner pixels
- b) the boundingbox is defined on the outer corner of the corner pixels
- First is default and will be used if this parameter is not set; second will be use if '-worldFileType outer' is defined.
- number of tree levels created (optional default = 1) Number of levels of different spatial resolutions to be calculated starting from the basis to the peak of the pyramid (Look at the examples for getting to understand in detail how the RTB works).
- name of the image format used for created tiles (png|jpg|jpeg|bmp|tif|tiff|gif default png|raw)
- image quality if jpeg is used as output format; valid range is from 0..1 (default 0.95)
- maximum size of created raster tiles in pixel (default 500)
- image bit depth; valid values are 32 and 16, default is 16
- boundingbox of the the resulting coverage. If not set the bbox will be determined by analysing the input map files. (optional) The bounding box coordinates must match the type of the spatial reference system flag (-srs). If you set a bigger bounding box than covered by the input images, you can have regions within the same coverage with and without rasterdata (It is like islands in the ocean). For example: If your input data is spreaded like a patchwork. For setting the color of the non-covered areas (ocean), use the bgColor-flag.
- spatial resolution of the coverage (at the basis of the pyramid). If not set, the resolution will determined by analysing the input map files. This parameter is conditional; if -bbox is defined -resolution must be defined, too. In order to calculate the spatial resolution of the coverage, you need to know the distance from N-S, W-E and the corresponding number of pixels. So if you give RTB the bbox, give it the pixel size as well.
interpolation method used for resampling raster images (Nearest Neighbor|Bicubic|Bicubic2|Bilinear default Nearest Neighbor) Be careful using Bicubic and Bicubic2 interpolation - there seems to be a problem with Java Advanced Imaging(JAI) !
- defines the background color of the created tiles for those regions where no data is available If no -bgColor is defined, a transparent background will be used for image formats that support transparency (e.g. png). Black is used for all other formats (e.g. bmp). For example a '-bgColor 0xFFFFFF' defines a white background.
- number of images that shall be cached. During the processing, input images are read in several times for different steps in the processing phase. A larger value (~20) increases speed (and reduces I/O operations) but also amount of required memory. (default = 5)
- directory where resulting tiles and describing shape(s) will be stored (mandatory)
- base name used for creating names of the raster tile files. It also will be the name of the created coverage. (mandatory)
name of a deegree WCS capabilities/configuration file. If defined the program will add the created rastertree as a new coverage to the WCS configuration. not implemented yet. Please add the configuration of your created coverage manually !
-h or -?
- print out the help
If you followed the preparations, you can use the RasterTreeBuilder with the following command:
LINUX: java -Xms300m -Xmx1000m -cp .:deegree2.jar org.deegree.tools.raster.RasterTreeBuilder WINDOWS: java -Xms300m -Xmx1000m -classpath .;deegree2.jar org.deegree.tools.raster.RasterTreeBuilder
You simply need to add the input parameters.
Otherwise you need to add the list of libraries to the classpath.
Creating a coverage with 5 levels out of a source directory on a Windows platform
java -Xms300m -Xmx1000m -classpath .;lib/deegree2.jar org.deegree.tools.raster.RasterTreeBuilder -outDir ../../data/utah/raster/saltlakecity -baseName saltlakecity -outputFormat jpg -maxTileSize 500 -noOfLevel 5 -srs EPSG:26912 -rootDir ../../data/utah/raster/orig
Creating a coverage with 4 Levels out of a dBase file on LINUX platform
java -Xms300m -Xmx1000m -classpath .:lib/deegree2.jar org.deegree.tools.raster.RasterTreeBuilder -dbaseFile /home/deegree/input.dbf -fileColumn LOCATION -sortColumn DATE -sortDirection DOWN -noOfLevel 4 -interpolation "Nearest Neighbor" -bbox 2573388.0,5618175.0,2574150.03,5618416.98686 -srs EPSG:31466 -resolution 0.09 -outDir /home/deegree/output -outputFormat png -bgColor 0xFFFF00 -baseName MyCoverage
If you like to tile big images (like the Blue Marble Mosaic with 176 MB as PNG (24 bit color depth)), you need to do some preprocessing first in order not to exceed the 2 GB RAM Limit of Java.
6. Calculation of one Coverage – Inside the RTB
To use the parameters correctly you should understand how RTB works and how the number of resolution levels and tile size is coupled.
Imagine a source image of 4000x4000px with a spatial resolution of 1x1m per pixel. Use '-noOfLevel 3' to force the creation of three resolution levels having 1m (base level of the pyramid), 2m and 4m (peak of the pyramid) spatial resolution. For each level, one tiling step will be performed on the hightest resolution level (1m). At the first tiling step, an image is splitted into four smaller images.
In our example in a first step, the source image is splitted in tiles each having 2000x2000px size; in the second step each of this files is splitted into four tiles each 1000x1000px and in the last, third step each of this tiles is splitted again into four tiles each 500x500px size (as stated above, this is a good size for normal visualization use cases). So you will receive 4³ = 64 tiles in total for the hightest resolution level.
For the second highest resolution level the original image will be resampled to a spatial resolution of 2m (double the highest resolution). After this, two tiling steps will be performed . We will receive 4² = 16 tiles of 500x500px size for the second hightest resolution level.
To create the third level the image will be resampled again to a resolution of 4m. Just one tiling step will be performed resulting in four tiles of 500x500px size. The result is a tiled and pyramidal datastructure.
RTB will create a deegree CoverageDescription as XML document and a sub directory for each source image and resolution level within the defined output directory. Shapefiles containing the location of the tiles (used as an index) as described above will also be created.
7. Referencing the new generated converage to the wcs_configuration.xml
Once you have run the RasterTreeBuilder successfully you need to link the wcs_configuration.xml file to the coverage decription file. Within the <wcs:ContentMetadata> you need to add a new <CoverageOfferingBrief...> increasing the gml:id="ID000001" by one. Change the <name> and <lonLatEnvelope> to the one used in the coverage description. Finally set the (absolute/relativ) path to the coverage description in <deegree:Configuration>.
<!-- mandatory; If missing it will be created by deegree and filled with 'CoverageOfferingBrief' descriptions for all coverages that can be found in directories listed in 'DataDirectoryList'. If 'ContentMetadata' isn't defined or is empty and no coverages that can be found in directories listed in 'DataDirectoryList' the configuration is invalid because a (deegree) WCS at least has to serve one coverage. If 'ContentMetadata' is defined deegree adds all coverages found in directories listed in 'DataDirectoryList' that are not defined ContentMetadata automaticaly. All attributes of 'ContentMetadata' are optional and don't have a default. -->
1 <wcs:ContentMetadata xmlns:wcs="http://www.opengis.net/wcs"> 2 <CoverageOfferingBrief gml:id="ID000001"> 3 <description></description> 4 <name>saltlakecity</name> 5 <label /> 6 <lonLatEnvelope xmlns:wcs="http://www.opengis.net/wcs" srsName="WGS84(DD)"> 7 <gml:pos dimension="2">-111.89985862992859,40.71940082135575</gml:pos> 8 <gml:pos dimension="2">-111.80644038553243,40.82820182642026</gml:pos> 9 </lonLatEnvelope> 10 <wcs:keywords> 11 <wcs:keyword></wcs:keyword> 12 </wcs:keywords> 13 <deegree:Configuration> 14 ./../data/utah/raster/saltlakecity/wcs_saltlakecity_configuration.xml 15 </deegree:Configuration> 16 </CoverageOfferingBrief> 17 </wcs:ContentMetadata>
Afterwards restart the tomcat and the coverage should be served.
Look for other deegreeTools.