be4.pl

Summary
be4.pl
BE4 tool
Variables
version
optionsContains be4 call options :
thisAll parameters by section :
Functions
mainMain method.
initChecks options and initializes the default logger.
configLoads environment and properties files and merge parameters.
checkParamsChecks presence of mandatory sections in parameters.
doItSteps in order, using parameters :
IsEmptyPrecises if an hash or a reference can be considered as empty.
Details
Command’s options
Write a configuration fileA file configuration can be composed of sections and parameters following :
BE4 global configuration fileProperties by section.
logger
datasource
pyramidFor more informations about properties, see BE4::Pyramid.
processFor more informations about properties, see BE4::Forest and BE4::Commands.
Paths in brief
DataSource configurationFor more informations about datasource properties loading, see BE4::DataSourceLoader and BE4::DataSource.
For all types
For an image sourceFor more informations about properties, see BE4::ImageSource.
For a WMS sourceFor more informations about properties, see BE4::Harvesting.
Old way to provide ONE datasourceNo specific configuration file for data sources.
Externals tools

BE4 tool

Synopsis

perl be4.pl --conf=path [ --env=path ]
# All parameters of the --env config file can be overided by --conf config file parameter

Tool allowing to generate a pyramid of tiled TIFF images, several resolutions, with compression (lossless or lossy) from serveral data sources, georeferenced image (ortho, scan, ...) or WMS service.  Output format respects WMS/WMTS server ROK4.

This tool write scripts, to execute to obtain the pyramid.

2 types of generation :

  • create a new pyramid with data
  • create a pyramid from an ancestor one (update)

2 data source type :

  • images
  • WMS service
Summary
Variables
version
optionsContains be4 call options :
thisAll parameters by section :
Functions
mainMain method.
initChecks options and initializes the default logger.
configLoads environment and properties files and merge parameters.
checkParamsChecks presence of mandatory sections in parameters.
doItSteps in order, using parameters :
IsEmptyPrecises if an hash or a reference can be considered as empty.

Variables

version

options

my %options

Contains be4 call options :

versionTo obtain the command’s version
helpTo obtain the command’s help
usageTo obtain the command’s usage
propertiesConfiguration file
environmentEnvironment file

this

my %this

All parameters by section :

loggerCan be null datasource -
harvestingDeprecated, use datasource section instead. pyramid -
tilematrixsetCan be in pyramid section
nodataCan be in pyramid section
tileCan be in pyramid section process -

Functions

main

sub main

Main method.

See Also

init, config, doIt

init

sub init

Checks options and initializes the default logger.  Check environment file (optionnal) and properties file (mandatory).

config

sub config

Loads environment and properties files and merge parameters.  Those in the properties file have priority over those in the environment file.

See Also

checkParams

checkParams

sub checkParams

Checks presence of mandatory sections in parameters.

  • pyramid
  • process
  • datasource

Sections tile, tms and nodata are inserted into the pyramid section.

Logger parameters are used to initialize logger (Log::Log4perl library).

doIt

sub doIt

Steps in order, using parameters :

IsEmpty

sub IsEmpty

Precises if an hash or a reference can be considered as empty.

Parameters (list)

valuevar - Variable to test

Details

Summary
Command’s options
Write a configuration fileA file configuration can be composed of sections and parameters following :
BE4 global configuration fileProperties by section.
logger
datasource
pyramidFor more informations about properties, see BE4::Pyramid.
processFor more informations about properties, see BE4::Forest and BE4::Commands.
Paths in brief
DataSource configurationFor more informations about datasource properties loading, see BE4::DataSourceLoader and BE4::DataSource.
For all types
For an image sourceFor more informations about properties, see BE4::ImageSource.
For a WMS sourceFor more informations about properties, see BE4::Harvesting.
Old way to provide ONE datasourceNo specific configuration file for data sources.
Externals tools

Command’s options

--helpDisplay the link to the technic documentation.
--usageDisplay the link to the technic documentation.
--versionDisplay the tool version.
--conf=pathPath to configuration file of the pyramid.  This option is mandatory.
--env=pathPath to environment file of all pyramid, it’s the common configuration.  This option is optional.  By default, the file configuration of install is used.

Write a configuration file

A file configuration can be composed of sections and parameters following :

[ section ]
parameter = value
;comments

In a data source configuration, sections are level ID, from which the data source is used.

BE4 global configuration file

Properties by section.

logger

log_pathDirectory path where to write log_file
log_filePath to print too output (in addition to STDOUT).  Log file path : log_path/log_file
log_levelDefine importance level from which we print messages.  From less to more important : DEBUG - INFO - WARN - ERROR - ALWAYS (WARN by default).

Example

[ logger ]
log_path = /var/log
log_file = be4_log_yyyy-mm-dd.txt
log_level = INFO

datasource

filepath_confComplete file path to configuration file for data sources (/home/ign/SOURCE/sources.txt)

Example

[ datasource ]
filepath_conf = /home/IGN/sources.txt

pyramid

For more informations about properties, see BE4::Pyramid.

pyr_name_newName of the new pyramid, used to name the descriptor, the pyramid’s list, the data directory and the temporary directory.
pyr_data_pathDirectory path, where to write the cache.  Mandatory.
pyr_desc_pathDirectory path, where to write the descriptor.  Mandatory.
pyr_name_oldIf existing pyramid, its name.  If this parameter is present, generation is an update.
pyr_data_path_oldIf existing pyramid, directory path where to find the cache.  Default : pyr_data_path value.
pyr_desc_path_oldIf existing pyramid, directory path where to find the descriptor.  Default : pyr_desc_path value.
update_modePossible values : slink, hlink, copy : existing pyramid, the way the new cache will reference the ancestor’s cache. inject : no new pyramid.  The old one is updated.  CAUTION : old pyramid is irreversibly modified Default : slink.
compressionPossible values : raw, jpg, zip, lzw, png, pkb.  PNG compression is not an official compression for tiff format (just readable by rok4).  Default : raw.
compressionoptionPossible values : none and crop.  Crop option have to be used with compression ‘jpg’.  Blocks which contain a white pixel are filled with white, to keep a pure white, in spite of compression.  Default : none.
colorThe color is a string and contain on value per sample, in decimal format, seperated by comma.  For 8 bits unsigned integer, value must be between 0 and 255.  For 32 bits float, an integer is expected too, but can be negative.  Example : “255,255,255” (white) for images whithout alpha sample, “-99999” for a DTM.
image_width, image_heightNumber a tile in the cache image, widthwise and heightwise.  16 * 16 for example.  Mandatory.
bitspersamplePossible values : 8, 32.  Have to be the same as source.  Mandatory.
sampleformatPossible values : uint, float.  Have to be the same as source.  Mandatory.
samplesperpixelPossible values : 1, 3, 4.  Have to be the same as source.  Mandatory.
photometricPossible values : gray, rgb.  Have to be consistent with the samples per pixel.  1 -> gray and 3,4 -> rgb.  Default : rgb.
interpolationPossible values : nn, linear, bicubic, lanczos.  Interpolation used to resampled images (in mergeNtiff).  Nodata pixel are excluded by the interpolation if we use masks.  Default : bicubic.
dir_depthImage’s depth from the level directory. depth = 2 => /.../LevelID/SUB1/SUB2/IMG.tif.  Mandatory.
dir_image, dir_nodata, dir_maskNames of subdirectories in the pyramid, in pyr_data_path/pyr_name/ Default : IMAGE, NODATA, MASK.
tms_nameTMS file name, with extension .tms.  Mandatory.
tms_pathDirectory, where to find the TMS file.  Mandatory.
pyr_level_topOptionnal.  If we don’t want pyramid to be generated to the TMS top level, we can force the top level.  It have to be consistent with base levels in the data sources configuration file.
export_masksAn option is used to precise if we want to write masks in the final pyramid.  So we can use masks but not export them (they may be useless for a pyramid which wont be never updated).  Only values “true” and “TRUE” lead to masks’ export.  Optionnal, FALSE by default.

Examples

[ pyramid ]
pyr_data_path = /home/IGN/PYRAMIDS
pyr_desc_path = /home/IGN/DESCRIPTOR
pyr_name_new = ORTHO
pyr_level_bottom = 19

tms_name = LAMB93_10cm.tms
tms_path = /home/IGN/TMS

dir_depth = 2
image_width = 16
image_height = 16

dir_image = IMAGE
dir_nodata = NODATA
dir_mask = MASK
export_masks = TRUE

compression         = png
bitspersample       = 8
sampleformat        = uint
photometric         = rgb
samplesperpixel     = 3
interpolation       = bicubic

; red as nodata color
color               = 255,0,0

process

For more informations about properties, see BE4::Forest and BE4::Commands.

job_numberLevel of parallelization for scripts.  Distribution details are different according to the kind of graph (see BE4::QTree and <BE4::Graph>).  Mandatory.
path_tempTemporary directory path proper to each script : temporary files are written in path_temp/pyr_name_new/SCRIPT_ID.
path_temp_commonCommon temporary directory path : temporary files which have to be shared between several scripts are written in path_temp_common/pyr_name_new/COMMON.  Mandatory.
path_shellDirectory path, to write scripts in.  Scripts are named in like manner for all generation.  That’s why the path_shell must be specific to the generation (contains the name of the pyramid for example).  Mandatory.
use_masksTo avoid some problems (data removing), we can use mask during generation, to know precisely where is data, in images.  This behaviour make process longer, that’s why default behaviour is without mask.  Only values “true” and “TRUE” lead to masks’ use.  Optionnal, FALSE by default.

Example

[ process ]
path_temp = /tmp/PRIVATE/
path_temp_common = /tmp/COMMON/
path_shell  = /home/IGN/SCRIPT/ORTHO
job_number = 16

Paths in brief

  • Descriptor file : pyr_desc_path/pyr_name.pyr
  • List file: pyr_desc_path/pyr_name.list
  • Cache directory: pyr_data_path/pyr_name/
  • TMS file: tms_path/tms_name
  • Scripts: path_shell/SCRIPT_X.sh
  • Temporary directory: path_temp/pyr_name_new/SCRIPT_X
  • Common temporary directory: path_temp_common/pyr_name_new/COMMON

DataSource configuration

For more informations about datasource properties loading, see BE4::DataSourceLoader and BE4::DataSource.

For all types

srsThe images’ or the extent’s SRS.  Mandatory.

For an image source

For more informations about properties, see BE4::ImageSource.

path_imageDirectory path, where to find georeferenced images to use to generate the cache.  Subdirectory are authorized and browsed.
preprocess_commandstring - command to call to preprocess source images (optionnal)
preprocess_opt_begstring - command arguments placed between the command and the source file (optionnal even with a command specified)
preprocess_opt_midstring - command arguments placed between the source file and the target file (optionnal even with a command specified)
preprocess_opt_endstring - command arguments placed after the target file (optionnal even with a command specified)
preprocess_tmp_dirstring - directory in which preprocessed images will be created.  Mandatory if a preprocessing command is given.

For a WMS source

For more informations about properties, see BE4::Harvesting.

wms_layerResource to harvest.
wms_urlURL of rok4.
wms_version1.3.0
wms_requestKind of request : “getMap”.
wms_formatFormat of the harvested image.
wms_styleOptionnal.
wms_transparentTRUE or FALSE.  Optionnal.
max_width, max_heightIf not defined, images will be harvested all-in-one.  If defined, requested image size will have to be a multiple of this size.
min_sizeUsed to remove too small harvested images (filled with nodata), in byte.  Optionnal.
wms_bgcolorFormat : 0xFFFFFF.  Optionnal.
extentArea where we have to harvest.  It can be a bbox “xmin,ymin,xmax,ymax” or the path of a file which contains a geometry in WKT format.  Mandatory if no georeferenced image to define extent (path_image is not present).
listPathy to a file, containing a list of images’ indices (I,J) to harvest.  It could replace an extent.

Examples

[ 14 ]
; We use a WMS service and we have to know extent to harvest. Georeferenced images are used to define this extent.
path_image          = /home/IGN/DATA/ORTHO_LAMB93
srs                 = IGNF:LAMB93

wms_layer   = WMS_ORTHO
wms_url     = http://localhost/rok4
wms_version = 1.3.0
wms_request = getMap
wms_format  = image/tiff

[ 17 ]
; No georeferenced images, we use a WMS service and we have to know extent to harvest. Extent's SRS is supplied too.
srs = IGNF:WGS84G
extent = /home/IGN/Polygon.txt

wms_layer   = ORTHO
wms_url     = http://localhost/rok4
wms_version = 1.3.0
wms_request = getMap
wms_format  = image/png
wms_style  = line
wms_bgcolor  = 0x80BBDA
wms_transparent  = true
max_width = 2048
max_height = 1024

[ 18 ]
; Georeferenced images
srs = EPSG:3857
path_image = /home/IGN/DATA/ORTHO_PM

[ 19 ] ; Georeferenced images with preprocessing command srs = EPSG:3857 path_image = /home/IGN/DATA/ORTHO_PM preprocess_command = composite preprocess_opt_beg = -watermark 50x50 -gravity center preprocess_opt_mid = grayscale_watermark_img.ext (end code)

Old way to provide ONE datasource

No specific configuration file for data sources.  All parameters regarding THE data source, with georeferenced images, are in the BE4 configuration file.

  • We have to precise the level from which the one data source is used.  pyr_level_bottom, in the pyramid section.
  • Image source parameters (path_image and srs) are in the datasource section (see For an image source for meanings).
  • Possibly WMS service parameters (wms_layer, ...) are in a harvesting section (see For a WMS source for options and meanings).

Example

; Georeferenced images + WMS
[ datasource ]
path_image          = /home/IGN/DATA/ORTHO_LAMB93
srs                 = IGNF:LAMB93

[ harvesting ]
wms_layer   = WMS_ORTHO
wms_url     = http://localhost/rok4
wms_version = 1.3.0
wms_request = getMap
wms_format  = image/tiff


[ pyramid ]
pyr_data_path = /home/IGN/PYRAMIDS
pyr_desc_path = /home/IGN/DESCRIPTOR
pyr_name_new = ORTHO
pyr_level_bottom = 19

tms_name = LAMB93_10cm.tms
tms_path = /home/IGN/TMS

dir_depth = 2
image_width = 16
image_height = 16

dir_image = IMAGE
dir_nodata = NODATA
dir_mask = MASK

compression         = png
bitspersample       = 8
sampleformat        = uint
photometric         = rgb
samplesperpixel     = 3
interpolation       = bicubic

; red as nodata color
color               = 255,0,0

Externals tools

Scripts’ execution needs following tools

  • pngcheck
  • wget
my %options
Contains be4 call options :
my %this
All parameters by section :
sub main
Main method.
sub init
Checks options and initializes the default logger.
sub config
Loads environment and properties files and merge parameters.
sub checkParams
Checks presence of mandatory sections in parameters.
sub doIt
Steps in order, using parameters :
sub IsEmpty
Precises if an hash or a reference can be considered as empty.
Store all informations about a pyramid.
Creates and manages all graphs, NNGraph and QTree.
Configure and assemble commands used to generate pyramid’s images.
Loads, validates and manages data sources.
Manage a data source, physical (image files) or virtual (WMS server) or both.
Define a data source, with georeferenced image directory.
Stores parameters and builds WMS request.
sub new
Pyramid constructor.
sub new
DataSourceLoader constructor.
sub updateLevels
Determine top and bottom for the new pyramid and create Level objects.
sub new
Forest constructor.
sub writeCachePyramid
Write the Cache Directory Structure (CDS).
sub writeListPyramid
Write the pyramid list.
sub writeConfPyramid
Export the Pyramid object to XML format, write the pyramid’s descriptor (pyr_desc_path/pyr_name_new.pyr).
sub computeGraphs
Computes each NNGraph or QTree one after the other and closes scripts to finish.
Representation of a quad tree image pyramid : pyramid’s image = Node
For more informations about properties, see BE4::ImageSource.
For more informations about properties, see BE4::Harvesting.
Close