YLUPC
DataFileManagement – Yukon Planning Atlas

Data File Management

Table of Contents

  1. PlanYukon Regional Planning Database Best Practices Wiki
    1. Background
    2. Purpose
    3. Framework
    4. Roles
    5. Acknowledgment
  2. Getting Started as an Administrator
    1. Introduction
    2. Helpful Software
    3. Trac
    4. WinSCP
  3. Data File Management
    1. Valid Spatial Data Sources
    2. Accepted Data Projection
    3. Data File Naming Conventions
    4. Data File Contents
    5. Data File Storage
    6. Database Architecture
    7. Government of Yukon CSW Catalog WMS Service
  4. Data Styling
    1. Background on MapServer .map files
    2. Theme Files
    3. Template Files
    4. Layer Files
    5. .Lay Guidelines
    6. Using Classes
    7. Using Styles
    8. Labeling
    9. Symbols
    10. Projection
    11. Using Scales
  5. Metadata Management
    1. Implemented Standards
    2. Layer Metadata
    3. Theme Metadata
    4. Accessing Metadata through the Atlas
    5. Accessing OGC Metadata through the WMS Service
    6. Management Tools
  6. Data Processing Work Flow
    1. Step 1: Preparing the Spatial Data
    2. Step 2: Create a .lay file
    3. Step 3: Submit for Publication
    4. Step 4: Publishing a Layer
    5. Step 5: Add Layer to Existing Theme File
    6. Step 6: Verify Layer Processing
    7. Refresh Layer Data
  7. Quality Control
    1. Layer Quality Assurance Checklist
    2. Data Optimization Checklist
    3. Publishing Atlas Data
  8. Recommended Data Management Tools
    1.  MXD2WMS
    2.  FGS
    3.  MapServer for Windows
    4.  QuantumGIS
    5.  uDig

Valid Spatial Data Sources

Although the underlying Open Source libraries behind the Atlas support hundreds of different data sources, for the Atlas to perform optimally please use the following guidelines:

Vector Files

  • all vector data sources should be converted to ESRI Shapefile format (*.shp)
  • each shapefile must have an associated quadtree index file created for it, using MapServer's shptree commandline utility (for examples see  http://www.mapserver.org/utilities/shptree.html)
    • example command: 
             shptree ny_roads.shp
  • contiguous shapefiles should be indexed with MapServer's tile4ms commandline utility (for examples see  http://www.mapserver.org/utilities/tile4ms.html)
    • example command: 
             tile4ms metafile.txt tileindex
      • where “metafile.txt” is a list containing the full path to all .shp files

Raster Files

  • all raster data sources should be converted to GeoTIFF format
  • raster formats such as ECW and MrSID can be converted to GeoTIFF with the gdal_translate commandline utility (for usage see  http://www.gdal.org/gdal_translate.html)
    • example command: 
             gdal_translate -of GTiff ny_dem.ecw ny_dem.tif
  • large GeoTIFF files should have overviews embedded, using the gdaladdo commandline utility (for usage see  http://www.gdal.org/gdaladdo.html)
    • example command: 
             gdaladdo ny_dem.tif 2 4 8 16
  • contiguous GeoTIFF files should be indexed with GDAL's gdaltindex commandline utility (for examples see  http://www.gdal.org/gdaltindex.html)
    • example command: 
             gdaltindex ny_dem_index.shp *.tif
  • GeoTIFF files should be in Yukon Albers projection. The following example takes a GeoTIFF with a corrupted projection (but known to be Yukon Albers), and "reprojects" it to Yukon Albers
    • example command: 
             gdalwarp -s_srs 'EPSG:3578' -t_srs 'EPSG:3578' ./defective/Moose.tif ./Moose.tif

Non-local Data

  • the Atlas does support display of remote data through the various Open Geospatial Consortium (OGC) interoperability standards, including Web Map Service (WMS) and Web Feature Service (WFS)
  • however all querying in the application requires local data, therefore non-local data (in the form of WFS) must be cached locally into a PostGIS database if querying that data is required
  • for more information on using remote data in MapServer, please see the “OGC” section of the MapServer documentation site ( http://www.mapserver.org/ogc/index.html)

Accepted Data Projection

Although the underlying libraries behind the Atlas support thousands of projections, for the Atlas to perform optimally please use the following guidelines:

  • all data should be converted to the Yukon Albers NAD83 projection
  • projection can be referenced in MapServer by using the EPSG CODE or with explicit projection parameters, which are both outlined below:
    • EPSG code: 3578
    • PROJ.4 parameters: 
              proj=aea
        lat_1=61.66666666666666 (latitude of first standard parallel)
        lat_2=68 (latitude of second standard parallel)
        lat_0=59 (latitude of false origin)
        lon_0=-132.5 (longitude of false origin)
        x_0=500000 (false origin easting)
        y_0=500000 (false origin northing)
        ellps=GRS80
        datum=NAD83
        units=m

Data File Naming Conventions

Data files must follow simple naming conventions:

  • file names may only contain letters, numbers and underscores
  • file names should never begin with a number
  • file names should never contain spaces
  • file names must be all in lowercase

Data File Contents

Wherever possible, any fields/columns not being used by the Atlas should be removed from the data.

Data File Storage

All local spatial data is stored in the /data folder, in an appropriate subfolder according to the extent of the data. Each existing subfolder is described as follows:

LocationDescription
/canadaCanada-wide extents
/canada/baseCanadian base data
/canada/base/1mCanadian base data at 1:1million scale
/canada/base/2mCanadian base data at 1:2million scale
/canada/thematicThematic Canadian base data
/licenseData license files in text format
/metadataXML metadata files
/nwtNWT-specific data files
/uploadData files uploaded by application users
/usaU.S. base data
/yukonYukon spatial data
/yukon/baseYukon base data
/yukon/thematicThematic Yukon data

Database Architecture

Specifications

  • DB Name: xxxxxx
  • PostgreSQL version: 8.1
  • PostGIS version: 1.2.1
  • Port: 5432
  • Username: xxxxxx
  • Password: xxxxxx
  • Host: contact Atlas admin

Tables

The !PostgreSQL databases structure required for the Atlas application.

geometry_columns

purpose: used by PostGIS to store geometry

groups

purpose: store the user groups for the application

Column Name Type Description
id(PK) serial unique id of group
name varchar(50) group name
description varchar(50) description of group

layers

purpose: store the layer file information for the application

Column Name Type Description
id (PK) varchar(255)unique id of layer
filename varchar(255)full path and name of .lay file (without the extension) relative to the /map/layers folder
catalog_group varchar(255)group name displayed in catalog
description varchar(255) description of layer
human_name varchar(255) readable name for the data file
contact_name varchar(255) full name of the contact person for this data
contact_email varchar(255) email of the contact person for this datav
access_constraints varchar(255) access limitations for this data file
keywords varchar(255) general keywords describing the data
scale varchar(255) scale of data, in 1:xxxxx format
data_accuracy varchar(255) accuracy of data, in “+/- x units” format
data_currency varchar(255) date the data was created and/or last updated, in YYYY,MM,DD format
data_source varchar(255) source of the data
data_update_frequency varchar(255) frequency of data updates, such as “annual”, “monthly”, or “daily”
projection varchar(255) projection of data (include all projection parameters, or refer to standard lookup file such as EPSG)
more_information_link varchar(255) URL to more information about the data
current_data_format varchar(255) format of the data currently (such as “ESRI Shapefile”)
original_data_format varchar(255) original format of the data (such as ASCII Grid)
data_filename varchar(255) full path and name of data file, with the extension (shapefiles should be referenced using only the .shp extension) relative to the /data folder
indexed boolean does the data file have an associated qix file for shapes
published boolean is the data file included in the Atlas' WMS service
wfs_cached boolean is the WFS data cached locally
processed booleanhas the data and layer been processed (updated for mapserver 5.x, and contains required metadata)
metadata_processed booleandoes the layer contain accurate values for all required Atlas metadata

layers_perm

purpose: store permissions for layers and groups

Column Name Type Description
layerid bigintunique identifier pointing to the layers.id column
groupid bigint unique identifier pointing to the groups.id column
viewable boolean viewing permissions for layer and group
extractable boolean extracting permissions for layer and group

spatial_ref_sys

purpose: used by PostGIS to store projection systems, in both EPSG and

PROJ.4 formats

themes

purpose: store the theme file information for the application

Column Name Type Description
id (PK) serial unique id of theme
filename varchar(255) name of theme file (without the extension) relative to the /map/themes folder
description varchar(255) description of theme
author varchar(255) name of author of the theme

themes_perm

purpose: store permissions for themes and groups

Column Name Type Description
themeidbigint unique identifier pointing to the themes.id column
groupid bigint unique identifier pointing to the groups.id column
viewable boolean viewing permissions for theme and group

users

purpose: store user information for the application

Column Name Type Description
id (PK) serial unique id of user
groups varchar(255)comma-separated list of groups.id values
login varchar(50) login name for user
password varchar(32) encrypted password for user
fname varchar(50) first name of user
lname varchar(50) last name of user
emailvarchar(50) email address of user
hint varchar(255)password hint for user

widgets

purpose: store widget information for the application

Column Name Type Description
id (PK) serial unique id of widget
namevarchar(255) readable name of widget
description varchar(255) description of widget

widgets_perm

purpose: store permissions for themes and groups

Column Name Type Description
widgetidbigint unique identifier pointing to the widgets.id column
groupid bigint unique identifier pointing to the groups.id column
viewable boolean viewing permissions for widget and group

Government of Yukon CSW Catalog WMS Service

GetCapabilities

A description of the production service of the CSW Catalog WMS service can be accessed with the following request:

 freehand.gov.yk.ca/wmsconnector/com.esri.wms.Esrimap/cswCatalog_wms?SERVICE=wms&REQUEST=GetCapabilities

The resulting file can be saved locally and renamed to have a “.xml” extension, and opened in any text editor. A nice way to view this xml file is to drag it into your favorite Web browser, such as Firefox or Internet Explorer.

'Example GetMap Request  freehand.gov.yk.ca/wmsconnector/com.esri.wms.Esrimap/cswCatalog_wms?VERSION=1.1.1& REQUEST=GetMap&LAYERS=Shaded Relief 500M Raster&SRS=EPSG:4326& BBOX=-616492.253978353,57087.6866405592,1617007.74602165,2348587.68664056& WIDTH=400&HEIGHT=400&FORMAT=image/png

Supported Projections

The production service of the CSW Catalog only lists 3 supported projections: 

<SRS>EPSG:4326</SRS>
<SRS>EPSG:4267</SRS>
<SRS>EPSG:4269</SRS>

Whereas the Yukon Planning Atlas has adopted all of the projections currently supported by the Atlas of Canada's WMS service, and additionally the Yukon Albers projection (EPSG:3578): 

<SRS>EPSG:4326</SRS>
<SRS>EPSG:3578</SRS>
<SRS>EPSG:42304</SRS>
<SRS>EPSG:42101</SRS>
<SRS>EPSG:4269</SRS>
<SRS>EPSG:2294</SRS>
<SRS>EPSG:2295</SRS>
<SRS>EPSG:26922</SRS>
<SRS>EPSG:26921</SRS>
<SRS>EPSG:26920</SRS>
<SRS>EPSG:26919</SRS>
<SRS>EPSG:26918</SRS>
<SRS>EPSG:26917</SRS>
<SRS>EPSG:26916</SRS>
<SRS>EPSG:26915</SRS>
<SRS>EPSG:26914</SRS>
<SRS>EPSG:26913</SRS>
<SRS>EPSG:26912</SRS>
<SRS>EPSG:26911</SRS>
<SRS>EPSG:26910</SRS>
<SRS>EPSG:26909</SRS>
<SRS>EPSG:26908</SRS>
<SRS>EPSG:26907</SRS>
<SRS>EPSG:26722</SRS>
<SRS>EPSG:26721</SRS>
<SRS>EPSG:26720</SRS>
<SRS>EPSG:26719</SRS>
<SRS>EPSG:26718</SRS>
<SRS>EPSG:26717</SRS>
<SRS>EPSG:26716</SRS>
<SRS>EPSG:26715</SRS>
<SRS>EPSG:26714</SRS>
<SRS>EPSG:26713</SRS>
<SRS>EPSG:26712</SRS>
<SRS>EPSG:26711</SRS>
<SRS>EPSG:26710</SRS>
<SRS>EPSG:26709</SRS>
<SRS>EPSG:26708</SRS>
<SRS>EPSG:26707</SRS>
<SRS>EPSG:2295</SRS>
<SRS>EPSG:2294</SRS>
<SRS>EPSG:54003</SRS>
<SRS>EPSG:54004</SRS>
<SRS>EPSG:54008</SRS>
<SRS>EPSG:54009</SRS>
<SRS>EPSG:54030</SRS>
<SRS>EPSG:102016</SRS>
<SRS>EPSG:102017</SRS>
<SRS>EPSG:102018</SRS>
<SRS>EPSG:102008</SRS>
<SRS>EPSG:102009</SRS>

The full list of projections supported by ArcIMS 9.2 can be viewed at:  http://webhelp.esri.com/arcims/9.2/general/mergedProjects/wms_connect/wms_connector/..%5Cwms_admin%5Csrs_list.htm

Native Projection

The native projection of the data in the CSW Catalog WMS service is 4326; therefore any request for a projection other than 4326 will take longer to execute and will affect the application performance.

Layer Names

Layer names in the CSW Catalog WMS service do contain spaces, which could possibly cause problems with client software used in the Atlas.

Scale Ranges

A document containing the scale ranges for all layers in the CSW Catalog WMS service should be provided.