YLUPC
DataProcessing – Yukon Planning Atlas

Data Processing Work Flow

Table of Contents

  1. PlanYukon Regional Planning Database Best Practices Wiki
    1. Background
    2. Purpose
    3. Framework
    4. Roles
    5. Acknowledgment
  2. Getting Started as an Administrator
    1. Introduction
    2. Helpful Software
      1. Text editor
      2. Subversion
        1. Troubleshooting
      3. MS4W
        1. Trouble-Shooting
    3. Trac
    4. WinSCP
    5. pgAdmin III
  3. Data File Management
    1. Valid Spatial Data Sources
      1. Vector Files
      2. Raster Files
      3. Non-local Data
    2. Accepted Data Projection
    3. Data File Naming Conventions
    4. Data File Contents
    5. Data File Storage
    6. Database Architecture
      1. Specifications
      2. Tables
        1. groups
        2. layers
        3. layers_perm
        4. spatial_ref_sys
        5. themes
        6. themes_perm
        7. users
        8. widgets
        9. widgets_perm
    7. Government of Yukon CSW Catalog WMS Service
      1. GetCapabilities
      2. Supported Projections
      3. Native Projection
      4. Layer Names
      5. Scale Ranges
  4. Data Styling
    1. Background on MapServer .map files
    2. Theme Files
    3. Template Files
    4. Layer Files
    5. .Lay Guidelines
      1. Naming
      2. Groups
      3. Layer TYPEs
      4. Metadata
      5. Field Formatting
    6. Using Classes
    7. Using Styles
    8. Labeling
      1. Available Fonts
      2. Label Size
      3. Label Scale
      4. Label Force
      5. Label Partials
      6. Label Minfeaturesize
      7. Label Buffer
      8. Label Priority
    9. Symbols
    10. Projection
      1. EPSG Usage
      2. PROJ.4 Usage
    11. Using Scales
      1. MINSCALE and MAXSCALE Deprecated
      2. Atlas Scale Ranges
  5. Metadata Management
    1. Implemented Standards
    2. Layer Metadata
      1. Atlas Keywords
    3. Theme Metadata
    4. Accessing Metadata through the Atlas
    5. Accessing OGC Metadata through the WMS Service
      1. GetCapabilities Request
    6. Management Tools
      1. ArcCatalog?
        1. About
        2. Functionality
        3. Assessment
      2. CatMDEdit
        1. About
        2. Functionality
        3. Assessment
      3. gvSIG
        1. About
        2. Functionality
        3. Assessment
  6. Data Processing Work Flow
    1. Step 1: Preparing the Spatial Data
      1. Guidelines
      2. Getting the Required Tools
      3. Documentation For Command Tools
      4. Useful Commands
        1. Reproject a Vector File
        2. Merge 2 Vector Files
        3. Keep Only Specific Attribute Values
        4. Keep Only Specific Fields
        5. Prep Raster Files
    2. Step 2: Create a .lay file
      1. .lay Templates
        1. Vector File Template
        2. Raster File Template
        3. WMS/OGC Template
      2. Generate .lay Metadata
      3. .lay Guidelines
    3. Step 3: Submit for Publication
    4. Step 4: Publishing a Layer
    5. Step 5: Add Layer to Existing Theme File
    6. Step 6: Verify Layer Processing
    7. Refresh Layer Data
  7. Quality Control
    1. Layer Quality Assurance Checklist
      1. Published Layer QA Checklist Forms
    2. Data Optimization Checklist
    3. Publishing Atlas Data
      1. WMS Mapfile Location
      2. Publish Local Data to WMS
      3. MapServer Required Metadata
      4. WMS GetCapabilities Warnings
      5. GeoConnections Discovery Portal Registration
  8. Recommended Data Management Tools
    1.  MXD2WMS
    2.  FGS
    3.  MapServer for Windows
    4.  QuantumGIS
    5.  uDig

The purpose of this section is to document the necessary steps to prepare spatial data for publication in the Atlas.

Step 1: Preparing the Spatial Data

Guidelines

Atlas guidelines for Vector and Raster data should be followed at: DataFileManagement.

Getting the Required Tools

See the DataTools page for links to FGS and MS4W.

Documentation For Command Tools

Here are some relative pages:

Useful Commands

Reproject a Vector File

  • reproject from UTM Zone 9N (EPSG:3716) to Yukon Albers (EPSG:3578) 

    ogr2ogr -f "ESRI Shapefile" -t_srs EPSG:3578 -s_srs EPSG:3716 new9.shp raw9.shp

Merge 2 Vector Files

  • merge the projected zone 10 file ("new10.shp") into the projected zone 9 file ("new9.shp") 
   ogr2ogr -f "ESRI Shapefile" -append -update new9.shp new10.shp -nln new9

Keep Only Specific Attribute Values

1. test first with ogrinfo, using an SQL statement 

   ogrinfo -sql "SELECT * FROM new9 WHERE REMAIN_IND = YES" new9.shp -summary

  • look at the Feature Count value for verification: 
           Layer name: new9
       Geometry: Polygon
       Feature Count: 23
       Extent: (537629.899405, 611923.720739) - (969241.728370, 951845.595063)
       Layer SRS WKT:
       ...

2. use ogr2ogr with an SQL statement to create a file ("output.shp") with only that value 

   ogr2ogr -f "ESRI Shapefile" -sql "SELECT * FROM new9 WHERE REMAIN_IND = YES" output.shp new9.shp

Keep Only Specific Fields

  • use ogr2ogr with the '-select' parameter to choose fields to keep
  • select parameter takes a comma-delimited list 
   ogr2ogr -f "ESRI Shapefile" -select DESIGNATOR,REMAIN_IND newfile.shp old.shp

Prep Raster Files

Step 2: Create a .lay file

A "wrapper" is required around the data files for each layer, to enable integration of multiple data formats and layers into a common web mapping standard (WMS). This wrapper is a text file that contains descriptions of:

  • data source
  • metadata
  • data projection
  • data stratification
  • data styling

These wrapper files are identified by a .lay file extension. The .lay files contain the  Mapserver LAYER object for one representation (STYLE) of a single data source.

.lay Templates

Vector File Template

  • use the following file as a template for vector data files: /map/layers/canada/base/1m/drainage_1m.lay
    • which contains: 
         #
         # Drainage Lines 1:1m
         #

         LAYER
           # layername must not contain any spaces or special characters
           NAME "drainage_1m"
           # groupname must not contain any spaces or special characters
           GROUP "Drainage"
           # set the required metadata for the Atlas
           # see the /docs/yukon_atlas_metadata_management.doc file
           METADATA
            "ows_abstract"           "The Atlas of Canada 1,000,000 National Frameworks Data are ..."
            "ows_attribution_title"  "Atlas of Canada National Scale Frameworks"
            "ows_dataurl_format"     "application/zip"
            "ows_dataurl_href"       "http://geogratis.cgdi.gc.ca/download/frameworkdata/hydrology/analytical/drainage_network/yukon/yukondrnetwork_1m_v6-0.shp.zip"
            "ows_group_title"	     "Drainage"
            "ows_keywordlist"        "base,drainage,hydology,reference,water,yukon"
            "ows_metadataurl_format" "application/msword"        
            "ows_metadataurl_href"   "http://geogratis.cgdi.gc.ca/download/frameworkdata/hydrology/analytical/drainage_network/1M_HYDRO_NETWORK_EN_2008.doc"
            "ows_metadataurl_type"   "FGDC"
            "ows_srs"                "EPSG:4326 EPSG:3578 EPSG:42304 EPSG:42101 EPSG:4269 EPSG:2294 EPSG:2295 EPSG:26922 EPSG:26921 EPSG:26920 EPSG:26919 EPSG:26918 EPSG:26917 EPSG:26916 EPSG:26915 EPSG:26914 EPSG:26913 EPSG:26912 EPSG:26911 EPSG:26910 EPSG:26909 EPSG:26908 EPSG:26907 EPSG:26722 EPSG:26721 EPSG:26720 EPSG:26719 EPSG:26718 EPSG:26717 EPSG:26716 EPSG:26715 EPSG:26714 EPSG:26713 EPSG:26712 EPSG:26711 EPSG:26710 EPSG:26709 EPSG:26708 EPSG:26707 EPSG:2295 EPSG:2294 EPSG:54003 EPSG:54004 EPSG:54008 EPSG:54009 EPSG:54030 EPSG:102016 EPSG:102017 EPSG:102018 EPSG:102008 EPSG:102009" 
            "ows_title"              "Drainage Lines 1M"    
          END
          TYPE LINE
          OPACITY 65
          STATUS ON
          MAXSCALEDENOM 2500000
          MINSCALEDENOM  500001
          TILEINDEX "../data/canada/base/1m/drainage_l_1m/drainage_1m"
          TILEITEM "location"
          # layer projection
          # this is the actual projection of the data 
          PROJECTION
            "proj=aea"
            "ellps=GRS80"
            "lat_0=59"
            "lon_0=-132.5"
            "lat_1=61.667"
            "lat_2=68"
            "x_0=500000.0"
            "y_0=500000.0"
            "units=m"
            "no_defs"
          END  
          # style the data  
          CLASS
            NAME "River"     
            STYLE
              COLOR 35 85 200
              WIDTH 1
            END
          END
          # configure for GetFeatureInfo requests
          # should not be modified, this is just a placeholder to tell MapServer
          # to allow GetFeatureInfo requests (meaning to allow for querying of the
          # map image on the WMS client-side)
          # TEMPLATE "dummy.html"  
        END # layer

Raster File Template

  • use the following file as a template for raster data files: /map/layers/yukon/base/yukon_dem.lay
    • which contains: 
         #
         # Yukon Digital Elevation Model
         #

         LAYER
           # layername must not contain any spaces or special characters
           NAME "yukon_dem"
           GROUP "Yukon_Digital_Elevation_Model"
           # set the required metadata for the Atlas
           # see the /docs/yukon_atlas_metadata_management.doc file
           METADATA
             "ows_abstract"           "The Department of the Environment GIS (ENVY GIS) 30 Meter DEM is a ..."
             "ows_attribution_title"  "Government of Yukon - Department of Environment - Geomatics"
             "ows_dataurl_format"     "application/zip"
             "ows_dataurl_href"       "ftp://ftp.geomaticsyukon.ca/DEMs/30m/"
             "ows_group_title"	     "Yukon Digital Elevation Model"
             "ows_keywordlist"        "base,elevation,reference,yukon"
             "ows_metadataurl_format" "text/html"        
             "ows_metadataurl_href"   "http://www.environmentyukon.gov.yk.ca/geomatics/data/30m_dem.html"
             "ows_metadataurl_type"   "FGDC"
             "ows_srs"                "EPSG:4326 EPSG:3578 EPSG:42304 EPSG:42101 EPSG:4269 EPSG:2294 EPSG:2295 EPSG:26922 EPSG:26921 EPSG:26920 EPSG:26919 EPSG:26918 EPSG:26917 EPSG:26916 EPSG:26915 EPSG:26914 EPSG:26913 EPSG:26912 EPSG:26911 EPSG:26910 EPSG:26909 EPSG:26908 EPSG:26907 EPSG:26722 EPSG:26721 EPSG:26720 EPSG:26719 EPSG:26718 EPSG:26717 EPSG:26716 EPSG:26715 EPSG:26714 EPSG:26713 EPSG:26712 EPSG:26711 EPSG:26710 EPSG:26709 EPSG:26708 EPSG:26707 EPSG:2295 EPSG:2294 EPSG:54003 EPSG:54004 EPSG:54008 EPSG:54009 EPSG:54030 EPSG:102016 EPSG:102017 EPSG:102018 EPSG:102008 EPSG:102009" 
             "ows_title"              "Drainage Lines 1M"     
           END
           TYPE RASTER
           STATUS ON
           DATA "../data/yukon/base/dem/50k/dem_hs.tif"
           # layer projection
           # this is the actual projection of the data 
           PROJECTION
             "proj=aea"
             "ellps=GRS80"
             "lat_0=59"
             "lon_0=-132.5"
             "lat_1=61.667"
             "lat_2=68"
             "x_0=500000.0"
             "y_0=500000.0"
             "units=m"
             "no_defs"
           END 
           # set the layer to only display at zoomed-in scales (otherwise it takes 53 seconds
           # at full extents!)
           MAXSCALEDENOM 2500000
           # ignore black missing data values
           OFFSITE 0 0 0
           # style the data  
           CLASS
           END   
           # configure for GetFeatureInfo requests
           # should not be modified, this is just a placeholder to tell MapServer
           # to allow GetFeatureInfo requests (meaning to allow for querying of the
           # map image on the WMS client-side)
           #TEMPLATE "dummy.html"  
         END # layer

WMS/OGC Template

  • use the following file as a template for OGC data sources: /map/layers/wms/geomatics_yukon_borders_1m.lay
    • which contains: 
         #
         # CSW Borders 1m
         #

         LAYER
           # layername must not contain any spaces or special characters
           NAME "geomatics_yukon_borders_1m"
           # groupname must not contain any spaces or special characters
           GROUP "WMS"
           # set the required metadata for the Atlas
           # see the /docs/yukon_atlas_metadata_management.doc file
           METADATA
             "ows_abstract"           "Yukon borders. Source: CSW WMS"
             "ows_attribution_onlineresource" "http://geomaticsyukon.ca/"
             "ows_attribution_title"  "Geomatics Yukon"
             "ows_dataurl_format"     "image/png"
             "ows_dataurl_href"       "http://freehand.gov.yk.ca/wmsconnector/com.esri.wms.Esrimap/cswCatalog_wms?SERVICE=wms&VERSION=1.1.1&REQUEST=GetMap&LAYERS=Borders%20(1M)&SRS=EPSG:4326&BBOX=-170.2190480364,53.657025112,-94.7809519636,77.056656139&WIDTH=400&HEIGHT=400&FORMAT=image/png"
             "ows_group_title"	      "WMS"
             "ows_keywordlist"        "administrative,base,political,reference,yukon"
             "ows_metadataurl_format" "application/xml"        
             "ows_metadataurl_href"   "http://freehand.gov.yk.ca/metadataexplorer/full_metadata.jsp?docId={57DDAD8A-2847-4431-80C1-6CBDBD575103}&loggedIn=false"
             "ows_metadataurl_type"   "FGDC"
             "ows_title"              "Yukon Borders 1M"   
             "ows_name"               "Borders (1M)"
             "ows_format"             "image/gif"
             "ows_server_version"     "1.1.1"
             "ows_srs"                "EPSG:4326"    
          END
          TYPE RASTER
          STATUS ON
          CONNECTION "http://freehand.gov.yk.ca/wmsconnector/com.esri.wms.Esrimap/cswCatalog_wms?REASPECT=false&"
          CONNECTIONTYPE WMS  
          # Layer projection
          # this is the projection advertised by the CSW WMS server
          PROJECTION
            "init=epsg:4326"
          END
          # configure for GetFeatureInfo requests
          # should not be modified, this is just a placeholder to tell MapServer
          # to allow GetFeatureInfo requests (meaning to allow for querying of the
          # map image on the WMS client-side)
          # TEMPLATE "dummy.html"  
        END # layer

Generate .lay Metadata

In order to generate the Atlas' required metadata inside the .lay file (which are documented at MetadataManagement), you can use the  Atlas Layer Metadata Generator.

.lay Guidelines

Atlas guidelines for .lay files (including styling, labelling, scales) must be followed at: DataStyling.

Step 3: Submit for Publication

  • verify that all of the data submission guidelines have been followed, as described in QualityControl#QualityControl.
  • If access to the data is to be restricted to a specified user group, this group should be identified.
  • submit request to the Atlas Administrator, identifying location of .lay file

Step 4: Publishing a Layer

  • publication of layers in the Atlas requires Administrative privileges
  • data and layers files must be properly checked in on the server to be published (FTP and SVN are required tools)
  • Administrator verifies that all of the data publication guidelines have been followed, as described in QualityControl#QualityControl.
  • Administrator adds layer to Atlas DataCatalog
    • login to the  Atlas admin interface
    • on the Layers page, insert a new layer
      • the Layer field must contain:
        • valid path to a new .lay file (relative to the /layers folder)
        • no .lay extension on name in layer field 
             yukon/thematic/250k/sheep_wka
        • .lay file must be checked in on server
    • assign view/extract privileges to appropriate user groups. Once a request for publication has been approved, the layer is added to the Atlas by the Administrator using the Layer Management tool in the Admin Interface. View privileges for unregistered users only affects layers that are included in a published Theme. Layers accessible by unregistered users are also available using WMS, as described in MetadataManagement#GetCapabilitiesRequest.

Step 5: Add Layer to Existing Theme File

Themes are a collection of layer files published as a single map image. A Theme file is identified by a .map file extension. The Theme file contains a template reference and list of included layer files.

  • publication of a new Theme, or modification of a published Theme requires Administrative privileges
  • a layer file cannot be added to a Theme until the layer has been approved and published in the Atlas
  • visibility of a Theme may be restricted to one or more specified user groups
  • layer visibility within a Theme is overriden by layer view/extract privileges

To modify and existing Theme file

  • open an existing .map theme file (in /map/themes/) and add a pointer to the new .lay file, such as: 
       INCLUDE "../layers/yukon/thematic/250k/sheep_wka.lay"
  • NOTE: Include .lay extension on file name

The template reference file (.ref) included the .map file is used to specify items such as the projection and extent of the initial map window, file paths to the web images, and service metadata.

Step 6: Verify Layer Processing

To check what layers have been processed in a theme, you can use  layer_list.phtml.

Refresh Layer Data

Several layers obtained from other sources are regularly updated. For example, the land disposition and mineral claims data from Geomatics Yukon are updated weekly. Updating these layers requires:

  • uploading the current data to the Atlas (NB: Be sure to create a spatial index using the shptree command;
  • modification of the layer file; and
  • testing of the updated theme map.

The use of the ogrinfo command will facilitate confirmation of consistent attribute structure and projection in the updated data file. Note any changes to the data source, or date of data in the Layer metadata (ows_abstract).