Ex2 - Custom reader

Introduction

You work in a land surveyor firm and regularly receive data coming from the public administration. This data is a text file containing GPS coodinates (positions and other attributs). This type of file is inserted into your land measurement devices (GPS, Theodolite), to have access to the official reference points during your measurement session. Of course, the initial format must be adapted.

In this exercise, we would like to be able to read the following text file :

As you can see, this file has the extention ".ptp". Unfortunately, this type of file does not exist as a regular reader.

The only solution for us is to create our own reader !

Objectives

In this exercise, we will develop our own python custom reader in order to read the .ptp file into FME.

Workspace description

We will start with an empty workspace.

Exercise

1) Start Workbench

2) Add a parameter

Add a new published parameter :

Define the parameter as follows (you can set up the path of the ptp as "default") :

A new parameter has been created, we will use it as the path when reading the data file.

3) Add a PythonCreator

Just type PythonCreator in the canvas (central part of the workbench) and press Enter to insert the transformer.

If you open the transformer, you will see the default python code template.

import fmeobjects

class FeatureCreator(object):

    def __init__(self):

        pass

    def input(self,feature):

        # input method must be defined

        #

        # output features here to create the features

        # before reader features have been processed

        newFeature = fmeobjects.FMEFeature()

        self.pyoutput(newFeature)

    def close(self):

        # close method must be defined

        #

        # output features here to create the features

        # after reader features have been processed

        pass

4) Add an Inspector just after the PythonCreator

If you run the workspace, you will see that an empty feature has been created. It is because the default python code creates an empty feature :

newFeature = fmeobjects.FMEFeature()

and outputs it :

self.pyoutput(newFeature)

5) Read the file

To read the .ptp file, we add the following lines to the input function :

    file = FME_MacroValues['source_file']

        with open(file, 'rb') as lines:

            for line in lines:

                fmeobjects.FMELogFile().logMessageString("--> %s" % line)

Note that "FME_MacroValues['source_file']" parameter can be added just by double clicking in the left panel

The FMELogFile() will output a text message in the log file.

The code should look like this :

import fmeobjects

# Template Class

class FeatureCreator(object):

    def __init__(self):

        pass

    def input(self,feature):

        # input method must be defined

        #

        # output features here to create the features

        # before reader features have been processed

        file = FME_MacroValues['source_file']

        with open(file, 'rb') as lines:

            for line in lines:

                fmeobjects.FMELogFile().logMessageString("--> %s" % line)               

        newFeature = fmeobjects.FMEFeature()

        self.pyoutput(newFeature)

    def close(self):

        # close method must be defined

        #

        # output features here to create the features

        # after reader features have been processed

        pass

Run the workspace again and check into the log window that the lines have been read.

6) Define the attributes

In the loop, we can now define the corresponding attributes.

The '.ptp' format is a strict column based text file. For exemple, the value between the column 25 and the column 26 is the reliability of the point. Therefore it is quite easy to set up the attributes just knowing the start and end index.

Copy and paste the following code into the loop :

                newFeature = fmeobjects.FMEFeature()

                # Attributes definition

                newFeature.setAttribute('theme', line[0:2])

                newFeature.setAttribute('commune', line[2:6])

                newFeature.setAttribute('plan', line[6:10])

                newFeature.setAttribute('point', line[10:22])

                newFeature.setAttribute('nature', line[22:24])

                newFeature.setAttribute('fichier_pfp3', line[24:25])

                newFeature.setAttribute('fiabilite', line[25:26])

                newFeature.setAttribute('precision', line[26:31])

                self.pyoutput(newFeature)

The code should now look like this :

import fmeobjects

# Template Class

class FeatureCreator(object):

    def __init__(self):

        pass

    def input(self,feature):

        # input method must be defined

        #

        # output features here to create the features

        # before reader features have been processed

        file = FME_MacroValues['source_file']

        with open(file, 'rb') as lines:

            for line in lines:

                fmeobjects.FMELogFile().logMessageString("--> %s" % line)

                newFeature = fmeobjects.FMEFeature()

                # Attributes definition

                newFeature.setAttribute('theme', line[0:2])

                newFeature.setAttribute('commune', line[2:6])

                newFeature.setAttribute('plan', line[6:10])

                newFeature.setAttribute('point', line[10:22])

                newFeature.setAttribute('nature', line[22:24])

                newFeature.setAttribute('fichier_pfp3', line[24:25])

                newFeature.setAttribute('fiabilite', line[25:26])

                newFeature.setAttribute('precision', line[26:31])

                self.pyoutput(newFeature)
        
        newFeature = fmeobjects.FMEFeature()

        self.pyoutput(newFeature)

    def close(self):

        # close method must be defined

        #

        # output features here to create the features

        # after reader features have been processed

        pass

And remove these two lines that creates the empty feature (only the one ouside the loop) :

        newFeature = fmeobjects.FMEFeature()

        self.pyoutput(newFeature)

The code should now look like this :

import fmeobjects

# Template Class

class FeatureCreator(object):

    def __init__(self):

        pass

    def input(self,feature):

        # input method must be defined

        #

        # output features here to create the features

        # before reader features have been processed

        file = FME_MacroValues['source_file']

        with open(file, 'rb') as lines:

            for line in lines:

                fmeobjects.FMELogFile().logMessageString("--> %s" % line)

                newFeature = fmeobjects.FMEFeature()

                # Attributes definition

                newFeature.setAttribute('theme', line[0:2])

                newFeature.setAttribute('commune', line[2:6])

                newFeature.setAttribute('plan', line[6:10])

                newFeature.setAttribute('point', line[10:22])

                newFeature.setAttribute('nature', line[22:24])

                newFeature.setAttribute('fichier_pfp3', line[24:25])

                newFeature.setAttribute('fiabilite', line[25:26])

                newFeature.setAttribute('precision', line[26:31])

                self.pyoutput(newFeature)

    def close(self):

        # close method must be defined

        #

        # output features here to create the features

        # after reader features have been processed

        pass

Before running the workspace, expose the newly created attributes :

Run the workspace and verify that the attributes have been created

However we still don't have any geometry !

7) Create the geometry

Within the loop, the point geometry can be created as follows :

                # Geometry definition

                point = fmeobjects.FMEPoint()

                y = float(line[32:43])

                x = float(line[43:55])

                point.setXYZ(y, x)

                newFeature.setGeometry(point)

If you run the workspace again , the inspector will now display the points as well as the attributes :

8) Set up the coordinate system

You may have noticed at the lower corner of the Data Inspector that no coordinate system has been defined.

The last step will be to set the coordinate system. Just add this in the loop :

                # Coordinates system definition 

                newFeature.setCoordSys('EPSG:21781')

The final code looks like this :

import fmeobjects

from fmeobjects import FMEReprojector

# Template Class

class FeatureCreator(object):

    def __init__(self):

        pass

    def input(self,feature):

        # input method must be defined

        #

        # output features here to create the features

        # before reader features have been processed

        file = FME_MacroValues['source_file']

        with open(file, 'rb') as lines:

            for line in lines:

                fmeobjects.FMELogFile().logMessageString("--> %s"%line)

                newFeature = fmeobjects.FMEFeature()

                # Attributes definition

                newFeature.setAttribute('theme', line[0:2])

                newFeature.setAttribute('commune', line[2:6])

                newFeature.setAttribute('plan', line[6:10])

                newFeature.setAttribute('point', line[10:22])

                newFeature.setAttribute('nature', line[22:24])

                newFeature.setAttribute('fichier_pfp3', line[24:25])

                newFeature.setAttribute('fiablite', line[25:26])

                newFeature.setAttribute('precision', line[26:31])

                # Geometry definition

                point = fmeobjects.FMEPoint()

                y = float(line[32:43])

                x = float(line[43:55])

                point.setXYZ(y,x)

                newFeature.setGeometry(point)

                # Coordinates system definition 

                newFeature.setCoordSys('EPSG:21781')

                self.pyoutput(newFeature)

    def close(self):

        # close method must be defined

        #

        # output features here to create the features

        # after reader features have been processed

        pass

Well done ! The file is now correctly read into FME. You can start building your workspace !

Advanced task

You may have noticed that :

1. The attributes sometimes have empty spaces at the beginning or at the end
2. The attributes are all defined as strings, and not as integers or floats

Could you please try to improve on that ?