On day one of the Sisense bootcamp in Tel Aviv Israel, myself and Ciaran arrived at the Atrium Tower to start our bootcamp. We were immediately wowed by the impressive skyscrapers and towers surrounding us. The building hosting us, had 40 floors and we were on floor 16. We were met by Einat Pick, the organiser of the bootcamp. We had a very welcoming reception where we were given goodie bags and the opportunity to speak to other Sisense partners, who were also attending the bootcamp from all around the world. …
In today’s big data world, we have seen an explosion of data from structured data sources such as databases to unstructured sources such as twitter feeds. This data growth has led to a movement away from the old ways of relational databases to NoSQL flat databases such as MongoDB.
This data growth has also driven the need for distributed systems such as Hadoop to drive the processing of these massive datasets, and where on-premise servers can’t handle this processing organisations are moving to the cloud for greater elasticity and scalability.
But with this data growth, we must ask the question are we any better off? Is this data helping or hindering decision making?
I feel this data growth is off course of great benefit but for us to benefit from this data growth we need to understand our data properly. To understand our data, we need to have metadata or data about our data.
Too many analytical projects fail because of a lack of understanding of the true single source of data, whether it be a single view of a customer or a single view of an asset. With data now stored in many different systems, formats and locations (on premise and in the cloud), using Metadata to better understand this information is essential.
Coming from a geospatial background I have talked about metadata for some years now and at one stage the EU INSPIRE initiative was the driving for force for organisations to manage their Geospatial data better and develop proper metadata catalogues.
Unfortunately INSPIRE never really achieved what it planned to in my opinion but from the 25th of May there will be new GDPR legislation which will drive us all to manage our data better.
|It is easy to scoff at the EU, and to parody the great Monty Python Life of Brian maybe ask ourselves what have the Romans EU ever done for us – except for the money, roads etc.
And it maybe its unpopular to say but most EU legislation is of benefit to the citizens of Europe and in my opinion GDPR is no different.
As we have seen in the last month with the Cambridge Analytica data breach scandal, protecting customer data is of utmost priority through proper data governance (and if you don’t understand your data – how can you protect it). Also, unlike INSPIRE, the GDPR regulations have teeth and any company who is in breach of these regulations will suffer large fines.
But managing metadata isn’t just about meeting GDPR or any other data regulations (and believe me more are coming). If we can understand our data better, we can make better decisions. By making better decisions our organisations can grow sales and cut costs through improved efficiencies. Before you make any decisions on new Big Data analytical projects, ensure you understand your data by implementing a proper meta management strategy in your organisation, and the ROI for your Big Data projects will increase substantially.
If you would like to hear more about our metadata solutions, please listen to my Irish Tech News podcast https://buff.ly/2G22pUM or attend our up coming webinar on how we can easily handle GDPR subject data access right requests using our metadata solutions https://bit.ly/2Fy5gb6
Author: Ciara Beddy, IMGS Senior GIS Engineer
Welcome to part two of Time Saving Tips in FME!
Using FME daily means you pick up on tips and tricks to make your daily tasks run quickly, smoothly and generally just make your FME life a bit easier. This blog is a follow up to last month’s time savings tips part 1.
Tips for Inspecting data
Inspecting your data is a key part of any workspace you develop, with every change you make its good practice to check your output. So it stands to reason FME has a number of tools available to you, all you need to do is use them.
Run with Full Inspection
Do you ever run your workspace, wait for it to finish only to find it’s taken a wrong turn somewhere? Now you need to find out where things went amiss, run with full inspection is the solution for you. This setting will cache the data at each transformer link. Each cache can then be shown in the Data inspector by right clicking on the count.
This feature is great for debugging, no need to keep adding inspectors to source the issue, but don’t forget to turn off this functionality when you are done as it slows down performance and takes longer to run than a normal translation.
Redirect to Inspection Application
When you are still in a testing phase then use “Redirect to Inspection Application”. This means you don’t alter your destination dataset and your output goes directly to the Inspector. You can run your process as many times as you want until you get it just right, the data is never written until you turn off this option. Especially important when you are making edit within a database and you only have one chance to get it right.
One Click Insect
Your readers and writers now allow you to inspect your data with ease, right click on any Reader/Writer to use the inspection button or open containing folder. No excuses for not examining your output it’s easy!
The terminator transformer stops a translation when it detects non-valid situations or input data conditions that should not exist.
This is to stop the output of data which may impact your results, for example you could filter the data’s geometry and terminate if you find points in a dataset you expected to be linear. When a feature enters this transformer, the translation immediately stops and an error message displays to notify you that the data conditions are not expected.
To redirect the Terminator data to an Inspector instead of stopping the translation, you can set this option in the Workspace Properties in the Navigator. This allows you to view the unexpected data during development of your workspace and then you can switch off and reverts back to terminating the translation when expected conditions are met.
Once you are inspecting the data its quite useful to compare to a background map, don’t forget the Data Inspector now provides this functionality under Tools>Options>background maps.
You have a number of options ranging from open source mapping like open street map to the ability to connect to your own AGOL Map Service or Web Map Service. But remember your data will be reprojected to display in the coordinate system of the background map. So don’t get confused like I do….everytime! The actual coordinate system is always specified within the feature information.
Sometimes you can add a file with many layers like a .dgn or Geodatabase. Nothing gives you a bigger headache than having to reconnect all those layers to your writers, particularly when they are located the far side of the canvas. The Feature Type Connections window, found under View > Windows allows you to connect readers to writers with ease. I don’t use it often but when the need occurs it’s a great time saver.
The most popular transformer the Tester is much more than what it seems. You should have noticed by now it has received an update in the last few releases. If not, then pay attention as this can be a huge time saver when performing data quality checks. There are now Operators to filter data on such as Attribute is Null or Empty. New to 2016 is the operator “Attribute has a value”. This means that without knowing all the possibilities of how the attribute could not have a value we only need to check if it contains a value in order to extract all those missing records.
Finally don’t forget you always have options! It’s important to customize your workspaces to your own needs so take 2 minutes to check out your options under the Tools Menu > FME Options. Here you can create a toolbar to your own specifications.
As well as control how your workbench functions, default paths, database connections and how your translation is run. There are plenty of choices here so they are worth reviewing.
Thanks for reading! Keep an eye here for next month’s blog.
Author: Ciara Beddy, IMGS Senior GIS Engineer
Welcome to part one of Time Saving Tips in FME!
Using FME daily means you pick up on tips and tricks to make your daily tasks run quickly, smoothly and generally just make your FME life a bit easier. This blog has been a long time coming, for the last year I have diligently kept note of some of the tools I use in Desktop that save time and because it’s nice to share I am departing these tips to you.
Functions may seem complex as they allow you to delve into the heart of FME’s core and fly solo so to speak. Using them means you are now stepping out on your own and calling functions that are in use in many of the transformers and that seems a bit daunting.
But this really is a smart move on your behalf, by using Feature Functions you can build processing directly into your attribute creation and save yourself the effort of placing multiple transformers. Really you can do A LOT with the text editor in FME, use it to build your queries and send them directly to the engine yourself instead of relying on transformers. When you use FME Functions as part of your attribute manager or attribute creator each resulting value is output as an attribute.
For example, @Area function calculates the area of a polygonal feature eliminating the need for the AreaCalculator. The @Count provides a mechanism for generating unique numbers for features during a translation eliminating the Counter. @Timestamp returns the current time and date, so you guessed it no need for a Timestamper.
Here are some of the common FME Functions I use in my workspaces, why don’t you try them out next time and see how simple it is to develop your own results without the use of multiple transformers.
Also for those experienced FME users out there did you know you can copy and paste a transformer into an open Text Editor and review the content, this allows you to see what really is at work inside the transformers you use.
The ability to save workspaces as templates was added in 2011 though it still isn’t as widely used as I would like! A template is basically a saved workspace, with the inclusion of the data required to run. Using a template means you can bundle everything together and provide to another user….very handy for sharing workspaces with your colleagues.
These templates are given the extension .fmwt, using the template is very simple just open it in Workbench. If data was included in the template file then it will be extracted, stored to a temporary location and the workspace paths updated automatically making it ready to just hit run.
There are many possibilities with templates, they can be great to give out to other FME users as predefined translations. Or when looking for support from the team at IMGS if you provide us with a template you will certainly rise of the top of our to-do list and receive quicker solutions!
While on the subject of templates, if you are reading data from a database and the end user of the template does not have access to this data source then it’s time to introduce a recorder transformer.
The recorder is a nifty little transformer that saves a copy of all the features that enter it and writes into a FME Feature Store file. It’s really simple to use;
- In your workspace identify the data you want to export.
- Add a Recorder transformer
- Connect the Reader feature types you need to export to the Recorder transformer
- Run your workspace and the data is stored to the FFS file at your specified location
Now you can send a copy of your workspace and the FFS file. The end user can then add a Player transformer to your workspace and play back your recording – as if they were reading directly from your database. Everyone wins!
Most people probably know everything they need to know about the data fanout, it has been around a long time. But when it comes to listing features that save you time and effort I have to include the fanout. This is a way you can split your output data into separate layers/files based upon an attribute value.
It’s easy to set, just go to the writer properties right click and pick an attribute value. So your data output will be divided on the fly as it writes to your destination.
For example, I want to create a set of contours from my LIDAR dataset. But I want to have a separate output file for every elevation. If I set the output to fanout by the value of the elevation my results will be quite different.
Instead of one file containing all contours your destination output will now contain multiple files one for each elevation level.
People always say there are no shortcuts in life, well that’s not quite true. Here are some custom keyboard shortcuts which I promise will make your life easier, no risk involved!
|Add New Bookmark||Ctrl+b|
|Zoom to 100%||Ctrl+0|
Another one of my favourites is the quick change from zoom in to select. Try it yourself, Zoom in and right click this will resets the cursor back to the select function. It’s the simple things in life.
Stay tuned for the follow up part two blog of Time Saving Tips in FME which will be published next month.
Author: Ciara Beddy, IMGS Senior GIS Engineer
For a new user looking at FME’s hundreds of available transformers, many with some interesting and unusual names, locating the transformer you need to get the job done can often feel like a difficult or daunting task.As an FME professional and Trainer I often get told in panic “I don’t think I will know what Transformer to use, there are so many!!” and that is true. In reality you can still be a proficient user of FME without knowing the ins and outs of every single Transformer. That is, if you arm yourself with the most useful ones!
Transformer knowledge comes down to experience. The more workspaces you create and projects you complete the larger your catalogue of Transformers becomes. To get new users started I would like to share with you a list of some of the unsung heroes of the transformer world, these are some of the tools I really couldn’t live without.
I am most often asked about how to quickly move spatial data between different coordinate systems. The answer is very simple, use the Reprojector. FME’s coordinate reprojection capabilities solve the common requirement to reproject data accurately and seamlessly. All you are required to do is place the transformer somewhere within your workflow.
This transformer works on both raster and vector data, It provides the option to reproject data with a library of over 5000 coordinate reference systems. This gives you the ability to reproject data with the click of a button eliminating all those coordinate system headaches you face as a spatial data user.
The Clipper is an excellent transformer for creating datasets that are relevant to your needs. Often users have national datasets containing more data than they actually require. The Clipper takes a number of clip boundaries and the number of features to be clipped and splits the data into features inside the boundary and features outside the boundary.
As the output is split into these two possible groups the Clipper allows you to keep the data relevant to your needs while disregarding the data that falls outside your defined boundaries. It is a great way of cleansing your datasets to only contain the data you will use, meaning any further analysis on these datasets are correct and more meaningful to your area of interest.
If you ever come across the need to provide a unique ID to the data running through your workspace than the Counter is the most simple way to achieve this. This transformer adds a numeric attribute to your features by assigning each feature an incremented value as it counts the features passing through.
You can also specify if this counter sequence is unique throughout the whole workspace by using the same sequence on multiple counter transformers or using a unique sequence to that single counter. This transformer means you can create unique values without using any complex expressions, calculators or calling database sequences.
Another key quality control task it to check your data for duplicates, sometimes no matter how ‘perfect’ you think your data is, an unwanted duplicate can creep in during a processing task. FME gives you the ability to ensure that your data is clean by eliminating all those duplicate objects so only one remains. The answer comes in the form of the appropriately name DuplicateRemover. This transformer allows you to select one or multiple key attribute values that you require to be unique. Any key attribute value that has already been encountered in the DuplicateRemover is sent to a separate port labelled as a duplicate.
Allowing you to maintain the data stream coming from the unique port, giving you confidence in the quality of your output data.
Lastly, users are often tasked with the requirement to provide labels with their output data. There are a number of options in FME for labelling the most basic being the LabelPointReplacer. This transformer enables you to quickly configure labels and rules that will dictate how labels appear.
This transformer replaces the geometry of the coming feature with a label point while also giving you a number of options to define the label value.
If the feature was already a point, it is turned into a text feature at that location.
- If the feature was a line, it is turned into a point text feature at the midpoint of the line and rotated parallel to the line.
- If the feature was an area feature, the resulting point is placed in the interior of the feature.
The LabelPointReplacer removes the hassle of creating labels for your spatial data and should be your first stop when tackling label requirements.
These are some of the key transformers I regularly use when tackling data problems. Hopefully it will give new users a head start on which of the transformers they may want to try out in the future. But don’t forget if you are ever in doubt or lost on what is the transformer for the job;
- Use the associated Help File– Every transformer you select will link to the help file which pops up in the help tab in your workspace. If you are unsure of how to use a certain transformer the help file gives you detailed information on the use of the transformer and how to configure the parameters, input and output ports.
- Use the Transformer search functionality in the Transformer Gallery to pinpoint your possible transformer options. This searches in both Transformer name and description so your search term may be the exact name of a transformer, or it may be a general keyword referring to functionality.
- If all else fails then just try it out, trial and error can often be the best option in understanding exactly how each transformer effects your data. Don’t worry you won’t do any damage to your dataset and can run your workspace as many times as you wish. Just connect an inspector transformer to the end of your workflow instead of connecting it to the writer and you can keep testing until you get it right!
Also help is always at hand just contact us at email@example.com for FME related queries we are happy to point you in the right direction when you are lost in a maze of Transformer options.
Author: Vijay Sambhe, Senior Applications Engineer at IMGS
Hexagon’s Geospatial Portal – Locate a coordinate in the Map
Hexagon’s Geospatial Portal is a full-featured, configurable and customisable thin client application that can be used for finding, viewing, querying, analysing, and consuming geospatial data published by Hexagon Geospatial products and/or other standard-based web services. The portal can be accessed by multiple users via web browsers as it is installed and configured on a web application server.
When the Locate button is clicked the map zooms in to the location and also it shows the message as shown below.
Plenty of help and code examples are available on Hexagon’s Knowledge base here.
Author: Ciarán Kinane, Applications Engineer at IMGS
Hello everyone and welcome to this month’s blog. This month we are going to focus on map based asset management using hexagon’s GeoMedia Smart Client.
Nowadays people can see the value of GIS within your organization, but they often lack the expertise to run a desktop GIS. Online maps delivered by a web GIS solution are helpful, but these implementations in many cases do not support specific geospatial workflows required by the user. They also may not provide some of the main functionality of a desktop GIS such as snapping etc. Today, users look more and more for market-focused workflows specified to their needs.
The vast majority of people who could really benefit from geospatial data need more than what a lightweight, browser-based web map can deliver. Likewise, they don’t have the time or expertise to learn a technically demanding desktop environment. These users are eager to geospatially accelerate their business workflows and ignite the power of GIS. This requires a new approach, working outside limiting departmental deployments of GIS that have plagued organizations for decades.
This is where GeoMedia Smart Client comes in. It provides a platform for users which is easy to use and does not require the same training or knowledge required by a desktop GIS. Smart Client is delivered to users via a browser and runs on the client machine using Java. The thing I really like about Smart Client is that it only requires 1 license whether you have 5 users or 500 users running concurrently. Smart Client has the ability to open up GIS to your whole organisation.
Here in IMGS we have provided separately configured asset management systems to both Shannon Properties PLC and Sligo County Council using Hexagon Geospatial’s Smart Client. Out of the box functionality provides, measurement, dimensioning, snapping, redlining and configurable printing, but the real power in Smart Client is the ability to create interactive workflows to meet the user’s needs.
For Sligo County Council we used Smart Client to replace their fully customized map based asset management solution. Using Workflow Manager, which comes with the professional version of Smart Client, we were able to create an Asset Search workflow.
The Asset Search has the ability to filter on multiple attributes within the asset database. For each entry returned in the search, we have the ability to map to, edit or view that asset and sort by any attribute. The asset list can also be printed to csv, excel, word or PDF and can include XY coordinates of the asset, criteria which are a requirement of the Office of Public Works in Ireland.
We also created a workflow which allows the user add a new asset to their system. When the user selects the ‘NEW’ button they are presented with a form whereby the user can enter details pertaining to the asset. The workflow can have many rules to prompt and aid the user with the compiling of asset details. For example, the user can enter no more than 5 digits as the asset number, or the asset ID can be created based on the asset number with a prefix coincident with the asset type specified in a drop down box.
Once the user has entered in the minimum details, they can begin to digitizing the asset in the map window.
While digitizing, the user is afforded many features which would not be available within a web based mapping solution, such as snapping to vector data (as seen on the image below), dimensioning and the ability to scroll the map while in capture mode etc.
Various geometry rules can be applied at this point to guide the users. For instance in the Shannon Properties project they requested some rules which would not allow the user to capture the boundary of a building outside the boundary of a particular site. Another rule requested was to not allow the user to capture the boundary of a site outside a particular Landbank feature. These rules can be applied and guides set in place to direct the user reducing anomalies in the data and standardising your digitizing processes.
Once captured in the map window, the user is brought back to the workflow where they can enter additional details into attribute fields. The form is comprised of tabs along the top which contain different aspects of the asset such as terms of leasing etc. The area of the polygon is calculated in the database and added into the form. A specific request from Sligo County Council was to show the area measurement in hectares if the area was above 5000 meters squared to have it shown in squared meters otherwise.
The workflow can be configured to calculate and report such things as the townland or townlands the asset was captured in via a spatial query. Adjoining streets or roads can be also calculated in this way and a report can be produced and printed with the associated planning applications.
Other features within the workflow are links to external applications within or outside the organisation such as iDocs., the ability to upload files or images, and the ability to add notes and flag one of the notes so that it then appears as the tooltip for the asset in the map.
Various actions can be added to the map which can directly open any workflow, for instance the image shows the ability to be able to copy an existing land parcel to one of your assets just by highlighting the land parcel and right clicking the mouse.
Smart Client has some neat features around bookmarking, you can bookmark an area of interest, save it and then e-mail it to other users or you can share the bookmark across your organisation so it is available to all users.
Access to Smart Client can be based on a windows user login and it not only has the ability to restrict users to the application but it also has the ability to restrict windows users to Smart Client Projects, Features, Workflows and even functions within a workflow. So where one user can have the ability to edit an asset other users can be restricted from doing so within the same project or view.
In addition Smart Client can be configured to launch standalone workflows, for instance if a user wants to just add attribute data to an asset they can do so by launching a standalone workflow without having to launch the GIS application.
Overall, Smart Client’s most prominent capabilities are the ability to supply a GIS application to everyone in your organisation without having to license each seat. With a shallow learning curve and the ability to provide customized workflows there is something for everyone in the audience!
For further information on Smart Client please feel free to contact me firstname.lastname@example.org