imgsblog

FME 2022.1

FME 2022.1 New Features

The eagerly awaited FME 2022.1 release is now available for download from Safe Software.  FME 2022.1 makes new things possible with more speed and simplicity to help your organisation make effective data-driven decisions. So dive in and see what’s new!

Enhance Collaboration And Communication

Visually Compare And Merge Workspaces

This exciting new feature allows users to collaborate with team members and maintain workspace updates in a simple and intuitive interface. Now, you can compare two workspaces side-by-side to see the changes that were made and instantly merge changes together. Easily manage and review your work in a secure and reliable environment, without leaving Workbench. And the best part?  No coding required.

Identify Runtime User

Safe Software have made it easy for admins and authors to see who accessed and ran an authenticated Workspace App in the Jobs history page. Gain visibility into user data to troubleshoot or send timely communications to the right people.

Improve Data Quality

Achieve high data quality and reduce storage cost with dynamic control of large datasets in your enterprise systems. Take the guesswork out of the process and produce best-fit data outputs at runtime with SchemaScanner, helping you manage active metadata and detect schema drift.

The enhanced EsriReprojector transformer now supports vertical and time-dependent coordinate systems. Make time-dependent transformations and position your data within an epoch.

FME 2022.1 Improve Data Quality

Ultimate Time Savings

FME 2022.1 Ultimate Time Savings

FME 2022 continues to drive speed to insight to help you advance your spatial analysis and unlock new opportunities. We’ve fine-tuned the inner workings of FME to help you tackle datasets of any size, enabling you to make data-driven decisions and achieve business results faster and easier than ever before.

You can now update records in one call with UPSERT* when writing to a database, helping reduce workflow complexity, increase efficiency, and improve data integrity.

Flexible Automation At Scale

Allocate resources where you need them the most. With the addition of Server Analytics, you’ll gain increased control over engine assignments to help you optimize future workloads. See historical data of running jobs at various time intervals and uncover trends that allow you to know when and where to add additional data processing resources.

FME 2022.1 Flexible Automation at Scale

Increased Organisational Security

With FME 2022.1, Safe Software have introduced the ability to centralise user authentication with SAML 2.0. Integrate FME Server with your existing identity providers and manage credentials through one centralised system.

Hrvoje Technical Lead Staff Profile

Hrvoje Vukasinovic Staff Profile

Hrvoje Vukasinovic Staff Profile


Hrvoje can you describe your role at IMGS?

I’m a technical Lead in IMGS, and work with our customers to solve their GIS/IT issues.

Before working at IMGS, what was the most unusual or interesting job you've ever had?

My most interesting job, has to be my previous role as Information Manager at Dublin Airport.  The airport is such a vibrant place to work, with lots of variety and opportunity for continuous learning.  

While working at the airport, I was lucky to get the chance to visit the runway and then go up to the air traffic control tower to observe flights as they take off and land.

What do you like to do in your spare time?

I have tons of different interests: Travelling would be at the top of that list (which I really missed during the pandemic), tennis, swimming, going to the sauna, hiking, and collecting soccer jerseys!, 

What would you do for a career if you weren't doing this?

When I was a kid, I dreamt of becoming an Architect, as I was obsessed with building stadiums out of small wooden bricks.  However, now I could very much see myself as a tourist guide based in some sunny Mediterranean country.

What's the best place that you've travelled to and why?

If I had to choose just one place, it would be Velebit mountain in Croatia.  It’s such a calm, peaceful place with spectacular views of the Adriatic Sea and surrounding mountain ranges.

What's the last app that you downloaded?

Revolut.  I put off signing up for some time, but finally succumbed and now can’t live without it!

What do you enjoy most about working at IMGS?

I enjoy being part of such a supportive team here in IMGS.  I love the variety of my role and the challenge of finding solutions to issues as they arise.

AGI Northern Ireland Annual Conference

IMGS Platinum Sponsor AGI-NI Conference 2022

IMGS
Platinum Sponsor
AGI-NI
Conference 2022

AGI Northern Ireland Annual Conference

IMGS are delighted to announce that we will be attending this years AGI-NI Conference as Platinum Sponsors.  The AGI-NI Conference is an annual event, and a key focus for GIS professionals across Northern Ireland.   The theme of this year’s conference is “Geographic Information in Support of a Better World”.

Taking place on Thursday 23 June 2022 conference speakers will touch on topics including the use of GI to support sustainable spatial infrastructures, public health, the environment, and technological innovation.

To register for this event click the button below.

IMGS Presentations at AGI-NI 2022

Ciaran Kirk, Operations Director IMGS

Ciaran Profile Headshot cropped

Ciaran Kirk, Operations Director at IMGS will present “EU Fisheries Protection by the Irish Naval Service”.

This Presentation will outline how the Irish Naval Service are using Hexagon M.App Enterprise to analyse live fishing vessel information both on-shore and on Naval vessels to enforce these policies.

Garrett Cronin, Commercial Manager IMGS

Garrett Cronin, Commercial Manager, IMGS

Garrett Cronin, Commercial Manager IMGS will present a short 5 min overview on “Geospatial Data Integration”.

This presentation will highlight how data is a business critical asset in every organisation, but working with data can be challenging.  You need the right integration solution to turn data into insights to better serve modern customers.  The FME Platform powers business decisions with enterprise integration and no-code automation.  Spend more time using data, and less time fighting it.

AGI NI Conference Agenda

AGI NI 2022 Programme of Events

Conor McIntyre Staff Profile

Conor McIntyre Staff Profile

Conor can you describe your role at IMGS?

My role in IMGS is as Business Development Executive. I work with both new and existing customers to help them understand their data challenges. Together, using our IMGS Data Intelligence Platform, we create solutions to improve information sharing to curb the inefficiencies, visualise data and empower organisations to do more with their data.

Before working at IMGS what was the most unusual or interesting job you’ve ever had?

I worked 12-hour shifts in a bacon-only sandwich shop during my time in Toronto when I was 21. I’m instantly teleported back to those days when I smell bacon still.

What do you like to do when you aren’t working?

I try to keep as active as possible, hitting the gym and playing sports. I am captain and founder of a basketball club in Ballinteer.

If you could meet anyone in the world, dead or alive, who would it be and why?

Aside from my nan just one more time, I would say Michael Jordan, I’m sure I could find a challenge that would pique his betting interest and that I may stand a chance in winning too!

What is your favourite holiday destination?

Sun holiday: I love Greece. If I could go anywhere; New York wins every time. I’m hoping to go again early next year. I’m sure I’ll come back….maybe.

What’s the last book that you read?

The 100-year-old man who climbed out of the window and disappeared. Title is a mouthful, but (let me put my leaving cert English to use here) serves a foreboding measure for the story as a whole! Brilliant book and while the title gives the essence of the story, it follows into his life very much like Forrest Gump (favourite film by the way) and coincides with significant world events. 10/10 would reccommend (movie not so good, watch Forrest Gump 😊)

What do you enjoy most about working at IMGS?

It has to be the team; everyone is so close which makes for a great atmosphere to work in.

FME Platform 2022.0

FME 2022.0 New Release

FME 2022.0 New Release

Enhance Collaboration & Communication

FME 2022.0 is packed with new features that are designed for performance, collaboration and flexible automation that will help organisations make data-driven decisions faster and more effectively!

Visually Compare and Merge Workspaces

This exciting new feature allows users to collaborate with team members and maintain workspace updates in a simple and intuitive interface. Now, you can compare two workspaces side-by-side to see the changes that were made and instantly merge changes together. Easily manage and review your work in a secure and reliable environment, without leaving Workbench. And the best part? No coding required.

FME 2022.0 New Feature

Identify Runtime User

Safe Software have made it easy for admins and authors to see who accessed and ran an authenticated Workspace App in the Jobs history page. Gain visibility into user data to troubleshoot or send timely communications to the right people.

Improve Data Quality

Achieve high data quality and reduce storage cost with dynamic control of large datasets in your enterprise systems. Take the guesswork out of the process and produce best-fit data outputs at runtime with SchemaScanner, helping you manage active metadata and detect schema drift.

The enhanced EsriReprojector transformer now supports vertical and time-dependent coordinate systems. Make time-dependent transformations and position your data within an epoch.

Improve Data Quality

Ultimate Time Savings

FME 2022 continues to drive speed to insight to help you advance your spatial analysis and unlock new opportunities. Safe Software have fine-tuned the inner workings of FME to help you tackle datasets of any size, enabling you to make data-driven decisions and achieve business results faster and easier than ever before.

Additionally, you can now save time with UPSERT*. Synchronise and manage data in near real-time and simplify your workspace, all with no code! 

*Support in 2022.0 includes PostgreSQL, PostGIS, PostGIS Raster, SpatiaLite, SQLite, with more to come in the Summer Release.

Flexible Automation at Scale

Allocate resources where you need them most.  With the addition of Server Analytics, you’ll gain increased control over engine assignments to help you optimise future workloads. See historical data of running jobs at various time intervals and uncover trends that allow you to know when and where to add additional data processing resources

Flexible Automation at Scale

Increased Organisation Security

With FME 2022, Safe Software are introducing the ability to centralise user authentication with SAML 2.0. Integrate FME Server with your existing identity providers and manage credentials through one centralised system. 

New Formats & Systems

Unfolded Studio

Read or write to and from Unfolded Studio using FME. Visualise data in your browser. Also available for download on FME Hub

Unfolded Logo

Precisely Maps

Use FME to overlay your data points over a Precisely background map.

Truvision TGA

Read TARGA raster images and apply textures to older 3d animation models.

Targa

Flexible Image Transport System (FITS)

Read and write to this versatile image and data table container format recognised by NASA and IAU.

FITS

SpatiaLite

SpatiaLite (GDAL) and SpatiaLite (FDO) is now one centralized format in FME: SpatiaLite

Spatialite

OGC Catalogue Service for the Web

Read XML catalogues of geospatial records for the web.

OGC
Ciaran Kirk, Operations Director IMGS Headshot

Ciaran Kirk

Ciaran Kirk Staff Profile

Ciaran can you tell us about the origins of IMGS?

In 2002 IMGS was founded by Bob Stuart, our current managing director, as the Intergraph Mapping and Geospatial reseller for Ireland. At the time, the LGCSB (now LGMA) had selected GeoMedia WebMap as the preferred web GIS for the Local Authorities in the Republic.

I was hired as the first engineer in, to install and support the roll out of the solution. At the time I had no GIS experience and was a programmer, so it was a bit of a baptism of fire, but I tried to hit the ground running and the rest as they say is history!

In that first year we were also selected by the Irish Naval Service to provide their Fisheries Protection GIS System (in partnership with Hewlett Packard), which was our first major project and required us to expand our engineering team.

By the end of the year Intergraph had then asked IMGS to support their Utility & Telecommunications (U&C) division in Ireland with existing customers such as Eir (then Eircom), ESB Networks, BT Ireland, and the Dublin Regional Water GIS.

The U&C business was again another steep learning curve for us but we invested time in learning & upskilling and really understanding our customers business so that we could deliver an excellent support service.

Then in 2003 we were delighted to become the Safe Software FME distributor for Ireland (on the advice of a customer who recommended the solution – thank you Claire) which has been at the core of our business and projects ever since and is used by over 100 organisations across the island.

How has your role evolved with IMGS over the past 20 years?

Aoibhinn Stuart and Ciaran Kirk Business Feature

I have had several roles in IMGS – Engineer, Technical Manager, Technical Director, General Manager and now Operations Director.

Over the years as our team has grown, I have moved more to managing and mentoring our engineers and doing less & less of the hands-on work (but I still like to get my hands dirty when the team let me)!

On our major projects I also support our customers by serving on project boards to advise the project team on ensuring project success. We have really improved our project delivery since our Delivery Director, Aoibhinn joined IMGS.

Aoibhinn has added some much-needed process to our projects but even with this improvement, projects will always have their challenges so usually my hard-earned experience can provide guidance where needed!

I have also moved towards the commercial side of the business over the past decade which I have really enjoyed – I love working with customers to solve their problems using our solutions. I am a great believer in science & technology and that technology can solve most problems (I think the last year has proven that more than ever)!

You recently launched the IMGS Data Intelligence Platform. Can you describe this offering to us and explain the change in company strategy this past 2 years?

Zero to Dashboard

As a company we have always understood the power of data analytics and visualisation through our GIS platforms from Hexagon and Safe Software.

But with the explosion of big data, we could see the enormous potential for organisations that want to become data driven across the enterprise, and we are really excited by the potential.

Aoibhinn who had come from the financial sector could also see how our FME ETL skills were of great demand and how we would easily transfer these skills to this area.

Our Data Intelligence Platform provides a complete data management and analytics platform that works across any industry who wants to become data driven.

We are delighted to be working with our partners Snowflake, Talend & Sisense who are disturbing the industry with game changing innovation in Big Data and AI.

It has again been a learning curve for us to embrace this new area, but I personally love it and I am so excited by the change (to the fact that I constantly bombarding the team with the latest cool changes, and they are amused with my excitement even at my age).

What do you see as the greatest opportunities for the business over the next 5 years?

The IT industry is changing. I feel the days of large multi-year projects with big capital software expenditure are over.

Customers are now looking towards more flexible SAAS solutions that require minimal development. The 2 – 3 year projects are being replaced with 3-month implementations where customers expect results next quarter not next year.

The move to the cloud has been critical in this SAAS approach, which has decreased the infrastructure costs and implementation times and provides huge scale and accessibility.

In the past years we have had several new BI customers who were able to onboard Sisense cloud as a full BI platform within a quarter. They were able to do this because the Sisense solution met their requirements off the shelf, could be provisioned next day and was easy to integrate with their existing systems and data flows.

This flexibility is what customers will demand going forward.

In IMGS with our partners platforms we have the cloud-based solutions and the agile approach to support our customer demands. So far in the 20s IMGS have already seen considerable growth through supporting our customers digital transformation and I expect this to continue as we all succeed together.

Cloud

What do you enjoy about working at IMGS?

It’s a bit of cliché to say in a blog but we have great customers who, even though we have had our ups and downs, I have made some great friends with over the years.

I am truly proud of what we have done with our customers. For example, in a 6-month project we had supported Siro in rolling out their FTTH GIS system from green field to fully integrated solution – that was an extremely challenging project, but we did it by working closely with a wonderful team in Siro.

I have worked with customers across the country from ESB to NIE Networks, OSI to LPS, Dublin City Council to Shannon Airport Authority and I feel confident that I have made relationships that will outlast any software system and I have enjoyed every minute of it.

The other big thing in IMGS that I love – is the constant change. I usually get to be the first to get my hands on our new technology. This constant evolution means it’s never boring and as a tech geek I love the challenge of new systems (and I am loving our move to Sisense, as I said already).

Finally, the staff in IMGS make it a fun place to work and as the team gets younger and I get older I can safely say I laugh more today than I ever did (no coincidence this coincided with Garrett’s arrival a few years ago)!

If you could do another job just for a day what would it be?

I have always loved sport (never been much good at it) but as an avid Everton Football and Ireland & Leinster Rugby fan, being involved with one of these would be a dream come true for me.

Now that the world is opening up again, where would be the first place overseas that you would like to visit on holiday?

HXGN Live 2022

Before Covid I always enjoyed travelling to see our partners and attend the conferences and sales meetings and I really looking forward to that starting again (Las Vegas is calling for HXGN Live this summer).

Just before Covid I visited Tel Aviv to meet our friends in Sisense but didn’t get time to see the country and would love to spend more time there.

On the bigger scale I would love to go back to Cancun and have always wanted to see New Zealand (especially as I am a big Lord of the Rings Fan).

Find Out More

At IMGS, Data Intelligence and information are at the core of what we do.  We provide solutions to automate data flows, visualise information and power data insights for a wide variety of customers included Local Authorities, Government Agencies, Utilities and Communications Organisations.

Our data solutions are built on our partner’s technology platforms and serve the citizens of Ireland every day, from managing water and electric networks to publishing key e-Government information through Government Portals.  Through our market leading partners, we provide organisations with end to end solutions that drive operational and capital efficiencies.

To learn and to arrange an initial data session with us, please CLICK HERE to fill in our contact form.

IMGS

IMGS Year In Review 2021

2021 was a very busy and prosperous year for IMGS and we would like to share some of the key highlights with you in our “IMGS Year in Review Blog”.

February - Location Intelligence 2021

We hosted our first virtual event of the year: Location Intelligence 2021 with excellent presentations from Hexagon on the latest M.App Enterprise Updates and from our own Ciaran Kirk, Operations Director IMGS, who focused on Predictive Analytics using M.App Enterprise and how Photogrammetry is being used in Land and Property Services in Northern Ireland.

Photogrammetry in Land and Property Services
FME Desktop 2021.1

March – FME 2021 Sneak Peak

Garrett Cronin, Commercial Manager IMGS and Gavin Park, Lead Solutions Consultant IMGS hosted a webinar in early March giving us a sneaky peak of all that is cool and new in FME 2021!

May – FME World Fair 2021 Sponsorship

IMGS were delighted to be one of the partner sponsors for the Safe Software FME World Fair that ran 4-14 May 2021.  On the 2nd day of this event Garrett Cronin and Gavin Park teamed up again to present “Infusing & Embedding Analytics with FME, Snowflake & Sisense”.

Also in May, IMGS Operations Director Ciaran Kirk, took part in a webinar hosted by The Analytics Institute of Ireland which examined how the adoption of analytics are greatly increased by the embedding of analytics into enterprise systems.

June – Platinum Sponsor at the AGNI 2021

The AGNI Conference was hosted virtually on June 17th with the theme “Geographic Information: Supporting a sustainable future”.   IMGS, as one of the Platinum sponsors had the opportunity to address attendees and Ciaran Kirk, Operations Director IMGS delivered a presentation within the sustainable theme titled – “Solar Farm Monitoring with Hexagon M.App Enterprise”.

July – Launch of Zero to Dashboard in 3 Months

IMGS were featured in the Irish Tech News Business Showcase on July 17th to highlight the launch of our Zero to Dashboard Platform.  

To coincide with the launch of our new platform we also produced a Zero to Dashboard webinar providing an overview of our Data Intelligence platform solution and the benefits it delivers.

Aoibhinn Stuart and Ciaran Kirk Business Feature
The Future of FME

September – FME Intelligence 2021

Like many events in 2021 our annual FME Intelligence event went virtual aswell.  The highlights of this event included presentations from: Don Murray (Co-CEO and President) – “The Future of FME” and customer presentations from NI Water: “FME in NI Water – Helping things flow and Dublin City Council: “Dublin City Council Digital Transformation”.

November – Analytics Institute Summit 2021

On Nov 23rd IMGS attended the Analytics Institute Summit in the Convention Centre Dublin, as a partner sponsor.  We hosted branded booths both on-line and at the Convention Centre on the Day.  We were supported on the day by our Data Intelligence Platform Partners – Talend and Sisense.  This was an excellent event with fantastic content and we look forward to a larger presence in 2022.

Sisense Logo

December – Sisense Certification

Sisense certification was the focus for the last quarter of 2021 and we’d like to offer a huge congratulations to the following IMGS Team members that successfully completed their Sisense Certification with flying colours!

  • Keith Duffy – Applications Engineer IMGS – Sisense Solution Developer & Sisense Embedded Analytics Professional
  • Kieran Massey – Applications Engineer IMGS -Sisense Solution Developer & Sisense BI Consultant
  • Ryan Bannon – IT Solutions Lead IMGS – Sisense Solution Developer & Sisense Embedded Analytics Professional
  • Aoibhinn Stuart – Delivery Director – Sisense Analytic Project Manager

Training Courses Delivered by IMGS in 2021

IMGS had a very busy training schedule in 2021, with all 8 FME Training Courses that we ran booked to capacity.  The list below details the course topics delivered during this period.

  • FME Desktop Standard x 2
  • FME Desktop Advanced
  • FME Server & Cloud
  • FME & ArcGIS Online
  • FME & Open Data
  • FME & Reporting
  • FME & Python

Thank You

We would like to thank all of our customers and partners that supported us in 2021 and we look forward to continued success for 2022 and beyond.

Find Out More

At IMGS, Data Intelligence and information are at the core of what we do.  We provide solutions to automate data flows, visualise information and power data insights for a wide variety of customers included Local Authorities, Government Agencies, Utilities and Communications Organisations.

Our data solutions are built on our partner’s technology platforms and serve the citizens of Ireland every day, from managing water and electric networks to publishing key e-Government information through Government Portals.  Through our market leading partners, we provide organisations with end to end solutions that drive operational and capital efficiencies.

To learn and to arrange an initial data session with us, please CLICK HERE to fill in our contact form.

IMGS announced as AGI-Ni platinum sponsor

IMGS AGI-NI Platinum Sponsor

IMGS are delighted to announce that we will be attending this years AGI-NI Annual Conference as Platinum Sponsors.

The AGI-NI Conference is an annual event, and a key focus for GIS professionals across Northern Ireland.  Conference speakers will touch on topics including the use of GI to support sustainable energy infrastructures, forestry and farming, the natural environment and, with the ongoing pandemic, how spatial information can be leveraged for emergency planning and response now and in the future.

Conference Details

Date: June 17th 2021

Time: 9.30am – 1.30pm

Location: On-line Virtual Event

The theme for this years’ conference is “Geographic Information: Supporting a Sustainable Future”.

 

Ciaran Kirk, Operations Director at IMGS will present Solar Farm Monitoring with  M.App Enterprise, during the first lightening talk session.  We would like to invite you to join us at this FREE event and REGISTER today.   The Conference Programme can be found HERE.

IMGS will have representatives available during the breaks and networking sessions to discuss our GIS products and services or answer any questions that you may have.  

Please feel free to our visit our WEBSITE and get in touch HERE, as we’d love to hear from you!

Regular Expression Dialogue

Regular Expressions or understanding search Hieroglyphs!

Regular Expressions or Understanding Search Hieroglyphs

Regular expressions are a sequence of characters that specifies a search pattern. Usually, such patterns are used by string-searching algorithms for “find” or “find and replace” operations on strings, or for input validation. They are therefore incredibly powerful when it comes to validating data, ensuring data matches an expected structure or as we will see below in aiding tasks such as address cleaning and geocoding.

Whilst the example below will focus on their use in FME, the concepts discussed will apply to any tool or programming language that supports them, so even if you are not an FME user hopefully you will find the following discussion helpful.

Recently we were met with a support call that required removing the numeric digits at the beginning of an address. The problem was that the number of digits varied. For example, the data had the following, 9 OAKDALE, 23 AL UISCE and to complicate it further, some entries were similar to 44A CHERRYORCHARD HEIGHTS where the 44A would have to be removed. The end result was to return only the text that fell after the space (OAKDALE, AL UISCE, CHERRYORCHARD HEIGHTS).

Our initial thoughts were that a regular expression would be involved in solving the problem. These are a somewhat hidden, and intimidating bit of functionality within FME that can be hugely powerful once you have gained a little mastery of them. Our initial attempt to use them didn’t give us the output we were hoping for. The digits were stripped from the beginning of the value, but a space remained and worse 44A CHERRYORCHARD HEIGHTS became A CHERRYORCHARD HEIGHTS.

We went back to our workspace (Fig 1) and set about tweaking it to suit the issue at hand. The data was being fed into a StringSearcher transformer that had a Regular Expression parameter (Fig 2). We had found the initial expression from an online search and knew what this regular expression was doing from the output but we could not decipher what each symbol was doing or how to alter the expression to suit our needs. Our initial attempt was promising and looked like it was close to what we were after, so we decided it was time to get our head around the regular expression!

Regular Expressions Figure 1
Fig 1: FME Workspace
Regular Expressions Figure 2
Fig 2: StringSearcher Transformer Parameters

The first port of call was to look at the help for the StringSearcher parameter and try out the regular expression editor in the StringSearch transformer. Like the text editor which can be accessed in transformers that allow for text input this editor dialogue appears where you can insert a regular expression.

Regular Expression Dialogue
Fig 3: Regular Expression Dialogue

Our initial attempt let us see that we were matching the string and started to give us an idea of what the individual elements of the expression were doing but it was still quite confusing to see what the expression as a whole was doing. We then tried Michael Lovitt’s excellent online Rublar expression editor at https://rubular.com which let us see that our expression was actually matching two groups from the single match shown in FME’s editor.

Regular Expressions Rubular.com
Fig 4: Rubular.com Matched Groups

Before we show you how we solved it we want to give a quick primer with regards to regular expression and hopefully take someone that has never used them before and make them (slightly) more comfortable with attempting to use them in the future. We’ll dive straight into the deep end with the expression from Fig 2 and the break it down.

Regular expressions are used for pattern matching. It is a sequence of characters (or symbols) that forms a search pattern and is generally applied to string matching. Let’s take a look at that expression in Fig 2:

^([0-9]+)(.+)$

This might as well have been hieroglyphics but when you break down the expression it actually quite simple. This expression states that the string must begin with at least one or more digits with each digit being from 0-9. So this can be 1, 26, 45, 111. The ^([0-9]+) is the part of the expression that tells us this. ^ means at the start or begin with. This is followed by a set of parentheses ( ). The parentheses indicate a capturing group. This is group 1. The next set of parentheses would be group 2 and so on. Don’t worry if capturing groups do not make sense at this stage, when we return to FME to talk through and run an expression it will be a lot clearer. Inside the first group we have [0-9]+, the [0-9] is a character set of 0, 1, 2, 3, 4, 5, 6, 7, 8 ,9 and the + means one or more. So if we take ^([0-9]+) you can see that this means, match strings that begin with one or more numerical digits. If the + was not present, this would mean that the string could only begin with one digit from the character set.

The next capturing group (.+) had us scratching our heads at first because it just contains symbols. The dot or period simply means match any character, like a wildcard. We have already met the + symbol. This group says match at least one character to any amount of characters. Again, if we removed the + it would only match one (any) character. The $ means the end of the string. So, match one or more characters until the end of the string.

Let’s look at that expression again in its entirety. The string must begin with at least one digit or more followed by at least one or more characters until the end of the string. We will use 9 OAKDALE as an example. This begins with a single digit so the initial criteria ^([0-9]+) is met. 9 belongs to group 1. This is followed by a space, which is also a character, and followed by seven more characters until the end of the string. This satisfies the (.+)$ and places OAKDALE(with the space in front of it) into group 2.

In the FME workflow in Fig 1, both groups will be output as the matched result recreating what we had initially! As we are using groups, we need to create a subexpression List under the transformers Advanced settings. This will output a List with two elements, one for each of our subgroups, this includes both the found string but also its starting position within the original string value.

Fig 5: Matched sub groups as a List element
Matched sub groups as a List element b

So now we understand how FME is processing the current regular expression and can identify the sub groups how do we alter the expression to produce the desired output? We will need to look at a few more symbols and what they can do. To do this we are going to switch the focus away from the problem we are trying to solve and use another example relating to Irish mobile numbers. Take a deep breath and have look at the expression below:

^(\(08[3,5,6,7]\)|08[3,5,6,7][ -]?)[0-9]{7}$

What is this madness? This expression is verifying how an Irish mobile number can be entered let’s say into a form. The number must begin with 08 followed by 3, 5, 6, or 7 wrapped in parentheses, OR begin with 08 followed by 3, 5, 6, or 7 followed by an option space or hyphen, and then followed by seven digits.

Before we go through the new symbols we are meeting here we want you to open https://rubular.com/ and copy and paste the expression into the top textbox. In the lower textbox enter (087)1234567, the entered text should highlight in yellow meaning that it meets the criteria set out in the expression (Fig 3). Next try 0871234567, then 087 1234567, and then 087-1234567 and note that these are all accepted by the expression.

Fig 6: Rubular regular expression validator

Now try (087) 1234567, and then (087)-1234567. These will not highlight because they are invalid. Here is a list of other violations for you to test:

087-123-4567     hyphen separating after 123

087 123 4567       space separating after 123

084-1234567       4 is not a valid third digit

(081)1234567     1 is not a valid third digit

082 1234567        2 is not a valid third digit

0881234567         8 is not a valid third digit

Right, so we can see how a regular expression can detect a valid and invalid entry. Let’s go through this example and then use what we learn to fix the expression in our workspace. We know that ^ means at the start, this is followed by a set of parentheses. There is a large chunk within these parentheses. We will first look at everything to the left of the |

\(08[3,5,6,7]\)

The \ is an escape character meaning that whatever symbol immediately follows it should be considered a character literal to be searched for and not a part of the expression syntax. There are two in the snippet above that allows the expression to search for an opening and closing parenthesis. The opening parenthesis must be followed by a 0 and then an 8, so the beginning must match (08. The [3,5,6,7] is a character set. The next digit can only be one of these digits, so we are now at (083 or (085 or (086 or (087. These are then followed by the closing parenthesis. Therefore, our entry must begin with (08[3,5,6, or 7]).

But the | means OR, like a branch in computer programming. We will now look at the expression to the right of the | still within the original parentheses that are not escaped (capturing group). Our entry can begin with what was to the left of the | or the following:

08[3,5,6,7][ -]?

The above reads that our number must begin with 08 followed by 3, 5, 6, or 7, followed by an optional blank space or hyphen. The ? means zero or one occurrence, so after the initial three digits there can be one space OR one hyphen OR no space and no hyphen.

You should now be able to decipher what the first part of this expression is looking for:

^(\(08[3,5,6,7]\)|08[3,5,6,7][ -]?)

Entries that begin with the following:

(083)    OR      083 (followed by a space)       OR      083-     OR      083 (no space, no hyphen)

(085)    OR      085 (followed by a space)       OR      085-     OR      085 (no space, no hyphen)

(086)    OR      086 (followed by a space)       OR      086-     OR      086 (no space, no hyphen)

(087)    OR      087 (followed by a space)       OR      087-     OR      087 (no space, no hyphen)

The next part of the expression:

[0-9]{7}$

This reads seven occurrences of digits from the set of 0-9 and that is the end of the entry. An eight digit or entering a character that was not 0-9 would mean the entry is invalid. Therefore the curly braces { } is a quantifier of how many occurrences of the expression immediately before it should exist. If there was a {3} then only three digits could follow.

Let’s go through the full expression once more and make sure that we have a handle on this:

^(\(08[3,5,6,7]\)|08[3,5,6,7][ -]?)[0-9]{7}$

The entry must either begin with three digits wrapped in parentheses with the first two being 08 followed by either 3, 5, 6, or 7, OR begin with 08 followed by 3, 5, 6, or 7 with an optional space OR hyphen, followed by seven digits, each from 0 to 9. That reads so simple now that we know what we are doing.

We are now armed with what we need to rectify the expression to solve our issue. The problem we are faced with is that we have entries with one or more number of digits followed by a space, or one or more number of digits followed by a letter (44A) and then a space. We want to cut away those and only be left with the text that follows the space. Let’s begin to write our expression. We know that ^ means the start of or beginning of the string so we will start with that:

^

We have an OR situation where the value can have digits only or digits and a letter. We will begin to construct the first situation that will sit to the left of the |:

[0-9]+[ ]

Now to the right of the |:

[0-9]+[A-Z][ ]

We will wrap the left and right of the | in parentheses and put it together to give us the first part of our expression:

^([0-9]+[ ]|[0-9]+[A-Z][])

At this stage we have an expression that is matching values that begin with either one or more digits followed by a space OR one or more digits followed by a one character from the set A-Z followed by a space. This looks good so far and we can simply take the last part from the expression in Fig 2

(.+)$

and append it to our expression.

^([0-9]+[ ]|[0-9]+[A-Z][ ])(.+)$

Our full expression now matches values that begin with either one or more digits followed by a space OR one or more digits followed by a one character from the set A-Z followed by a space, followed by one or more amount of any characters until the end of the value.

We can now enter this into the Regular Expression parameter of the StringSearcher to see if it works. SUCCESS! The output now creates an initial subgroup containing the property number if present and the second group contains the estate name. We can add these as new attributes to our data as shown below in figure 7.

Fig 7: Output within the Visual Preview window

Before we finish up we want to talk you through the entire workspace presented in Fig 8 below. The Reader reads a CSV file consisting of three attributes. The Estate_Name is the one we are interested in and wherein our problem lies.

Final Workspace Fig. 8
Fig 8: Final Workspace

The Reader connects to a StringSearcher transformer where we set a few parameters (Fig 9) as shown below:

Fig 9: StringSearcher Settings

As mentioned earlier the Matched Result attribute will output the same value as Estate_Name because our regular expression matches all of the elements of the string, so it’s considered a match. As we are matching subgroups, we can leave this blank to stop FME creating an unneeded attribute and under the Advanced section we can enter a name for the output list that will contain the matched groups from our expression. Our first group from ([0-9]+[ ]|[0-9]+[A-Z][ ]) is going to be placed in _lstSubExpression{0}, therefore, the 1 , 22 , 101 , 44A , each followed by the space, will be placed in here. Our second group (.+) is what we are most interested in, this contains everything after the space and gets placed in _lstSubExpression{1}.

The results from both ports of the StringSearcher are passed into an AttributeManager which handles both the matched and unmatched results and creates two new attributes representing the House Number and the Street Name. This is done through the use of Conditional Tests as shown in the figure below.

Fig 10: Conditional Tests to create clean output values

The conditional tests, check to see if the first list element has a value and if it contains a property number adds this as the House Number, if it doesn’t which would be the case of the unmatched records, we explicitly declare the output as Null, so we know we do not have data for this field. A similar set of tests are carried out to populate the Street_Name attribute. Our list is well ordered as a result of the matching process, we know this will be the second item in the list we can use the list element order to determine which part of the list to use.

The AttributeManager demonstrates how it is possible to combine a number of steps by allowing us to wrap up the step of creating the attributes, merging those cases without a property number and cleaning up the data flow by removing the lists in a single step.

Hopefully this will have shown how powerful regular expressions can be and might be used to solve some of your future problems.

Just to test if you have learned anything from this post check out the expression below that does the same as our final expression but more compact and without an OR | branch. Does this make sense to you?

^([0-9]+[A-Z]?[ ])(.+)$

Hopefully our journey through regular expressions has shown how powerful they can be as a tool for pattern matching strings and can be used to validate data such as telephone numbers, Eircode’s, or to aid with address cleansing problems as we seen in this example.  

Alongside the examples we’ve touched on you can find online cheatsheets such as can http://www.cheatography.com/davechild/cheat-sheets/regular-expressions/that can help turn the hieroglyphics into understanding.

We will leave you with one final note; although most implementations of regular expressions are extremely similar there can be slight differences, this means small differences in your data can mean the difference between a match or not so it’s always good to thoroughly test your expressions and transformers like the Sampler can create a subset of your data for this.

Finally, here is one last challenge can you see how this slightly modified version of the expression above differs and do you think it’s a better solution?

^([0-9]+[A-Z]*[ ]*)(.+)$

Remote Working During a Pandemic

Remote working in a pandemic

Remote Working during a Pandemic

By Tanya McDonald, Marketing Executive, IMGS

Last year on March 12th the Taoiseach of the time, Leo Varadkar, gave a landmark speech from Washington announcing a raft of Covid restrictions for Ireland, which included the closure of schools and a switch to remote working for those that could.  Little did we know that this would be the ongoing reality for a large cohort of the population over a year later.

Following that announcement, my kitchen table doubled up as a classroom and home office in between mealtimes during the day.  Multi-tasking took on a whole new meaning as I took on the role of teacher alongside my day job!

Remote Working During a Pandemic

Home Schooling Support

In what was a very difficult situation, the schools offered as much support as they could by providing a combination of recorded lessons on Seesaw and Google Classroom, scheduled zoom calls and a timetable of work for the week.  The structure enabled my children to work mostly independently and freed me up to get my work done.  The RTE Home School Hub also provided some respite for an hr a day when I needed time to concentrate.

Transition to Remote Working

I have found the transition to remote working easy enough this past year, as in-person meetings were replaced with Microsoft Teams meetings and day to day operations within IMGS continued as normal.  Once it became clear that remote working was going to be a long-term arrangement, I moved from the kitchen table to an office set-up in my Son’s bedroom with an ergonomic friendly workspace, that my back has thanked me for!

Events move on-line

The pandemic has certainly accelerated the need for change in the way that we work and plan and run events.  As a Marketing Executive in IMGS, a large part of my role involves the planning and execution of events.  As the pandemic spread and the global situation deteriorated further, it became clear that our events would have to move online for the remainder of 2020 and a virtual alternative found.  We went with Accelevents as our virtual event platform and it has proven highly effective in delivering quality virtual events and exhibitions. It has been so successful that we may continue to use it post pandemic, and offer a hybrid mix of events in the future. 

Is remote working the way forward?

When I reflect on the past year, my overall experience of working from home has been largely positive and even more so since the schools reopened again 😊.  It offers a good work/ life balance and I am sure will be viewed as a more acceptable working arrangement going forward.  Personally, I would favour a hybrid approach encompassing a mix of in-office and remote working, this would fulfil the need for social interaction at work and the opportunity to enjoy a bit of office banter and a Friday lunch when hospitality reopens! 

While I write this blog things are looking brighter and hopefully the majority of us will be vaccinated by the Summer facilitating a return to a more normal way of life.