In today’s big data world, we have seen an explosion of data from structured data sources such as databases to unstructured sources such as twitter feeds.  This data growth has led to a movement away from the old ways of relational databases to NoSQL flat databases such as MongoDB.

This data growth has also driven the need for distributed systems such as Hadoop to drive the processing of these massive datasets, and where on-premise servers can’t handle this processing organisations are moving to the cloud for greater elasticity and scalability.

But with this data growth, we must ask the question are we any better off? Is this data helping or hindering decision making?

I feel this data growth is off course of great benefit but for us to benefit from this data growth we need to understand our data properly. To understand our data, we need to have metadata or data about our data.

Too many analytical projects fail because of a lack of understanding of the true single source of data, whether it be a single view of a customer or a single view of an asset. With data now stored in many different systems, formats and locations (on premise and in the cloud), using Metadata to better understand this information is essential.

Coming from a geospatial background I have talked about metadata for some years now and at one stage the EU INSPIRE initiative was the driving for force for organisations to manage their Geospatial data better and develop proper metadata catalogues.

Unfortunately INSPIRE never really achieved what it planned to in my opinion but from the 25^th of May there will be new GDPR legislation which will drive us all to manage our data better.

It is easy to scoff at the EU, and to parody the great Monty Python Life of Brian maybe ask ourselves what have the Romans EU ever done for us – except for the money, roads etc. And it maybe its unpopular to say but most EU legislation is of benefit to the citizens of Europe and in my opinion GDPR is no different.

As we have seen in the last month with the Cambridge Analytica data breach scandal, protecting customer data is of utmost priority through proper data governance (and if you don’t understand your data – how can you protect it).  Also, unlike INSPIRE, the GDPR regulations have teeth and any company who is in breach of these regulations will suffer large fines.

But managing metadata isn’t just about meeting GDPR or any other data regulations (and believe me more are coming).  If we can understand our data better, we can make better decisions.  By making better decisions our organisations can grow sales and cut costs through improved efficiencies. Before you make any decisions on new Big Data analytical projects, ensure you understand your data by implementing a proper meta management strategy in your organisation, and the ROI for your Big Data projects will increase substantially.

If you would like to hear more about our metadata solutions, please listen to my Irish Tech News podcast https://buff.ly/2G22pUM or attend our up coming webinar on how we can easily handle GDPR subject data access right requests using our metadata solutions https://bit.ly/2Fy5gb6