Sponsored

Bringing data home

Spatial data can now be integrated into existing data lakes to procedure a new dimension in analyses

Eamonn Doyle, chief technology officer, Esri Ireland: ‘You have an awful lot of data, and getting it to and from the cloud can be an issue’

Esri Ireland has been providing spatial data solutions to businesses and public sector organisations for some time. Anywhere where location is of paramount importance, from planning to service provision, it has been there providing sophisticated geographic information system (GIS) mapping tools that make sense of our physical world.

Recently, however, the company has found that its GIS services, with little tweaking, are able to bring value to sectors outside those we traditionally associate with mapping, be those telecoms or utilities.

Now, all kinds of organisations can take space and location into account in their analyses.

“We're seeing an overall trend of turning our product ArcGIS inside out, bringing what we do in spatial data into IT. A trend now is to bring the processing to the data,” said Eamonn Doyle, chief technology officer at Esri Ireland.

In practice, this means that spatial data can be drawn into a database and then ArcGIS set to work on the data where it is, rather than having to repatriate it to the cloud or an on-prem system.

“You have an awful lot of data, and getting it to and from the cloud can be an issue,” said Doyle.

Typically, data sits in a data lake environment, and usually not an SQL database as they are not suited to processing that sort or volume of data produced in spatial analyses.

“What people are doing is leaving their data where it is and processing it there,” he said.

This could, for instance, mean working with the open-source unified analytics engine for Apache Spark and processing on a Databrick.

For Esri Ireland, it means offering clients, in the form of ArcGIS GeoAnalytics Engine, a new flexibility in how they integrate spatial data, as well as how they interrogate it.

“We’re responding to that. We had been saying, ‘give us your data and we will process it’. As we have moved to bring the processing to them, the sort of people we were dealing with changed from GIS analysts deep in an organisation to data scientists,” Doyle said.

“GeoAnalytics Engine takes our secret sauce and brings it to the data scientists,” he said.

The rise of data science was of significant importance in this, Doyle said, as it demonstrates the maturity of data analytics as a discipline.

“Ten years ago, data science wasn't a thing, but because of the development particularly of social media and the technologies it tends to use, data scientists have become much more commonplace across organisations. Most organisations we deal with have a cadre of them,” he said.

Notably, data scientists take a different approach to things: exploring data looking for patterns rather than processing it looking for patterns. In turn, this results in greater flexibility as the data can be used to lead the analysis.

“They don't have a specific end goal in mind. We did some work with people analysing crop production and global supply chains. That sort of thing requires a different approach,” he said.

However, Esri Ireland brings specialist capabilities to what is often an otherwise primitive approach to spatial information, instantly giving data scientists access to a new dimension in their analyses.

“A lot of data science systems can do some mapping, some spatial stuff, but it does tend to be basic, whereas we bring over 100 spatial functions to it. We come in with a plug-in, a set of libraries, and they can add into their workflows,” said Doyle.

Industries and sectors that are now working in this way span the globe and include retail, logistics, intelligence and transport: “Anywhere where there's a lot of information generated. You can't really process that in a SQL database. 10,000 people producing a GPS location every ten seconds is a colossal glut of information, so anything that produces big data like that needs that kind of approach,” Doyle said.

Also on the rise is combining both spatial and non-spatial data in graph databases, a kind of database which shows the links between entities.

“We've extended a database called ‘Neo4j’ to induce spatiality so it can show not just the links between people, but the spatial relationships. It has a really clear use in logistics,” said Doyle.

Examining the utility of GIS has also seen Esri move toward the Microsoft ecosystem, where it can add spatial information to traditional business intelligence.

“There's a lot of interest in Power BI, which is very powerful for visualisation of data. Increasingly; people want to use mapping in that so we've built a plugin called ArcGIS for Power BI which allows you to link ArcGis to Power BI and report inside it. Alternatively, it can extract the data from ArcGIS and bring it into PowerBI”.

For Doyle, this increase in accessibility is a real milestone that will help more businesses take the journey to incorporating spatial data in their operations.

“We're enabling people to use location in a meaningful way,” he said.