Extracting intelligence from your data, quickly and easily

Everything leaves a trace: every transaction, from sales to salaries, and every interaction, from emails to social media. This creates a reservoir of valuable data that businesses can tap into to inform both tactical and strategic decision-making.

But do Irish businesses typically have sophisticated data analysis techniques? Some do, having seized the moment, said Muhammad Zeeshan Khan, chief technology officer of the Microsoft services division at TEKenable, but others have had slower starts.

“It really depends on the business. Many businesses are, indeed, using sophisticated analysis techniques and there are also many others who are starting on this journey,” he said.

“It’s a journey that requires careful planning, investment and a culture that values data-driven decision-making. It’s a real mixed bag in Ireland.”

Businesses produce huge mountains of data – and ironically, the sheer amount of data could be slowing organisations’ moves in the area.

“Many businesses have a lot of data, [but] I come across businesses using data mining and predictive analytics quite rarely,” he said.

“To do any of the ML [machine learning] or analytics, you really have to get your data in order. I think there’s a struggle here, because the systems that are collecting the data do it a lot faster than people can organise it, so you end up with more and more data. They can be collected, and across different systems where the structures are different, too.”

Typical factors that have an impact on progress include size of the business, the industry, and their internal technical capability. And if something is done, it can be quite ad hoc and not scalable.

“In the end, people take the initiative and do it in Excel,” Khan said. “In some cases they use Tableau or Power BI, but there is a lack of automation.”

Khan said that Excel is a fine piece of software, but using it is a long way from unlocking the potential that data has to offer.

“They need to consider the tools and the skillset at a broader level. Excel is a powerful tool for data analysis, but it is limited. To an extent, the reason Excel is so successful is that the complex systems, the ERPs, the CRMs, produce data in a structured column row format so it is delivered as a CSV file,” he said.

“What happens is that there is a tipping point and that is typically where we come into the picture. The Excel is too unwieldy, it takes too long and only the person who created it can use it.”

Add to this the fact that most businesses are not disciplined in planning and deploying systems. Instead, things grow organically and processes build up.

The skills that are required to make progress can also act as a roadblock.

“Databases are a key skill, and you likely need a BI system, SQL server and analysis tools like Power BI, and languages like Python and R. Frankly, we’ve come across large global businesses who do all of their planning in Excel,” he said.

Nevertheless, a push is under way to change this. Microsoft’s end-to-end analytics solution, Fabric, Khan said, has the potential to make data accessible to businesses of all kinds.

“Microsoft Fabric is trying to democratise access. You use it just like Power BI, but using the processing power in the cloud,” he said.

Added intelligence

Artificial intelligence (AI) can also give businesses an edge with data. Arguably, the ability to analyse data is the key to AI – and it makes entirely new datasets available, such as unstructured data in text documents, social media posts and customer reviews.

“There is a wealth of information that can provide valuable insights for a business, but extracting this can be difficult,” Khan said.

“You would have to develop a special way to extract and understand data. That is not sustainable,” he said.

This has changed because of AI’s abilities in the areas of natural language processing (NLP) and machine learning (ML).

“The NLP will understand the data and ML will make predictions,” said Khan.

The results can be quite impressive: information can be sorted, parsed and synthesised and given back to the user in the form of answers to their questions.

“Even at TEKenable we have hundreds of thousands of documents that have evolved in SharePoint. Other people have them on-premise, however it may be. The first thing you need to do is index the data that is relevant using Azure Cognitive Search. Wherever that data is, it allows us to index it semantically. It’s not like the indices at the back of a book; it understands the context,” he said.

TEKenable has developed this technology into its own Azure app, Chat With Your Documents, which allows organisations to access information currently locked up in documents. Users ask a question to the app and it comes back with an answer based on what is contained in company files such as emails, sales charts and presentations.

“What we can do is put together a solution, which we’ve done with, for example, Chat With Your Documents, and you get the most pertinent documents. They are fed into the Large Language Model [LLM] and it comes back with the exact answer or the closest approximation.

The kinds of questions that LLMs can answer are surprisingly complex, he said, such as ‘what were the sales for Canada for Q1 2022 compared to Q1 2021?’.

“Before the language models, if you wanted to do something like that, you had to get a specialist team, to then go through it to get the answer,” said Khan.

Impressive as the technology is, Khan said, the ultimate goal was to help businesses achieve their goals. As a result, success in AI or analytics should be quantified depending on the goals of the business.

“Some metrics should be accuracy of prediction, speed of analysis and, of course, cost savings as the bottom line is always a key thing in business. By automating tasks this leads to significant cost savings, but it’s about making the employee more productive, not replacing them,” he said.