How You Can Benefit from Native Connectors as Part of Your Data Intelligence Platform

Integration is key

As the variety of data sources rapidly increases, the ability to integrate tools is essential for the data and software ecosystems of every organization to easily connect to data sources and business applications. The data integration becomes the rope that ties everything together in your stack. It allows you to get the best outcomes from the data and add value to your business.

Application integration is the process of merging and optimizing data and workflows which enables better business efficiency.

It is vital for any new applications and tools that are to be added to your stack to help solve the integration and interoperability challenges, allow faster adoption by users, lower implementation downtimes, and shorten the time to value.

We already highlighted “connectivity compatibility” as one of the challenges of enterprise data catalog connectors in our Empowering Enterprise Application Integration (EAI) with the Accurity Platform blog post. So, let’s take a closer look at “connectors”.

Connectors to the rescue

What are data connectors? They extract data from a source, write it to a destination, and they can do this on a schedule. Hence, they “connect data” from one place to another. To do this they need to know the extraction location (URI), the access credentials, the parameters of what they are supposed to be doing, and the destination. So, there can be substantial differences in sources, destinations, and schedules making each connector unique.

There are several different ways of connecting data. First up is the proprietary connector where systems from the same vendor can talk to each other e.g., Microsoft Power BI can connect to Microsoft Azure to pull metadata. We’re not talking about those here as they are closed systems. Then there are Application Programming Interfaces (APIs) such as REST which are usually easier to work with.

Then there are data and metadata connectors. Basic data connectors such as connecting your data catalog to a database like PostgreSQL are the sort of thing that might be provided out of the box. Or they can be developed fairly quickly, on demand, based on our customer needs. For instance, Accurity built, tested, and deployed data connectors for Google BigQuery and the NoSQL database MongoDB for one of our clients in less than 24 hours!

All data connectors must be able to handle schema changes and errors – this is where working directly with a data tool vendor really shows its strength versus just buying a box with little or no support. Once a schema changes the data connector could become useless and it may not even fail nicely!

The next type are more sophisticated metadata connectors that integrate with various repositories. For our Accurity platform, these particular types of metadata connectors could communicate with third-party reporting tools like Power BI, Qlik, and Tableau, and connect them to the Accurity Data Catalog. When we create such connectors, they enable great opportunities including:

• creating a centralized list of reports plus knowing how these reports are used and which parts of the data catalog are used in the reports.

• when sales reports are linked to databases/tables such as orders and customers there could also be others e.g., returns, and there is a need to link all of this to the business glossary so that you know which reports use which business terms.

And most importantly, some of this stuff, well, it just needs to happen automatically so that you get an up-to-date map of what is used. Of course, all of this sounds like a great idea but it is not simple!

Parsing the code

The primary concern is that we have to parse the third-party code of the report to get the relevant information/metadata. This is something that is not provided by the third party in their own tools – we solve this problem! You still use your own tools for the reports themselves but with our solution, the changes in the data catalog will be known, automatically.

You might have no explanation of business terms in your third-party tools, but we integrate the business glossary and data catalog with those tools – we can link them!

The challenge

It’s a big one – the third-party reporting tools can easily pull data from their sources but taking a report and unwinding it back to which parts of the data catalog are used and what business terms are referenced from the business glossary, is no easy task.

The reports can be incredibly complex, and these reports need to be analyzed and then deciphered, decoded, and unraveled to give the connector its parameters. Is this even possible? It is often the case that the third party deliberately doesn’t document the architecture of their report. And they often restrict what metadata can be retrieved through their API. In such cases, we deconstruct the report and reconstruct it within our connector.

This is nowhere near as easy as developing a simple data connector or database connector though, as the metadata connector needs to be created each time for each unique reporting tool. This does not end for us with one new connector! We are continuously enhancing our portfolio of ready-to-go connectors for the most used/requested repositories and reporting tools. You can see which are planned in our public roadmap.

Why are we doing this?

Well-documented API and prepared connectors help support the adoption of tools in your data stack by the users within your team. They also enhance scalability to quickly meet your growing data needs without the long and expensive downtime.

We listen to our customers! What connectors would you like? What integration would your business love to have? Come talk to us or get a demo of what Accurity can do right now, and let’s integrate your world of data with our tools.