Feature
Data silos are a problem for many businesses, and often create barriers to information sharing and collaboration across departments within an organization. Although AI and associated technologies are not a panacea for siloed data, they can provide brands with ways to minimize the otherwise tedious efforts to manually eliminate data silos.
Let's take a l look at how artificial intelligence (AI), machine learning (ML) and natural language processing (NLP) can be used to tame the data silo beast.
Data Silos: How AI Can Help Businesses Unlock Hidden Insights
Simply put, a data silo is a repository of data controlled by a single department and isolated from other departments within a business. Siloed data is a problem for many businesses, particularly for those brands that are or will be using AI applications. According to a McKinsey Global Survey about AI capabilities, only 8% of those polled across industries indicated that their AI-relevant data is accessible by systems across the organization.
There are many reasons that data ends up in a silo, ranging from vendor lock-in, to legacy systems, to data lakes. Here are some of the most common causes of siloed data:
- Vendor Lock-in: Many businesses depend on software platforms as part of their martech stack. Quite often, these platforms use data that is in a proprietary format, or they rely upon databases that are not interchangeable with other platforms. This “vendor lock-in” occurs when vendors do not incorporate the ability for data to be exported to other applications.
- Data Limitations: Often, data siloes are not caused by software limitations, but rather they are caused by employees who do not feel comfortable sharing data with other departments, or they simply do not understand how their data could be useful to other teams. Although permissions and limitations are often put in place by default to reduce the misuse of data, such “data ownership” can reduce the overall usefulness of the data within a business.
- Data Lakes: It’s been said that businesses today are drowning in data, and with so much data coming from myriad sources, it’s often challenging for brands to design an effective data infrastructure without disrupting business.
- Legacy Systems: Especially for well-established brands, older legacy software systems can be extremely limited in their ability to work with other, more current platforms. Without a large budget (and incentive) to replace legacy systems that are currently working just fine, sharing data is often not a priority.
Simon Tanné, head of data science at Echobox, a publishing automation platform provider, told CMSWire most companies today strive to base their decisions on data, but quite often, their data is siloed in different teams or within different tools, rendering this data hard to access or practically invisible.
“Because AI and ML draw and learn from multiple data sources across a business, AI technology can consequently break these silos by automatically generating insights or recommendations that are visible, accessible and actionable across various business units,” said Tanné. “This allows for more cross-team collaborations and deeper insights that can ultimately impact the company culture and innovation across the board."
Related Article: Your Silos Are Showing in Your Customer Experience
How AI ad NLP Can Break Down Data Silos
Often, there is information buried within unstructured data that lives in a silo, so the challenge is not only obtaining access to the data, but structuring and formatting it into a usable form. Through the use of AI and natural language processing, brands can overcome this challenge by extracting structured facts from unstructured documents and textual data.
NLP is a computational methodology that can process natural human language. Recently, NLP has been used as a text-mining solution with unstructured data. NLP is able to decipher unstructured data such as social media posts, pre-processing the data to create structured data which can then be used for analysis. NLP is able to quickly standardize mass amounts of unstructured data into actionable information.
Bob Rogers, former chief data scientist at Intel, and CEO at Oii.ai, a data science business specializing in supply chain modeling, told CMSWire that using AI and ML effectively to manage siloed data will depend on which industry the algorithms are tackling.
Learning Opportunities
Conference
May
1
Qualtrics X4 Salt Lake City 2024
Register
Webinar
May
2
Unleash AI for Crisis Management: Transform Threats into Triumphs
Learn how Sprinklr Insights provides real-time insights on current events, trends, and brand crises to empower your team.
Register
Webinar
May
9
Driving Customer-Centricity in the Financial Services Industry
Discover how financial organizations are leveraging generative AI technologies.
Register
Conference
May
9
An Irish Roundtable in the Valley: The Economic Impact of AI on the World of Work Stanford 2024
Exploring the Impact of AI On the World of Work with Irish diaspora leaders and tech business champions from the North of Ireland
Register
Rogers gave the example of work he did at University of California, San Francisco, leading a data science team that was trying to solve a particular healthcare problem: 1.4 million yearly faxes that resulted in three separate data silos. Without AI, the process of eliminating the data silo was very tedious for Rogers and his team. “First, the raw data from faxes were dumped into a processing queue. Next, patient appointments were added to an electronic health record. And finally, various diagnostic reports were scanned and uploaded to the patient’s chart,” said Rogers.
Using AI, Rogers was able to greatly simplify the process. “Use AI feature extraction (a combination of computer vision and natural language understanding) to pull key information from each fax,” said Rogers. “Connect the fax directly to the charts of existing patients and create a new record for new patients. Now, schedule the patient electronically. Use AI feature extraction to index key contents of additional uploaded diagnostic files.”
This process allowed the silos to be connected together through shared identifiers in the electronic health record, and the data was then digitally actionable. “This is a game-changer for hospitals and health systems.”
Related Article: NLP and Text Analytics Enhance VoC Programs, Boost CX Engagement
Managing Complex Data Challenges With AI and ML
The challenges of data silos are multifaceted and are largely related to the reasons that data ends up in silos to begin with. Along with those causes, data overload continues to be a problem for many brands — and not all data is useful.
Kevin Gordon, vice president of AI technologies at NexOptic, an AI imaging solutions company, told CMSWire many brands took the axiom "data is the new oil" literally and implemented their own version of "data hoarding," with the result of massive stores of data with different levels of management (organized, consistent, up to date, etc.).
At that point, these brands had to ask themselves the question "is this an untapped resource?" "The main challenge of siloed data is managing it well so it can be used," said Gordon. "Essentially doing something useful with it. A lot of data is sufficiently complex that both managing and utilizing it is a hard problem.” Gordon believes that machine learning (ML) has changed this somewhat, as it enables complex patterns and usages of siloed data. “There’s still the issue of managing the data well so that it can be fed into these machine learning algorithms."
“Depending on where a company is in the data/ML cycle, they’ll need different advice to overcome challenges," said Gordon. "If they’re just starting the data collection process, they’ll need advice on what and how to do it. If they’re collecting data, they’ll need advice on scaling and connecting their data to machine learning workflows, and if they’re using ML on collected data, they’ll need advice for optimizing and deploying,” explained Gordon, who suggested that after deployment, organizations may want to revisit the cycle to see if they’re missing anything associated with the company’s "big picture"goals.
Ironically, while AI and ML can be effective tools in the fight to eliminate data silos, in order for AI and ML applications to be most effective, data silos must be dealt with to remove conflicting versions of the truth. This requires every team within an organization to be on board with the goal of eliminating data silos. Communication and collaboration between teams must be encouraged and supported. Finally, a business must embrace a data warehouse solution that has the scale and performance to facilitate every department’s data needs.
Final Thoughts on Eliminating Data Silos
Many businesses struggle with the problem of siloed data, an obstacle that effectively strips away the value of that data for other departments and teams within the business. By using AI and ML, the often tedious and mind-numbing work of manually extracting useful data from data silos can be simplified and greatly improved.
Paradoxically, the elimination of data silos will enhance and improve the efficiency and accuracy of AI and ML applications, providing brands with a win-win solution to creating a single source of truth for data in their organization.
