Federated Data Lakes Could Make Sense of Enterprise Data ‘Mess’ to Power AI

Australian organizations in recent decades have made efforts to consolidate data. They have transitioned from data marts, which contained specific information for business units, to data warehouses, data lakes, and now lakehouses that store structured and unstructured data. However, the concept of federated lakehouses may now be gaining traction. Vinay Samuel, CEO of data analytics virtualization firm Zetaris, explains that reality is pushing organizations to build pathways to data where it is located rather than attempting to centralize it. The founders of Zetaris realized that data could never be fully centralized. When asked about the decision to start Zetaris in 2013, Samuel explains that the company emerged from his experience in data warehousing in the 1990s. He noticed that customers were constantly searching for the next best data platform and struggled to bring all their data together in a timely manner. This led the company to challenge the idea that data had to be consolidated for analytics and reporting. Instead, Zetaris developed a system where data could be analyzed in its original location, rather than moving it to another platform. Samuel acknowledges that this approach was challenging and required significant computational power and new software. However, with the rise of real-time analytics and AI, Zetaris’ solution has become critical for organizations that want to access all their data in real time. The initial value proposition of Zetaris was cost savings, as customers could avoid moving their data to a central warehouse. This approach also offered security benefits. However, with the shift to AI, organizations now expect their AI systems to have access to all their data in a format suitable for analysis and governance. Today, Zetaris’ unique selling proposition is enabling customers to analyze all their data, regardless of its location, while ensuring privacy and data governance. Zetaris has also become a data server for AI, preparing data for AI applications such as retrieval-augmented generation. The company has observed that many organizations face challenges in AI adoption due to data readiness and data access issues. Zetaris aims to address these challenges and prevent hallucinations in AI models by applying reasoning models and data filtering techniques. Another common data challenge is blending different types of data, such as structured and unstructured data, to answer business questions effectively. For example, banks struggle to combine object data (e.g., pictures, sound, device data) with transactional bank statement data to enhance their decision-making processes.



Source link