Welcome to insideBIGDATA’s “Heard on the Street” round-up column! In this regular feature, we highlight thought-leadership commentaries from members of the big data ecosystem. Each edition covers the trends of the day with compelling perspectives that can provide important insights to give you a competitive advantage in the marketplace. We invite submissions with a focus on our favored technology topics areas: big data, data science, machine learning, AI and deep learning. Click HERE to check out previous “Heard on the Street” round-ups.
Use science and math when making infrastructure design decisions for LLMs.
Commentary by Colleen Tartow, Ph.D., Field CTO and Head of Strategy, VAST Data
“When designing an AI system for training and fine-tuning LLMs, there is a lot of vendor-driven conversation that confounds the landscape. However, relying on common sense along with some math and science can get organizations far in a situation where small mistakes can be incredibly costly.
For example, let’s look at calculating the bandwidth necessary for checkpoint operations in a recoverable model. There are multiple modes of parallelism to consider in an LLM training environment, each of which makes recoverability more streamlined. Data parallelism takes advantage of many processors by splitting data into chunks, so any individual GPU is only training on a portion of the full dataset. Model parallelism is similar – the algorithm itself is sharded into discrete layers, or tensors, and then distributed across multiple GPUs or CPUs. Finally, pipeline parallelism splits the model training process into smaller steps and executes them independently on different processors. Combining these modes of parallelism ensures that recoverability is possible with a much smaller checkpoint overall. In fact, since the model and data are copied in full to each octet (group of 8 DGXs) and parallelized within the octet, only one checkpoint is needed per octet, which drastically reduces the bandwidth needed to write a checkpoint.
This is an example of how understanding the intricate details of parallelism and LLM training can help organizations design a system that is well-built for checkpoint and recoverability operations. Given the scale of infrastructure required here, it is paramount to neither over- nor under-size resources to avoid overpaying for hardware (wasting money) nor underprovisioning the model architecture (negatively affecting deliverables). Simply put, organizations need to rely on real technical knowledge and calculations when designing an AI ecosystem.“
Generative transformation is the future of the enterprise.
Commentary by Kevin Cochrane, CMO, Vultr
“Gartner’s annual IT Infrastructure, Operations & Cloud Strategies Conference explored the future of infrastructure operations, highlighting the biggest challenges and opportunities heading into 2024. One of the big themes for 2024? Generative transformation. Enterprises across industries – from healthcare and life sciences to financial services and media & entertainment – are racing to embrace AI transformation. However, the past year has highlighted the need for enterprises to actively pursue generative change to flourish in the evolving landscape of AI-driven business.
Generative transformation involves implementing both technological and organizational shifts to integrate generative AI into the fundamental aspects of business operations. The three foundational steps needed to achieve this include: formulating a strategy around how your enterprise will use generative AI, planning for the organizational change needed to fully roll out generative transformation, and building and deploying a platform engineering solution to empower the IT, operations and data science teams supporting your generative transformation journey. With these three steps, enterprises will be well on their way to diving head first into generative transformation, and ultimately thriving in today’s dynamic business landscape.“
GPT store allows for better ChatGPT user experience.
Commentary by Toby Coulthard, CPO at Phrasee
“There’s been interesting discourse in the AI ‘hypefluencer’ space (namely on X) that paints GPTs as glorified instruction prompts and provide no more capability or functionality than before, and therefore provide no utility. They’re missing the point; this is a user experience change, not a capability one. A large part of what OpenAI has been struggling with is that most people beyond the early adopters of ChatGPT don’t know what to do with it. There’s nothing scarier than a flashing cursor in a blank field. ChatGPTs ‘chasm’ in the product adoption curve is that these early majority users want to use ChatGPT but don’t know how. No one was sharing prompts before, now they’re sharing GPTs, and the GPT Store facilitates that. The GPT Store opens the door to the next 100m+ weekly active users. With the recent launch, I expect there to be a few very profitable GPTs, with a very large, long-tail of GPTs that are free to use. Plugins will allow for further monetization through third-party services via APIs, but that’s further down the line. The comparison with an ‘app store’ is misguided, OpenAI isn’t facilitating the dissemination of applications, they’re facilitating the dissemination of workflows. Principally from the most advanced ChatGPT users with experience in prompt engineering, to the least. It’s improving the usability and accessibility of ChatGPT, and its intention is increased adoption and improved retention – in that regard it will improve OpenAI’s competitiveness. They also act as ‘agents lite’. OpenAI has even admitted this is a first step towards autonomous agents. They have a reputation of releasing earlier versions to democratize the ideation of use cases – both beneficial and harmful to inform where the product should go. OpenAI is aware that even they don’t know all the use cases for their models – the GPT Store enables them to see what people produce and what’s useful, popular, and potentially dangerous before they build in more capability and autonomy to these GPTs. The challenges lie in the edge-cases that OpenAI is yet to think of.”
The Information Commissioner’s Office (ICO) in the UK is investigating the legality of web scraping for collecting data used in training generative AI models.
Commentary by Michael Rinehart, VP of AI, Securiti
“Enacted on May 25, 2018, the EU General Data Protection Regulation (GDPR) is having a profound impact on the enforcement of global privacy regulations. Emphasizing the importance of obtaining consent before processing information, GDPR is even more relevant today than it was six years ago when it was first implemented. In this evolving landscape, the surge of AI has fundamentally altered how data is handled, further reinforcing the significance of adhering to these laws. The recent investigation by the UK’s Information Commissioner’s Office (ICO) into the legality of web scraping for training generative AI models is therefore unsurprising. These models generate text or images based on large datasets, raising privacy concerns due to automated data collection.
The ICO’s focus on data protection standards aligns with GDPR principles. Responsible AI implementation requires robust data governance practices. As AI systems continue to be integrated with sensitive data, organizations must establish strong controls, including strict access control, anonymization, and governance frameworks to optimally balance AI potential with data privacy and security.“
Lacking Proper Data Governance, ‘Black Box’ AI Could Be Disastrous for the Enterprise.
Commentary by John Ottman, Executive Chairman of Solix Technologies, Inc
“Over the past year OpenAI’s ChatGPT has dominated the press, business discussions and influenced every company’s stance and guidelines on the use of generative AI. The impact has been profound as the entire world has now largely seen how generative AI automates repetitive or difficult tasks, reduces workloads and simplifies processes. But more recently, concerns have arisen over enterprise use of ChatGPT, noting the data governance challenges caused by exposing enterprise data to ‘black box’ AI and ‘being too reliant on one company’s AI tech.’ Enterprise AI users are concerned that the risks posed by ‘black box’ AI models are just too great, and some have already banned their use. Chief amongst the list of concerns are data security, data privacy and compliance with laws regulating the handling of sensitive data and personally identifiable information (PII). Others even worry that ChatGPT would become so integral to their business that a failure at OpenAI would lead to a failure at their business as well. It’s an unavoidable conclusion that training an external, proprietary ‘black box’ AI model with your private enterprise data is dangerous and may expose your company to data breach, legal risk and compliance findings. ‘Black box’ training inputs and operations aren’t visible for peer review and prompts may arrive at conclusions or decisions without providing any explanation as to how they were reached. ChatGPT introduced the world to generative AI, but so far data governance concerns rule out a central role in the enterprise. In the wake…
Source link
