The digital world cannot exist without the natural resources needed to sustain it. What are the implications of the technology we are using to develop and operate AI?
Photo by ANGELA BENITO on Unsplash
There is a fundamental concept in machine learning that I often explain to non-experts to help illustrate the philosophy behind my work. This concept revolves around the idea that the world is constantly changing around every machine learning model, often influenced by the model itself. Therefore, the world that the model is attempting to imitate and predict is always rooted in the past, never in the present or the future. The model is, in a sense, predicting the future – that’s how we typically perceive it. However, in many other ways, the model is actually striving to bring us back to the past.
I like to discuss this concept because the philosophy surrounding machine learning provides us with valuable insight as both machine learning practitioners and as users and subjects of machine learning technology. I often mention that “machine learning is us” – meaning, we generate the data, conduct the training, and utilize and apply the results of the models. Models are attempting to follow our instructions, utilizing the raw materials we provide them, and we have significant control over how this process unfolds and what the outcomes will be.
Another aspect of this concept that I find important is the recognition that models are not isolated within the digital realm but are deeply interconnected with the analog, physical world. If our models are not impacting the world around us, it raises questions about the purpose of these models. In reality, the digital world is only separated from the physical world in a limited, artificial sense – primarily in terms of our interactions with it as users and developers.
Today, I want to delve into how the physical world influences and shapes machine learning, and how machine learning and AI, in turn, impact the physical world. In a previous article, I mentioned that I would explore how the constraints of resources in the physical world intersect with machine learning and AI, and that is the focus of our discussion.
If we delve into it, it becomes evident that the digital world heavily relies on natural resources for its existence. There is a common joke about defeating sentient robot overlords by simply turning them off or unplugging the computers. However, this joke holds a kernel of truth. Those of us working in machine learning, AI, and computing in general are entirely dependent on natural resources such as mined metals and electricity for our industry to thrive. This dependency mirrors a piece I wrote previously about how human labor is essential for the existence of machine learning. Today, we will explore two vital areas – mining/manufacturing and energy, particularly in the form of electricity.
If you explore further, you will find a wealth of research and journalism on these areas, not only in direct relation to AI but also in connection with previous technological booms such as cryptocurrency, which shares significant resource usage similarities with AI. I will provide a general overview of each area, along with references for more in-depth reading to delve into the details and original research. It is challenging to find research that takes into account the recent boom in AI over the last 18 months, so some of the research may underestimate the impact of these new technologies in the generative AI space.
What goes into the production of a GPU chip? These chips play a crucial role in developing modern machine learning models, and Nvidia, the leading producer of these chips, has capitalized on the cryptocurrency and AI trends, propelling them to become one of the most valuable companies globally. The production of GPU chips involves various specialized raw materials that are rare and challenging to obtain, such as tungsten, palladium, cobalt, and tantalum. Some elements, like mercury and lead, are easier to acquire but pose significant health and safety risks. The mining and manufacturing processes for these materials have notable environmental impacts, including emissions and ecological damage in mining areas. Even the most well-run mining operations result in severe ecosystem alterations. Additionally, there is a risk of using “Conflict Minerals,” mined under conditions of human exploitation, child labor, or slavery. (Nvidia has taken a strong stance against using such minerals, particularly calling out the Democratic Republic of Congo.)
Following mining, these materials must undergo careful processing to create the small, highly potent chips essential for running complex computations. Workers face substantial health risks when dealing with heavy metals like lead and mercury, as evidenced by industrial history. Nvidia’s chips are predominantly manufactured in Taiwan by Taiwan Semiconductor Manufacturing Company (TSMC). Since Nvidia does not own or operate factories, they avoid scrutiny regarding manufacturing conditions and emissions, and data on this process is limited. The power required for this manufacturing process is not attributed to Nvidia. Notably, TSMC has reached its production capacity and is working on expanding it, while Nvidia plans to collaborate with Intel on manufacturing capacity.
Once a chip is produced, it can have a significant useful lifespan of 3-5 years if maintained properly. However, Nvidia constantly develops new, more powerful, and more efficient chips (producing 2 million chips annually), potentially limiting a chip’s lifespan due to obsolescence and wear and tear. When a chip becomes obsolete, it enters the e-waste stream. While theoretically, the rare metals in a chip hold recycling value, chip recycling is a specialized and challenging process, with only about 20% of all e-waste, including less complex devices like phones, being recycled. The recycling process involves workers dismantling equipment, exposing them to the heavy metals and other elements used in manufacturing. If a chip is not recycled, it is likely disposed of in landfills or incinerated, releasing heavy metals into the environment through water or air – particularly impacting developing countries and local communities.
Most research on the carbon footprint and environmental impact of machine learning has focused on power consumption. Therefore, let’s explore this aspect further.
After acquiring the necessary hardware, electricity consumption emerges as a significant concern in the realm of AI. Training large language models consumes vast amounts of electricity, and deploying these models also requires substantial electricity usage.
A research paper indicates that training GPT-3 with 175 billion parameters consumes around 1,300 megawatt hours (MWh) or 1,300,000 kilowatt hours (KWh) of electricity. In comparison, training GPT-4 with 1.76 trillion parameters, estimates suggest power consumption during training ranges between 51,772,500 and 62,318,750 KWh of electricity. For context, an average American home consumes slightly over 10,000 KWh per year. Conservatively, training GPT-4 once could power nearly 5,000 American homes for a year. (This calculation does not consider the power consumed by preliminary analyses or tests needed to prepare the data for training.)
The power consumption during training GPT models significantly increased between GPT-3 and GPT-4, highlighting the immense electricity requirements of advanced machine learning models.
