Time Series Forecasting: A Practical Guide to Exploratory Data Analysis | by Maicol Nicolini | May, 2024

“`html

How to use Exploratory Data Analysis to drive information from time series data and enhance feature engineering using Python

Time series analysis certainly represents one of the most widespread topics in the field of data science and machine learning: whether predicting financial events, energy consumption, product sales or stock market trends, this field has always been of great interest to businesses.

Obviously, the great increase in data availability, combined with the constant progress in machine learning models, has made this topic even more interesting today. Alongside traditional forecasting methods derived from statistics (e.g. regressive models, ARIMA models, exponential smoothing), techniques relating to machine learning (e.g. tree-based models) and deep learning (e.g. LSTM Networks, CNNs, Transformer-based Models) have emerged for some time now.

Despite the huge differences between these techniques, there is a preliminary step that must be done, no matter what the model is: Exploratory Data Analysis.

In statistics, Exploratory Data Analysis (EDA) is a discipline consisting in analyzing and visualizing data in order to summarize their main characteristics and gain relevant information from them. This is of considerable importance in the data science field because it allows to lay the foundations to another important step: feature engineering. That is, the practice that consists on creating, transforming and extracting features from the dataset so that the model can work to the best of its possibilities.

The objective of this article is therefore to define a clear exploratory data analysis template, focused on time series, which can summarize and highlight the most important characteristics of the dataset. To do this, we will use some common Python libraries such as Pandas, Seaborn and Statsmodel.

Let’s first define the dataset: for the purposes of this article, we will take Kaggle’s Hourly Energy Consumption data. This dataset relates to PJM Hourly Energy Consumption data, a regional transmission organization in the United States, that serves electricity to Delaware, Illinois, Indiana, Kentucky, Maryland, Michigan, New Jersey, North Carolina, Ohio, Pennsylvania, Tennessee, Virginia, West Virginia, and the District of Columbia.

The hourly power consumption data comes from PJM’s website and are in megawatts (MW).

Let’s now define which are the most significant analyses to be performed when dealing with time series.

For sure, one of the most important thing is to plot the data: graphs can highlight many features, such as patterns, unusual observations, changes over time, and relationships between variables. As already said, the insight that emerge from these plots must then be taken into consideration, as much as possible, into the forecasting model. Moreover, some mathematical tools such as descriptive statistics and time series decomposition, will also be very useful.

Said that, the EDA I’m proposing in this article consists on six steps: Descriptive Statistics, Time Plot, Seasonal Plots, Box Plots, Time Series Decomposition, Lag Analysis.

1. Descriptive Statistics

   Descriptive statistic is a summary statistic that quantitatively describes or summarizes features from a collection of structured data.

   Some metrics that are commonly used to describe a dataset are: measures of central tendency (e.g. mean, median), measures of dispersion (e.g. range, standard deviation), and measure of position (e.g. percentiles, quartile). All of them can be summarized by the so called five number summary, which include: minimum, first quartile (Q1), median or second quartile (Q2), third quartile (Q3) and maximum of a distribution.

   In Python, these information can be easily retrieved using the well know describe method from Pandas:

           import pandas as pd
           # Loading and preprocessing steps
           df = pd.read_csv('../input/hourly-energy-consumption/PJME_hourly.csv')
           df = df.set_index('Datetime')
           df.index = pd.to_datetime(df.index)
           df.describe()
         

   1. PJME statistic summary.

2. Time Plot

   The obvious graph to start with is the time plot. That is, the observations are plotted against the time they were observed, with consecutive observations joined by lines.

   In Python, we can use Pandas and Matplotlib:

           import matplotlib.pyplot as plt
           # Set pyplot style
           plt.style.use("seaborn")
           # Plot
           df['PJME_MW'].plot(title='PJME - Time Plot', figsize=(10,6))
           plt.ylabel('Consumption [MW]')
           plt.xlabel('Date')
         

   2.1 PJME Consumption Time Plot.

“`



Source link