{ "cells": [ { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "wFv8jhZ0BtcY" }, "source": [ "# Recitation 9: Forecasting\n", "\n", "This assignment will be handed in differently from past assignments.\n", "\n", "**To get credit for this assignment as a recitation, you only need to do a part of it. Aim to finish parts 1-3 and if doing this takes you past the end of the in-person recitation then talk to your TA and you can hand in the recitation to get full credit as long as you put in a reasonable effort.**\n", "\n", "**The complete assignment will be due as part of the next homework, and full points will require completion of the entire assignmen with correct answers.**\n", "\n", "In this assignment, you will be making predictions based on time series data from a sales of a collection of consumer products from the company that sells Stovetop Stuffing. These products exhibit strong seasonal sales patterns. You'll be exposed to many different ways of making predictions and you'll get to see some of the drawbacks of each. You should be largely self-sufficient with python and pandas for data science by this point." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": {}, "colab_type": "code", "id": "x1DReW-GBfTX" }, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": {}, "colab_type": "code", "id": "6gmWlYgOBfTi" }, "outputs": [], "source": [ "from datetime import datetime" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "wKlp8Mf7CgoN" }, "source": [ "## Part 1: Data" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "2wjtKKh5CxRY" }, "source": [ "1.1. Read in the SalesAndForecastsByLine.csv into a dataframe. Then print some of the data to see what it looks like." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": {}, "colab_type": "code", "id": "8qyi96KtBfTp" }, "outputs": [], "source": [ "# YOUR CODE HERE" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "9tnF2xMVDD-V" }, "source": [ "1.2 Create a new column in the table called \"Date\".\n", "\n", "Fill this column with the dates as strings in the form month-year. For example May 2020 would be 5-2020. Print the data again to see if the column was created properly." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 221 }, "colab_type": "code", "executionInfo": { "elapsed": 424, "status": "ok", "timestamp": 1588955619865, "user": { "displayName": "Zaeem Rana", "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgqvsyKIjE7K_gx0nrQU-86JpEmCfHYnaueQVTU=s64", "userId": "11030483233786659768" }, "user_tz": 240 }, "id": "IVtcSvNsBfT_", "outputId": "fce860bf-7a61-423f-c3d9-a52eef3d1317" }, "outputs": [], "source": [ "# YOUR CODE HERE" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "Lcc2-8vFD9sO" }, "source": [ "1.3 Now create a column in your dataframe that contains each date in the Date field in your dataframe (which is a string) converted to the datetime type. `datetime` is the datatype that Pandas uses to store dates and it is very helpful when plotting quantities versus date or time. \n", "\n", "One way to do this is with `iterrows()`, as in the following example code:\n", "\n", "```\n", "for index, row in df.iterrows():\n", " df.loc[index,'t'] = # calculate value from row['Date']\n", "```\n", "\n", "There are other ways as well. Once you are done, output some rows from the dataframe again to make sure that you are computing your datetime field correctly.\n", "\n", "Hint: Use the strptime from the datetime module. Documentation can be found here: https://docs.python.org/3/library/datetime.html#strftime-strptime-behavior. the strptime function will parse string arguments into datetime objects, based on the format that you provide. If you have a string stored in a variable called `date` in a format where 5-2020 represents May 2020, then calling `datetime.strptime(date,'%m-%Y')` will return a datetime object representing May 2020." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# YOUR CODE HERE" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "UrhMPdMjDpoD" }, "source": [ "1.4 Now create 5 different dataframes, one for each of the product line numbers, by filtering on the `LineNumber` field.\n", "\n", "Then plot `TotalProductSales` vs. date (using your new datetime field) for each of the 5 product lines. Put them all on one plot. This can be accomplished by calling plt.plot for each of the 5 dataframes all in 1 cell. matplotlib should figure out on its own to use different colors for each." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": {}, "colab_type": "code", "id": "aJ0ye7DGBfUQ" }, "outputs": [], "source": [ "# YOUR CODE HERE" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1.5 What times of year are particularly busy for these products? You may find that you need to expand the size of your graph to see the dates. You can do this with the code `plt.figure(figsize=(10,5))` run in the same cell and before any `plt.plot` commands. You can also zoom in to particular regions of the figure using `plt.xlim`. For example, `plt.xlim('2000-04','2000-12')` zooms in to April to December 2000." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Answer Here**" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "6PiT2J9dEWJm" }, "source": [ "## Part 2: Linear Fit" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "W6h84PM2EZ_K" }, "source": [ "We use statsmodels.api to fit the data using ordinary least squares (OLS)." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 70 }, "colab_type": "code", "executionInfo": { "elapsed": 1540, "status": "ok", "timestamp": 1588621714842, "user": { "displayName": "Jenny Cao", "photoUrl": "", "userId": "01180664302881281385" }, "user_tz": 240 }, "id": "6faD_ncCBfUr", "outputId": "1e66ec0e-2434-4ed8-82fa-84afbddce065" }, "outputs": [], "source": [ "import statsmodels.api as sm" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "PhvCp87gEg_G" }, "source": [ "2.1. Fit OLS on the data for product line 102. Use the sales data as your outcome y. Include an intercept. We will use an integer that increments by 1 each month as our single predictor. \n", "\n", "To quickly get this integer, first use the code `df2_linear = df2.reset_index()`, where this assumes that `df2` is the name of the dataframe containing data for product line 102. This creates a dataframe that contains an additional column, `index`, that contains a sequence of integers. This relies on the fact that our dataframe had data for product line 102 in increasing time order.\n", "\n", "Then call `summary()` to see the results of your fit." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": {}, "colab_type": "code", "id": "IZReSz7JBfUy" }, "outputs": [], "source": [ "# YOUR CODE HERE" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "bMwRm9h2FCY0" }, "source": [ "2.2: Plot the predictions from linear regression and your data (TotalProductSales) for product line 102." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 320 }, "colab_type": "code", "executionInfo": { "elapsed": 538, "status": "ok", "timestamp": 1588621722567, "user": { "displayName": "Jenny Cao", "photoUrl": "", "userId": "01180664302881281385" }, "user_tz": 240 }, "id": "CAiN3I_5BfVM", "outputId": "b760e639-3ebf-4c09-8014-077709be764b" }, "outputs": [], "source": [ "# YOUR CODE HERE" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "dYUMQLesFjTs" }, "source": [ "2.3 What are some drawbacks of this method?" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "k83SqZRVGVFI" }, "source": [ "**Answer here.**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "2.4 Calculate the sum of squared errors resulting from this method. To do this, calculate a series containing the difference between the actual and predicted values. Then calculate the square of each component of this series. Then sum the values. You should be able to do this in a single line of code.\n", "\n", "Given a series `x`, the series `x.pow(2)` contains the square of each element of `x`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# YOUR CODE HERE" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "7rggyhJNHJB6" }, "source": [ "## Part 3: Exponential Smoothing" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "46s6bs9IHmF6" }, "source": [ "We now use simple exponential smoothing to predict, now using product line 102. This assumes that the data has no consistent trend and consists only of a mean that slowly varies up and down and has noise added to it." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": {}, "colab_type": "code", "id": "9bGgMgTrBfVk" }, "outputs": [], "source": [ "from statsmodels.tsa.api import SimpleExpSmoothing" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "3.1: To use simple exponential smoothing, we need to create a Series containing TotalProductSales for product line 102 that has our datetime object as the index. The easiest way to do this is to create a new dataframe using set_index() and then extract the `TotalProductSales` field from this dataframe. Name the resulting Series `sales2`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# YOUR CODE HERE" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "VCUA7X-XHx8S" }, "source": [ "Then, the following line of code creates a simple exponential fit for the data.\n", "\n", "The code will say that \"No frequency information was provided, so inferred frequency MS will be used.\" This is ok -- it just means that we didn't tell SimpleExpSmoothing about a period over which we might expect to see repeating patterns." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 70 }, "colab_type": "code", "executionInfo": { "elapsed": 294, "status": "ok", "timestamp": 1588621755076, "user": { "displayName": "Jenny Cao", "photoUrl": "", "userId": "01180664302881281385" }, "user_tz": 240 }, "id": "dZ58FiAGBfVt", "outputId": "72707dd7-fa45-4974-915b-09dd02e14e2c" }, "outputs": [], "source": [ "fit1 = SimpleExpSmoothing(sales2).fit()" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "FZI4GXmZH7Bb" }, "source": [ "Then, the following code gets a forecast for the next 4 upcoming periods." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": {}, "colab_type": "code", "id": "cw2xZZj5BfVy" }, "outputs": [], "source": [ "fore1 = fit1.forecast(4)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "gx5MM7iNIU7B" }, "source": [ "3.2: Plot the values fitted to our original timeseries, the original sales data for line 102, and the forecasted values. The fitted values are in `fit1.fittedvalues`. The forecasted values can be plotted using `fore1.plot()`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 312 }, "colab_type": "code", "executionInfo": { "elapsed": 548, "status": "ok", "timestamp": 1588621759078, "user": { "displayName": "Jenny Cao", "photoUrl": "", "userId": "01180664302881281385" }, "user_tz": 240 }, "id": "iO2bNm5aIK-1", "outputId": "429f2d1f-5ba7-43a3-8333-14a073001169" }, "outputs": [], "source": [ "# YOUR CODE HERE" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "3.3 Calculate the sum of squared errors between this method's predictions and the actual value." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# YOUR CODE HERE" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "nl2I3gOVI88O" }, "source": [ "3.4: Based on the sum of squared errors, is this method better or worse than linear regression?" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "7AyMPMCAJNQ1" }, "source": [ "**Answer here.**" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "d-5NHBkHJPJV" }, "source": [ "3.5: Where does this method do particularly badly? How does this relate to your answer to question 1.5?" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "6NsJpdMeJRFZ" }, "source": [ "**Answer here.**" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "EAWKJTM_IZ8D" }, "source": [ "## Part 4: Holt Linear" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "9LYKericIgeW" }, "source": [ "Now we will fit this same data using the `Holt` method. We import the package `Holt` for this part and `ExponentialSmoothing` (also called the Holt-Winters method) for the next part.\n", "\n", "The Holt method assumes that the data has a trend and that the slope of the trend is slowly varying over time. It also assumes that there is an additional change to the overall mean that is slowly shifting everything up and down, and that the observations have noise. \n", "\n", "This method is able to make predictions that have the same form as simple exponential smoothing, and also the same form as linear regression, as two special cases." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": {}, "colab_type": "code", "id": "uBSXy56LBfV7" }, "outputs": [], "source": [ "## Holt Double Exponential Linear\n", "from statsmodels.tsa.holtwinters import Holt, ExponentialSmoothing" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "xyLjx_grIqbe" }, "source": [ "4.1: Fit the Holt method on the data series created for the previous part and generate a forecast for the next 5 values. Above we used the code \n", "```\n", "fit1 = SimpleExpSmoothing(sales2).fit()\n", "fore1 = fit1.forecast(4)\n", "```\n", "to fit simple exponential smoothing and forecast for 4 values. The code for the Holt method is similar, except we replace `SimpleExpSmoothing` by `Holt`, and we forecast for 5 values instead of 4." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 70 }, "colab_type": "code", "executionInfo": { "elapsed": 360, "status": "ok", "timestamp": 1588621779195, "user": { "displayName": "Jenny Cao", "photoUrl": "", "userId": "01180664302881281385" }, "user_tz": 240 }, "id": "MitpjBxrBfWB", "outputId": "a2981334-a56e-4224-fe96-89ca12ea2278" }, "outputs": [], "source": [ "# YOUR CODE HERE" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "MnHTNRuKI0Oe" }, "source": [ "4.2: Now plot the forecast, fitted values, and orignal sales data." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 312 }, "colab_type": "code", "executionInfo": { "elapsed": 474, "status": "ok", "timestamp": 1588621783872, "user": { "displayName": "Jenny Cao", "photoUrl": "", "userId": "01180664302881281385" }, "user_tz": 240 }, "id": "lvWyJ9OpBfWM", "outputId": "ddf35519-11c2-420c-f75e-7222cbe72b3f" }, "outputs": [], "source": [ "# YOUR CODE HERE" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "4.3 Calculate the sum of square errors between this method's predictions and the actual value." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# YOUR CODE HERE" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "H1RjBjuvJcEo" }, "source": [ "4.4 How does the sum of squared errors for this method compare to simple exponential smoothing? To linear regression? Why do you think this is?" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "tYQ1sWJeJt4E" }, "source": [ "**Answer here.**" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "uAyYTc2wJwCg" }, "source": [ "## Part 5: Holt-Winters Method" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "FrmBJScyJysG" }, "source": [ "5.1: Now we will use the Holt-Winters method, also called ExponentialSmoothing, on the same series of data. This allows for a trend (either additive or multiplicative), and seasonality (also, either additive or multiplicative), which we specify when we create the ExponentialSmoothing object. We also need to specify the number of periods to use for our seasonality.\n", "\n", "We'll create this object using this code:\n", "\n", "```\n", "ExponentialSmoothing(sales2, trend = \"add\", seasonal = \"add\", seasonal_periods = ?)\n", "```\n", "where you need to fill in the value of seasonal_periods. This is the number of periods (each datapoint is 1 period) over which we see the strongest repeating period. Keep in mind that each datapoint is a month, and keep in mind what you learned in question 1.4 and 1.5 about patterns in the data.\n", "\n", "Fit the Holt-Winters method and make a forecast for 12 periods." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 70 }, "colab_type": "code", "executionInfo": { "elapsed": 353, "status": "ok", "timestamp": 1588621786691, "user": { "displayName": "Jenny Cao", "photoUrl": "", "userId": "01180664302881281385" }, "user_tz": 240 }, "id": "pjJEuVujKPGH", "outputId": "20af3612-6b2c-48d4-9eba-fae34ce58644" }, "outputs": [], "source": [ "# YOUR CODE HERE" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "mEFzi_G2KZ_e" }, "source": [ "5.2: Plot the fit, forecasts, and original sales data." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 312 }, "colab_type": "code", "executionInfo": { "elapsed": 459, "status": "ok", "timestamp": 1588621791790, "user": { "displayName": "Jenny Cao", "photoUrl": "", "userId": "01180664302881281385" }, "user_tz": 240 }, "id": "hjdGZNpbBfWR", "outputId": "8a147bca-b483-4907-ad62-2533ebd65c7f" }, "outputs": [], "source": [ "## YOUR CODE HERE" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "IdtvGSvEKijX" }, "source": [ "This is a good opportunity to changing the trend and seasonality type to see what happens. You can set this to `\"mul\"` for either `trend` or `seasonal` to use a multiplicative trend or seasonality." ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "HJKPXPKIKuW-" }, "source": [ "If you chose seasonal_periods well, you should see that this method works really well --- much better than the other methods.\n", "\n", "5.3 Calculate the sum of squared errors between this method's predictions and the actual value. How does it compare to the sum of squared errors for the other methods?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# YOUR CODE HERE" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "5.4: What is it that this method does that allows it to do so well?" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "EmwKfX8fK1C8" }, "source": [ "**Answer here.**" ] } ], "metadata": { "colab": { "name": "Rec12 SOLUTIONS.ipynb", "provenance": [], "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.4" } }, "nbformat": 4, "nbformat_minor": 1 }