import pandas as pd
import numpy as np
df = pd.read_csv('taxi.csv')
df is a "pandas dataframe"
This is like an SQL table:
Each row also has an "index" associated with it.
When we output the dataframe, as in the following cell, we see the values in each row and we also see the value of the index in the first column. In this example, the index is just a number between 0 and 2302086.
The data that is in this dataframe is the NYC taxi data, downloaded from https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page. Each record is a taxi trip in NYC.
df
Some of the important fields are:
More information about the fields can be found at, https://www1.nyc.gov/assets/tlc/downloads/pdf/data_dictionary_trip_records_yellow.pdf
Dataframes have methods associated with them, which you can call by using syntax of the form df.method(arguments)
For example, df.head shows the first few records from the dataframe --- useful if the dataframe is big.
df.head()
Another nice way to explore the data is to use describe. This tells you, for each field:
This is quite meaningful for some fields (like fare_amount)
df.describe()
We can pull out an individual field from a dataframe like in the following cell. This creates a "Series".
A Series is essentially a single column of a dataframe. It contains:
The following creates a series for the amount of money paid for each taxi trip.
df['total_amount']
We can do operations on Series, like the following
df['total_amount']/10
This takes each value in the series, e.g., the first one 11.16, divides it by 10, and then returns a Series with the corresponding values.
We can also do operations on collections of Series, like the following cell. This gives us the tip, as a fraction of the pre-tip fare.
df['tip_amount']/(df['total_amount']-df['tip_amount'])
We can also assign variables to the results of these operations, like the following. Note that when we do this, the notebook doesn't output the value. This is because a notebook cell evaluates to the return value of the last command. When the last command is an assignment, this doesn't have a return value.
tip = df['tip_amount']/(df['total_amount']-df['tip_amount'])
In addition to pulling out just one column from a dataframe, we can create a new dataframe that has a subset of its columns, like this. What this line does is to create a list of column names, ['tip_amount','total_amount'], and then pass this as an argument to df[].
df[['tip_amount','total_amount']]
It might be easier to understand what we did in the cell above if we do it in two steps, like the following.
x = ['tip_amount','total_amount']
df[x]
Question 1: How would I calculate the amount of money per mile?
df[total_amount]/df[trip_distance]df['total_amount']/df['trip_distance']df[total_amount/trip_distance]df['total_amount'/'trip_distance']# Put answer to question 1 here
matplotlib is a really useful plotting package.
matplotlib.pyplot is a part of matplotlib that makes it a little easier to plot.
import matplotlib.pyplot as plt
Let's make a histogram of the tip percentage
tip.hist()
This histogram might seem strange at first. There is a single bar between 0 and 100 and a lot of space on the x-axis that appears empty. Let's see what's going on.
tip.describe()
It seems that there are some very large tip fractions --- the largest is 9.5e02 = 95. This is someone who gave a tip that was 95 times larger than the untipped original fare. But most tip fractions are smaller -- the 75% quantile is 2e-01, or 20%.
To focus on just a part of the data, we can filter. Let's create a dataset that only contains records whose tip fraction is less than or equal to 1.
To do this, look at the following code. It produces a Series of boolean values whose index is the same as tip and the original dataframe, which is True whenever tip is <= 1, and False otherwise.
tip<=1
We can then get a Series which only has the values of tip associated with indices for which tip<=1 is True. Note that this has 2300307 records, while the original tip Series had 2302087 records. This must mean that there are 2302087 - 2300307 = 1780 records where tip is strictly bigger than 1.
tip[tip<=1]
We can also use this Series of boolean values to get a subset of our original dataframe, or any other Series or dataframe that uses the same index.
df[tip<=1]
You might be surprised that pandas produces meaningful results in both of the following cases:
df[tip <= 1]df['total_amount']When you call df[x], pandas looks at what is in x and then figures out what to do.
We can make a histogram of this subset of the tip data. This makes a lot more sense --- most tips are somewhere between 0 and 30%.
tip[tip<=1].hist()
Here's a better version of this graph that would be suitable to show to an external stakeholder. It has these improvements:
(100*tip[tip<=1]).hist(bins=30)
plt.xlabel('Tip Percentage (%)')
plt.ylabel('Number of trips')
plt.title('Tip Percentage Histogram, Discarding 1780 Outliers')
Whenever we remove outliers, we should dig in to understand them.
A first thing to do is to look at the original data corresponding to large tip amounts
df[tip>1]
This is its own dataframe, so we can pull out just a subset of the columns to get a more refined view.
Q2: How would we get a dataframe that includes only tip_amount and total_amount, and only those records whose tip fraction is strictly above 1?
df[tip>1][['tip_amount','total_amount']]df[tip>1]['tip_amount','total_amount']df['tip_amount','total_amount'][tip>1]df[['tip_amount','total_amount']][tip>1]# Plug in answer to Q2 here
From this, it seems like the data is reasonable --- occasionally people seem to give generous tips, but it doesn't seem outrageous.
We can also investigate tips that have a large magnitude on an absolute scale.
Q3. Which of the following lines of code would produce a dataframe containing all fields for those trips whose tip amount (as an absolute \$ value) is above \$100.
df['tip_amount'>100]df[['tip_amount'>100]]df[df['tip_amount'>100]]df[df['tip_amount']>100]# Insert answer to Q3 here
These trips are clearly unusual. For example, second from the top we see a trip with a total fare of \$360.80, of which \\$350 was a tip. This trip only took 0.4 miles and lasted about 15 minutes. Note that most people can walk more than 0.4 miles quite easily in 15 minutes. Looking at https://www1.nyc.gov/assets/tlc/images/content/pages/about/taxi_zone_map_manhattan.jpg, this picked up in Chelsea (68) and dropped off in Midtown (230).
There could be a very interesting story behind this, fraud, or a data glitch.
The fourth row has a total amount of \$285.30 and a tip of \\$285.
Another useful way to understand the distribution of these large tips is to make a histogram of the log of the tip fraction. To do this, we can use the log10 function from numpy. If x is a series, then np.log10(x) will produce a series whose values are the base-10 log of the values in x.
Q4. Which of the following lines of code would produce a histogram of the base-10 log of the tip fraction, for those tip fractions bigger than 1?
np.log10(tip[tip>1]).hist()np.log10(tip>1).hist()tip(np.log10(tip>1)).hist()hist(np.log10(tip>1))# Insert answer to Q3 here