Part 1

  • Dataframes
  • describe()
  • head()
  • Series
  • operations on Series
  • pulling out subsets of columns
In [29]:
import pandas as pd
import numpy as np
In [3]:
df = pd.read_csv('taxi.csv')

df is a "pandas dataframe"

This is like an SQL table:

  • it has fields, which have a name and a datatype
  • it has rows, which have a value (or NA, which is like NULL) for each column

Each row also has an "index" associated with it.

When we output the dataframe, as in the following cell, we see the values in each row and we also see the value of the index in the first column. In this example, the index is just a number between 0 and 2302086.

The data that is in this dataframe is the NYC taxi data, downloaded from https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page. Each record is a taxi trip in NYC.

In [5]:
df
Out[5]:
VendorID tpep_pickup_datetime tpep_dropoff_datetime passenger_count trip_distance RatecodeID store_and_fwd_flag PULocationID DOLocationID payment_type fare_amount extra mta_tax tip_amount tolls_amount improvement_surcharge total_amount
0 2 2017-06-08 07:52:31 2017-06-08 08:01:32 6 1.03 1 N 161 140 1 7.5 1.0 0.5 1.86 0.0 0.3 11.16
1 2 2017-06-08 08:08:18 2017-06-08 08:14:00 6 1.03 1 N 162 233 1 6.0 1.0 0.5 2.34 0.0 0.3 10.14
2 2 2017-06-08 08:16:49 2017-06-08 15:43:22 6 5.63 1 N 137 41 2 21.5 1.0 0.5 0.00 0.0 0.3 23.30
3 1 2017-06-09 17:59:37 2017-06-09 18:04:40 1 0.60 1 N 164 234 1 5.0 1.0 0.5 1.02 0.0 0.3 7.82
4 1 2017-06-09 17:59:37 2017-06-09 18:19:26 1 2.80 1 N 229 113 1 14.5 1.0 0.5 3.25 0.0 0.3 19.55
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
2302082 2 2017-06-11 23:59:50 2017-06-12 00:04:10 5 1.31 1 N 7 193 2 6.0 0.5 0.5 0.00 0.0 0.3 7.30
2302083 2 2017-06-11 23:59:52 2017-06-12 00:11:03 1 3.17 1 N 234 236 2 11.5 0.5 0.5 0.00 0.0 0.3 12.80
2302084 2 2017-06-11 23:59:58 2017-06-12 00:13:24 1 3.65 1 N 148 17 2 14.0 0.5 0.5 0.00 0.0 0.3 15.30
2302085 2 2017-06-11 23:59:58 2017-06-12 00:07:21 1 1.01 1 N 230 163 2 6.5 0.5 0.5 0.00 0.0 0.3 7.80
2302086 2 2017-06-11 23:59:59 2017-06-12 00:11:15 1 2.07 1 N 48 249 1 9.5 0.5 0.5 2.16 0.0 0.3 12.96

2302087 rows × 17 columns

Some of the important fields are:

  • tpep_pickup_datetime: this is the date and time when the trip started. (“tpep” stands for Taxicab Passenger Enhancements Project, which is the name of the project that collected this data).
  • tpep_dropoff_datetime: this is the date and time when the trip stopped.
  • passenger_count: this is the number of passengers in the taxi
  • trip_distance: this is the distance of the trip, in miles
  • PULocationID: this id corresponds to a LocationId in zones, indicating the zone in which the pickup occurred. “PU” stands for “pickup.”
  • DOLocationID: this id corresponds to a LocationId in zones, indicating the zone in which the dropoff occurred. “DO” stands for “dropoff.”
  • total_amount: this is the total amount of money paid by the rider for the trip. Components of this amount are broken out across the fields fare_amount, extra, mta_tax, tip_amount, tolls_amount

More information about the fields can be found at, https://www1.nyc.gov/assets/tlc/downloads/pdf/data_dictionary_trip_records_yellow.pdf

Dataframes have methods associated with them, which you can call by using syntax of the form df.method(arguments)

For example, df.head shows the first few records from the dataframe --- useful if the dataframe is big.

In [4]:
df.head()
Out[4]:
VendorID tpep_pickup_datetime tpep_dropoff_datetime passenger_count trip_distance RatecodeID store_and_fwd_flag PULocationID DOLocationID payment_type fare_amount extra mta_tax tip_amount tolls_amount improvement_surcharge total_amount
0 2 2017-06-08 07:52:31 2017-06-08 08:01:32 6 1.03 1 N 161 140 1 7.5 1.0 0.5 1.86 0.0 0.3 11.16
1 2 2017-06-08 08:08:18 2017-06-08 08:14:00 6 1.03 1 N 162 233 1 6.0 1.0 0.5 2.34 0.0 0.3 10.14
2 2 2017-06-08 08:16:49 2017-06-08 15:43:22 6 5.63 1 N 137 41 2 21.5 1.0 0.5 0.00 0.0 0.3 23.30
3 1 2017-06-09 17:59:37 2017-06-09 18:04:40 1 0.60 1 N 164 234 1 5.0 1.0 0.5 1.02 0.0 0.3 7.82
4 1 2017-06-09 17:59:37 2017-06-09 18:19:26 1 2.80 1 N 229 113 1 14.5 1.0 0.5 3.25 0.0 0.3 19.55

Another nice way to explore the data is to use describe. This tells you, for each field:

  • the number of non-NA records
  • the mean of these records
  • their standard deviation
  • the minimum and maximum values
  • three quartiles: 25%, 50%, 75%. The 25% quartile is the number x such that 25% of the records have a value less than or equal to x

This is quite meaningful for some fields (like fare_amount)

In [8]:
df.describe()
Out[8]:
VendorID passenger_count trip_distance RatecodeID PULocationID DOLocationID payment_type fare_amount extra mta_tax tip_amount tolls_amount improvement_surcharge total_amount
count 2.302087e+06 2.302087e+06 2.302087e+06 2.302087e+06 2.302087e+06 2.302087e+06 2.302087e+06 2.302087e+06 2.302087e+06 2.302087e+06 2.302087e+06 2.302087e+06 2.302087e+06 2.302087e+06
mean 1.546358e+00 1.623310e+00 2.972721e+00 1.044957e+00 1.633337e+02 1.613443e+02 1.329988e+00 1.317063e+01 3.384624e-01 4.972623e-01 1.886104e+00 3.360965e-01 2.996151e-01 1.653247e+01
std 4.978464e-01 1.264212e+00 4.667502e+00 5.583822e-01 6.680174e+01 7.058144e+01 4.908489e-01 1.458502e+01 4.613124e-01 6.238771e-02 2.685841e+00 1.949032e+00 1.422852e-02 1.692001e+01
min 1.000000e+00 0.000000e+00 0.000000e+00 1.000000e+00 1.000000e+00 1.000000e+00 1.000000e+00 -2.226600e+02 -5.056000e+01 -5.000000e-01 -5.000000e+01 -1.250000e+01 -3.000000e-01 -2.234600e+02
25% 1.000000e+00 1.000000e+00 1.000000e+00 1.000000e+00 1.140000e+02 1.070000e+02 1.000000e+00 6.500000e+00 0.000000e+00 5.000000e-01 0.000000e+00 0.000000e+00 3.000000e-01 8.750000e+00
50% 2.000000e+00 1.000000e+00 1.680000e+00 1.000000e+00 1.620000e+02 1.620000e+02 1.000000e+00 9.500000e+00 0.000000e+00 5.000000e-01 1.360000e+00 0.000000e+00 3.000000e-01 1.180000e+01
75% 2.000000e+00 2.000000e+00 3.100000e+00 1.000000e+00 2.330000e+02 2.330000e+02 2.000000e+00 1.500000e+01 5.000000e-01 5.000000e-01 2.460000e+00 0.000000e+00 3.000000e-01 1.788000e+01
max 2.000000e+00 9.000000e+00 4.208940e+03 9.900000e+01 2.650000e+02 2.650000e+02 5.000000e+00 9.999990e+03 7.060000e+00 5.136000e+01 4.115000e+02 9.000800e+02 1.000000e+00 9.999990e+03

We can pull out an individual field from a dataframe like in the following cell. This creates a "Series".

A Series is essentially a single column of a dataframe. It contains:

  • A series of values (the data itself)
  • The value of the corresponding index from the original dataframe
  • The name of the data (here, "total_amount"), derived from the original dataframe.
  • The number of values in the series
  • The datatype for the data (here, float64)

The following creates a series for the amount of money paid for each taxi trip.

In [18]:
df['total_amount']
Out[18]:
0          11.16
1          10.14
2          23.30
3           7.82
4          19.55
           ...  
2302082     7.30
2302083    12.80
2302084    15.30
2302085     7.80
2302086    12.96
Name: total_amount, Length: 2302087, dtype: float64

We can do operations on Series, like the following

In [34]:
df['total_amount']/10
Out[34]:
0          1.116
1          1.014
2          2.330
3          0.782
4          1.955
           ...  
2302082    0.730
2302083    1.280
2302084    1.530
2302085    0.780
2302086    1.296
Name: total_amount, Length: 2302087, dtype: float64

This takes each value in the series, e.g., the first one 11.16, divides it by 10, and then returns a Series with the corresponding values.

We can also do operations on collections of Series, like the following cell. This gives us the tip, as a fraction of the pre-tip fare.

In [40]:
df['tip_amount']/(df['total_amount']-df['tip_amount'])
Out[40]:
0          0.200000
1          0.300000
2          0.000000
3          0.150000
4          0.199387
             ...   
2302082    0.000000
2302083    0.000000
2302084    0.000000
2302085    0.000000
2302086    0.200000
Length: 2302087, dtype: float64

We can also assign variables to the results of these operations, like the following. Note that when we do this, the notebook doesn't output the value. This is because a notebook cell evaluates to the return value of the last command. When the last command is an assignment, this doesn't have a return value.

In [47]:
tip = df['tip_amount']/(df['total_amount']-df['tip_amount'])

In addition to pulling out just one column from a dataframe, we can create a new dataframe that has a subset of its columns, like this. What this line does is to create a list of column names, ['tip_amount','total_amount'], and then pass this as an argument to df[].

In [85]:
df[['tip_amount','total_amount']]
Out[85]:
tip_amount total_amount
0 1.86 11.16
1 2.34 10.14
2 0.00 23.30
3 1.02 7.82
4 3.25 19.55
... ... ...
2302082 0.00 7.30
2302083 0.00 12.80
2302084 0.00 15.30
2302085 0.00 7.80
2302086 2.16 12.96

2302087 rows × 2 columns

It might be easier to understand what we did in the cell above if we do it in two steps, like the following.

In [86]:
x = ['tip_amount','total_amount']
df[x]
Out[86]:
tip_amount total_amount
0 1.86 11.16
1 2.34 10.14
2 0.00 23.30
3 1.02 7.82
4 3.25 19.55
... ... ...
2302082 0.00 7.30
2302083 0.00 12.80
2302084 0.00 15.30
2302085 0.00 7.80
2302086 2.16 12.96

2302087 rows × 2 columns

Question 1: How would I calculate the amount of money per mile?

  • A. df[total_amount]/df[trip_distance]
  • B. df['total_amount']/df['trip_distance']
  • C. df[total_amount/trip_distance]
  • D. df['total_amount'/'trip_distance']
In [145]:
# Put answer to question 1 here
Out[145]:
0          10.834951
1           9.844660
2           4.138544
3          13.033333
4           6.982143
             ...    
2302082     5.572519
2302083     4.037855
2302084     4.191781
2302085     7.722772
2302086     6.260870
Length: 2302087, dtype: float64

Part 2

  • Making a histogram with matplotlib
  • Selecting a subset of the records (i.e., "filtering")
  • Titles and axis labels

matplotlib is a really useful plotting package.

matplotlib.pyplot is a part of matplotlib that makes it a little easier to plot.

In [20]:
import matplotlib.pyplot as plt

Let's make a histogram of the tip percentage

In [48]:
tip.hist()
Out[48]:
<matplotlib.axes._subplots.AxesSubplot at 0x7fc3d123e990>

This histogram might seem strange at first. There is a single bar between 0 and 100 and a lot of space on the x-axis that appears empty. Let's see what's going on.

In [49]:
tip.describe()
Out[49]:
count    2.301762e+06
mean     1.295830e-01
std      8.239313e-01
min      0.000000e+00
25%      0.000000e+00
50%      1.694915e-01
75%      2.000000e-01
max      9.500000e+02
dtype: float64

It seems that there are some very large tip fractions --- the largest is 9.5e02 = 95. This is someone who gave a tip that was 95 times larger than the untipped original fare. But most tip fractions are smaller -- the 75% quantile is 2e-01, or 20%.

To focus on just a part of the data, we can filter. Let's create a dataset that only contains records whose tip fraction is less than or equal to 1.

To do this, look at the following code. It produces a Series of boolean values whose index is the same as tip and the original dataframe, which is True whenever tip is <= 1, and False otherwise.

In [57]:
tip<=1
Out[57]:
0          True
1          True
2          True
3          True
4          True
           ... 
2302082    True
2302083    True
2302084    True
2302085    True
2302086    True
Length: 2302087, dtype: bool

We can then get a Series which only has the values of tip associated with indices for which tip<=1 is True. Note that this has 2300307 records, while the original tip Series had 2302087 records. This must mean that there are 2302087 - 2300307 = 1780 records where tip is strictly bigger than 1.

In [77]:
tip[tip<=1]
Out[77]:
0          0.200000
1          0.300000
2          0.000000
3          0.150000
4          0.199387
             ...   
2302082    0.000000
2302083    0.000000
2302084    0.000000
2302085    0.000000
2302086    0.200000
Length: 2300307, dtype: float64

We can also use this Series of boolean values to get a subset of our original dataframe, or any other Series or dataframe that uses the same index.

In [79]:
df[tip<=1]
Out[79]:
VendorID tpep_pickup_datetime tpep_dropoff_datetime passenger_count trip_distance RatecodeID store_and_fwd_flag PULocationID DOLocationID payment_type fare_amount extra mta_tax tip_amount tolls_amount improvement_surcharge total_amount
0 2 2017-06-08 07:52:31 2017-06-08 08:01:32 6 1.03 1 N 161 140 1 7.5 1.0 0.5 1.86 0.0 0.3 11.16
1 2 2017-06-08 08:08:18 2017-06-08 08:14:00 6 1.03 1 N 162 233 1 6.0 1.0 0.5 2.34 0.0 0.3 10.14
2 2 2017-06-08 08:16:49 2017-06-08 15:43:22 6 5.63 1 N 137 41 2 21.5 1.0 0.5 0.00 0.0 0.3 23.30
3 1 2017-06-09 17:59:37 2017-06-09 18:04:40 1 0.60 1 N 164 234 1 5.0 1.0 0.5 1.02 0.0 0.3 7.82
4 1 2017-06-09 17:59:37 2017-06-09 18:19:26 1 2.80 1 N 229 113 1 14.5 1.0 0.5 3.25 0.0 0.3 19.55
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
2302082 2 2017-06-11 23:59:50 2017-06-12 00:04:10 5 1.31 1 N 7 193 2 6.0 0.5 0.5 0.00 0.0 0.3 7.30
2302083 2 2017-06-11 23:59:52 2017-06-12 00:11:03 1 3.17 1 N 234 236 2 11.5 0.5 0.5 0.00 0.0 0.3 12.80
2302084 2 2017-06-11 23:59:58 2017-06-12 00:13:24 1 3.65 1 N 148 17 2 14.0 0.5 0.5 0.00 0.0 0.3 15.30
2302085 2 2017-06-11 23:59:58 2017-06-12 00:07:21 1 1.01 1 N 230 163 2 6.5 0.5 0.5 0.00 0.0 0.3 7.80
2302086 2 2017-06-11 23:59:59 2017-06-12 00:11:15 1 2.07 1 N 48 249 1 9.5 0.5 0.5 2.16 0.0 0.3 12.96

2300307 rows × 17 columns

You might be surprised that pandas produces meaningful results in both of the following cases:

  • df[tip <= 1]
  • df['total_amount']

When you call df[x], pandas looks at what is in x and then figures out what to do.

  • If x is a Series using the same index as df, then pandas returns the records whose value in the Series is True.
  • If x is a column name or list of column names, then pandas returns the fields with those names.

We can make a histogram of this subset of the tip data. This makes a lot more sense --- most tips are somewhere between 0 and 30%.

In [62]:
tip[tip<=1].hist()
Out[62]:
<matplotlib.axes._subplots.AxesSubplot at 0x7fc3f3260910>

Here's a better version of this graph that would be suitable to show to an external stakeholder. It has these improvements:

  • it increases the number of bins, so we can see more granularity between 0% and 20%
  • It adds labels for the x- and y-axis
  • It adds a title that informs the viewer that outliers have been removed
  • It reports results in terms of percentages rather than fractions. Most business people are more comfortable with percentages.
In [141]:
(100*tip[tip<=1]).hist(bins=30)
plt.xlabel('Tip Percentage (%)')
plt.ylabel('Number of trips')
plt.title('Tip Percentage Histogram, Discarding 1780 Outliers')
Out[141]:
Text(0.5, 1.0, 'Tip Percentage Histogram, Discarding 1780 Outliers')

Part 3

  • applying np.log10 to a Series
  • practice manipulating data

Whenever we remove outliers, we should dig in to understand them.

A first thing to do is to look at the original data corresponding to large tip amounts

In [88]:
df[tip>1]
Out[88]:
VendorID tpep_pickup_datetime tpep_dropoff_datetime passenger_count trip_distance RatecodeID store_and_fwd_flag PULocationID DOLocationID payment_type fare_amount extra mta_tax tip_amount tolls_amount improvement_surcharge total_amount
786 1 2017-06-09 18:04:59 2017-06-09 18:05:23 1 0.00 1 N 158 158 1 2.5 1.0 0.5 8.00 0.0 0.3 12.30
3408 1 2017-06-09 18:22:37 2017-06-09 18:24:30 1 0.50 1 N 48 163 1 4.0 1.0 0.5 10.00 0.0 0.3 15.80
3548 1 2017-06-09 18:23:33 2017-06-09 18:28:26 1 0.60 1 Y 229 233 1 5.0 1.0 0.5 15.00 0.0 0.3 21.80
3594 1 2017-06-09 18:23:53 2017-06-09 18:37:43 1 1.90 1 N 138 138 1 10.0 1.0 0.5 25.00 0.0 0.3 36.80
5149 1 2017-06-09 18:34:05 2017-06-09 18:34:25 1 0.00 1 N 142 142 1 2.5 1.0 0.5 5.00 0.0 0.3 9.30
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
2299885 2 2017-06-11 23:33:41 2017-06-11 23:41:36 4 1.33 1 N 164 48 1 7.5 0.5 0.5 10.00 0.0 0.3 18.80
2300550 2 2017-06-11 23:40:52 2017-06-11 23:54:10 4 1.63 1 N 239 143 1 8.5 0.5 0.5 37.06 0.0 0.3 46.86
2301013 2 2017-06-11 23:46:07 2017-06-11 23:46:11 1 0.00 1 N 264 170 1 2.5 0.5 0.5 10.00 0.0 0.3 13.80
2301189 2 2017-06-11 23:48:15 2017-06-11 23:50:05 1 0.41 1 N 177 35 1 3.5 0.5 0.5 12.00 0.0 0.3 16.80
2301898 2 2017-06-11 23:56:33 2017-06-11 23:57:11 1 0.32 1 N 107 107 1 3.0 0.5 0.5 5.88 0.0 0.3 10.18

1455 rows × 17 columns

This is its own dataframe, so we can pull out just a subset of the columns to get a more refined view.

Q2: How would we get a dataframe that includes only tip_amount and total_amount, and only those records whose tip fraction is strictly above 1?

  • A. df[tip>1][['tip_amount','total_amount']]
  • B. df[tip>1]['tip_amount','total_amount']
  • C. df['tip_amount','total_amount'][tip>1]
  • D. df[['tip_amount','total_amount']][tip>1]
  • E. A and D
  • F. B and C
In [98]:
# Plug in answer to Q2 here
Out[98]:
tip_amount total_amount
786 8.00 12.30
3408 10.00 15.80
3548 15.00 21.80
3594 25.00 36.80
5149 5.00 9.30
... ... ...
2299885 10.00 18.80
2300550 37.06 46.86
2301013 10.00 13.80
2301189 12.00 16.80
2301898 5.88 10.18

1455 rows × 2 columns

From this, it seems like the data is reasonable --- occasionally people seem to give generous tips, but it doesn't seem outrageous.

We can also investigate tips that have a large magnitude on an absolute scale.

Q3. Which of the following lines of code would produce a dataframe containing all fields for those trips whose tip amount (as an absolute \$ value) is above \$100.

  • A. df['tip_amount'>100]
  • B. df[['tip_amount'>100]]
  • C. df[df['tip_amount'>100]]
  • D. df[df['tip_amount']>100]
In [116]:
# Insert answer to Q3 here
Out[116]:
VendorID tpep_pickup_datetime tpep_dropoff_datetime passenger_count trip_distance RatecodeID store_and_fwd_flag PULocationID DOLocationID payment_type fare_amount extra mta_tax tip_amount tolls_amount improvement_surcharge total_amount
158645 1 2017-06-10 18:17:28 2017-06-10 19:41:13 1 43.30 5 N 48 265 1 133.00 0.0 0.0 250.00 16.50 0.3 399.80
196024 1 2017-06-10 22:50:08 2017-06-10 23:04:53 1 0.40 1 N 68 230 1 9.50 0.5 0.5 350.00 0.00 0.3 360.80
233164 1 2017-06-11 05:30:26 2017-06-11 06:08:29 1 30.60 1 N 242 168 1 80.00 0.5 0.5 120.00 0.00 0.3 201.30
381658 1 2017-06-05 16:25:59 2017-06-05 16:27:33 1 1.60 5 N 265 265 1 0.00 0.0 0.0 285.00 0.00 0.3 285.30
501877 1 2017-06-06 13:13:37 2017-06-06 13:13:50 1 0.00 2 N 265 265 1 52.00 0.0 0.5 115.00 0.00 0.3 167.80
573910 1 2017-06-06 21:29:10 2017-06-06 22:13:40 1 23.80 5 N 170 265 1 0.00 0.0 0.0 115.00 10.50 0.3 125.80
739352 1 2017-06-07 22:50:32 2017-06-07 22:50:52 1 2.80 1 N 265 265 1 2.50 0.5 0.5 110.00 0.00 0.3 113.80
762537 1 2017-06-08 06:13:59 2017-06-08 06:14:43 1 3.00 1 N 265 265 1 2.50 0.0 0.5 350.00 0.00 0.3 353.30
912782 1 2017-06-09 00:38:12 2017-06-09 00:38:21 1 0.00 1 N 265 265 1 2.50 0.5 0.5 127.00 0.00 0.3 130.80
988589 1 2017-06-09 14:47:20 2017-06-09 15:31:00 1 27.20 5 N 238 265 1 0.00 0.0 0.0 149.00 0.00 0.3 149.30
1009810 1 2017-06-09 17:37:27 2017-06-09 18:34:24 1 19.00 5 N 70 265 1 0.00 0.0 0.0 115.00 5.76 0.3 121.06
1095464 2 2017-06-05 07:34:21 2017-06-05 08:48:59 2 13.75 2 N 10 233 1 52.00 0.0 0.5 120.00 5.76 0.3 178.56
1325034 2 2017-06-06 10:59:00 2017-06-06 11:07:00 1 1.80 1 N 168 74 1 8.00 0.0 0.5 335.35 0.00 0.3 344.15
1425034 2 2017-06-06 20:50:34 2017-06-06 21:19:07 1 19.42 5 N 132 116 1 68.00 0.0 0.5 208.00 11.50 0.3 288.30
1514177 2 2017-06-07 11:12:14 2017-06-07 12:08:51 1 20.59 1 N 138 246 1 60.50 0.0 0.5 120.00 5.76 0.3 187.06
1575775 2 2017-06-07 23:15:02 2017-06-07 23:22:06 2 1.32 1 N 107 100 1 7.00 0.5 0.5 337.06 0.00 0.3 345.36
1927971 2 2017-06-09 20:58:18 2017-06-09 21:12:54 1 2.80 1 N 48 224 1 12.00 0.5 0.5 411.50 0.00 0.3 424.80
1947024 2 2017-06-09 22:46:20 2017-06-09 22:46:41 1 0.00 2 N 33 33 1 52.00 0.0 0.5 150.00 0.00 0.3 202.80
2009916 2 2017-06-10 09:53:13 2017-06-10 11:36:20 1 52.23 1 N 229 265 1 142.00 0.0 0.5 140.00 0.00 0.3 282.80
2101355 2 2017-06-10 19:32:21 2017-06-10 20:23:52 5 19.23 2 N 132 41 1 52.00 0.0 0.5 258.00 5.76 0.3 316.56
2163960 2 2017-06-11 02:14:11 2017-06-11 02:14:14 1 0.00 5 N 48 48 1 5.35 0.0 0.5 111.47 0.00 0.3 117.62
2267596 2 2017-06-11 19:06:06 2017-06-11 19:32:05 6 7.45 1 N 28 28 1 24.50 0.0 0.5 200.05 0.00 0.3 225.35
2272454 2 2017-06-11 19:40:05 2017-06-11 20:52:17 4 17.52 2 N 132 164 1 52.00 0.0 0.5 226.76 12.50 0.3 292.06

These trips are clearly unusual. For example, second from the top we see a trip with a total fare of \$360.80, of which \\$350 was a tip. This trip only took 0.4 miles and lasted about 15 minutes. Note that most people can walk more than 0.4 miles quite easily in 15 minutes. Looking at https://www1.nyc.gov/assets/tlc/images/content/pages/about/taxi_zone_map_manhattan.jpg, this picked up in Chelsea (68) and dropped off in Midtown (230).

There could be a very interesting story behind this, fraud, or a data glitch.

The fourth row has a total amount of \$285.30 and a tip of \\$285.

Another useful way to understand the distribution of these large tips is to make a histogram of the log of the tip fraction. To do this, we can use the log10 function from numpy. If x is a series, then np.log10(x) will produce a series whose values are the base-10 log of the values in x.

Q4. Which of the following lines of code would produce a histogram of the base-10 log of the tip fraction, for those tip fractions bigger than 1?

  • A. np.log10(tip[tip>1]).hist()
  • B. np.log10(tip>1).hist()
  • C. tip(np.log10(tip>1)).hist()
  • D. hist(np.log10(tip>1))
In [76]:
# Insert answer to Q3 here
Out[76]:
<matplotlib.axes._subplots.AxesSubplot at 0x7fc3f43df1d0>

Next lecture

  • scatter plots
  • subsampling data
  • filtering with AND clauses
  • group by
  • joins
  • pulling data from a SQL database
  • manipulating data in a SQL database