Key Dates

  • 3/11: Team Formation Assignment
  • 4/18: First Milestone due
  • 4/27: Within-team feedback for first milestone
  • 4/27: Peer grading due for first milestone
  • 5/16: Final Report due
  • 5/21 11:59pm: Within-team feedback for final report milestone
  • 5/22 11:59pm: Peer grading due for final report

Example high-quality projects from past years

Link to Google Drive folder

Project description

The class will culminate in a final project in which you use the skills you've learned in the course to analyze an interesting dataset. This project will give students experience in the kind of work that a data scientist might perform in industry, or at a government or nonprofit agency. These projects will be completed in groups of 3 or 4 students, and in groups of 2 with instructor permission.

There will be three two major components:

  1. Team Formation Assignment: You'll submit your team members, the dataset you plan to use, at least one question you plan to explore, and at least one link to a Google drive folder or github repository that you'll use to collaborate. This will not be graded but we do require all students to be on a team and all teams to submit this assignment. You can change your dataset up until the first milestone is due and also modify the link(s) to your Google drive folder / github repo.

  2. First Milestone: You will describe your dataset, discuss a few interesting and important questions that you wish to answer using the dataset, present at least three different visualizations of the data, and discuss how these visualizations give insight into the questions you seek to answer. The report will be submitted as a pdf (max 4 pages) or html (of approximately equivalent length).

  3. Final Report: You will use at least three data analysis methods learned in class to better answer the questions posed above about your dataset. The questions answered may be descriptive (what factors lead to high COVID-19 infection counts in US counties?), predictive (which countries will have more than 10 COVID-19 infections per 100K of population reported per week by September 2021?), or prescriptive (how should COVAX doses of COVID-19 vaccine be distributed in to minimize loss of life?). You may also propose a carefully-designed experiment to help go further in answering these questions. Data analysis tools that will be available for use in the class include linear regression, testing the assumptions of linear regression, logistic regression, forecasting, optimization via random search, causality (including A/B testing), simulation, and stochastic models. You may also use other tools to achieve your goal but they must be clearly explained. The report will be submitted as a pdf (max 8 pages) or html (of approximately equivalent length).

Both reports will be peer-graded and reviewed by your course instructors. Write a report you think your peers will enjoy reading!

Grading

Grades for the first milestone and project report will be based on their quality, as judged by the teaching team using input from peer grading. In most situations, all students on a team will earn the same grade for each project assignment. There are two situations where grades may differ across team members:

  • It may arise that a student does very little work compared to their teammates. In this situation, grades will be adjusted based on an individual's level of contribution. This will be assessed by within-team feedback collected for the assignment, via examination of individual contributions as assessed in the Google drive / github repository included with your team formation assignment, and, if necessary, conversations with team members. To help with this, teams are strongly encouraged to use Google docs, github, google collab, or other tools that track an individual's contribution history. Students should also be prepared to explain what they contributed individually to the project.
  • It may arise that a student's peer grading was not perfomed with sufficient care. In this situation, that student's project grade will be reduced to reflect this lack of care.

Suggested datasets

You are welcome to choose your own dataset. For those that would like suggestions, the instructors include some here.

  • Kaggle has a wide range of very interesting datasets. Two examples are:
  • Police Use of Force. There are a number of datasets studying the use of force by police and broader questions about the relationship between police activity, crime and race.
  • Mapping Police Violence
  • Police Violence and Racial Equity, Kaggle
  • Police Data Initiative

  • Data.World

  • Bot or Human. Can online marketplaces distinguish bots from humans when they make offers on a product? This auction dataset consists of three files: "bids.csv" contains information about each bid, such as the bid id, the id of the person who made the offer, auction type, type of device the bid was made from, (a random hash of) the buyer's ip address, etc; Interesting questions include:
    • Where are the bot "hot spots"? Do certain auction types, or IPs, or countries, have more bots?
    • How does the number of bots evolves over time? Could we predict this?
    • How many offers on average does each bot place?
    • What variables best predict being a bot?
  • Airbnb. This airbnb dataset consists of more than 50,000 Airbnb listings in New York City, categorized by neighborhood and housing type. The data set has longitude/latitude information, which makes for compelling visualizations. Interesting questions include:
    • How is the price of an Airbnb listing influenced by the neighborhood, housing type, or amenities? You could refer to the Global Power City Index (GPCI), for example, to analyze different neighborhoods. http://mori-m-foundation.or.jp/english/ius2/gpci2/index.shtml
    • Is there a relationship between minimum number of nights and number of reviews? The website offers archived data for the same city for different time periods as well.
    • How does the price change over time? Is the behaviour cyclic, or is there a trend?
  • UNHCR Refugee Migration. This data set records refugee migrations over 60 years, across hundreds of countries. Interesting questions include:
    • Are there any countries that are both large origin countries and asylum countries? Why might this be the case?
    • Do the countries with the most emigrants or immigrants change over time?
  • Youtube. Youtube calculates trending videos using a variety of factors. This dataset contains 200 trending videos across 205 days. Interesting questions include:
    • Can you forecast which videos will keep trending for several days?
    • What kind of tags occur frequently among trending videos?
    • Are there any similarities or differences across countries?
    • Are there channels that tend to produce many trending videos?
    • Do likes and dislikes affect views?
  • COVID. This dataset contains infections and hospitalizations by date, US state, and country. This is also interesting when merged with US Census data. Interesting questions include:
    • What impact have government interventions (social distancing, business closures, masking) had on infections and deaths?
    • What factors other than government interventions determine infections?
    • How similar are infection counts across different cities, states, and countries? What explains the similarities and differences?
    • What factors (e.g., demographic factors) determine the case fatality rate?
  • New York City Taxi Dataset contains taxi trips and trips on Uber / Lyft / other vehicle-for-hire platforms from 2009 to 2020. This can be combined with data from the Metropolitan Transit Authoroticy on how many people use the subway and bus system and subway delays. Interesting questions include:
    • How has the fraction of trips in NYC on taxi vs. other for-hire vehicles changed over time?
    • As this fraction has changed, how has the total amount of money generated on taxi changed over time? How has it changed in specific parts of the city?
    • How has traffic speed changed over time? You can estimate this by looking at the time to get from one point to another in a taxi.
    • When the subway is late, does that cause more people to take taxis / Uber / Lyft?