I think it is not just me who would think that, as technology advances, the flight carriers are able to reduce the arrival delay over time. Is that really the case? I looked into the historic flights data and found something surprising and interesting.
First I downloaded all the historic flights data from stat-computing.org. These 22 years (1987 -2008) of data add up to be more than 10 GB. So I wrote a piece of R code which can read each csv file and extract the information I need for each carrier in each year. I tried many different ways to aggregate the data, for example, using the yearly average, 75% quantile, 99% quantile, and the yearly maximum. Since there are so many flight records for each carrier each year, I only found the yearly maximum arrival delays had some clear trends over time.
Surprisingly, this exploratory data analysis suggested that the yearly maximum of arrival delays increased rather than decreased in these 22 years. This is somehow counter-intuitive to me because I thought the Information Technology has developed so much and should have helped to reduce the flights delay. Anyway, I used D3.js and created an interactive scatter-line plot to show these trends. Below is a thumbnail of the plot, which is linked to the real html file where the plot is hosted. The legend is clickable so you can select which carriers’ data you would like to see or not see.
After making this plot and looking back into the data set, I realized it is reasonable that the yearly maximum arrival delays have increased in the past. The major reasons I can speculate includes:
- More and more people are taking airplanes to travel therefore there are a lot more flights to manage for each carrier.
- The number of longer distance flights increased and chances of longer arrival delay increased.
- The yearly maximum delay are probably caused by some extreme weather conditions or natural disasters, which seem to happen more frequently in recent years.
If you come up with other possible reasons, please leave it in the comments!