Earth Observation Using Python. Rebekah B. Esmaili

notable, its quality is equally impressive; for example, infrared sounders can estimate brightness temperatures within 0.1 K from surface observations (Tobin et al., 2013), imagers can detect ocean currents with an accuracy of 1.0 km/hr (NOAA, 2020), and satellite‐based lidar can measure the ice‐sheet elevation change with a 10 cm sensitivity (Garner, 2015). Previously remote parts of our planet are now observable, including the open oceans and sparsely populated areas. Furthermore, many datasets are available in near real time with image latencies ranging from less than an hour down to minutes – the latter being critically important for natural disaster prediction. Having data rapidly available enables science applications and weather prediction as well as to emergency management and disaster relief. Research‐grade data take longer to process (hours to months) but has a higher accuracy and precision, making it suitable for long‐term consistency. Thus, we live in the “golden age” of satellite Earth observation. While the data are accessible, the tools and skills necessary to display and analyze this information require practice and training.

Python is a modern programming language that has exploded in popularity, both within and beyond the Earth science community. Part of its appeal is its easy‐to‐learn syntax and the thousands of available libraries that can be synthesized with the core Python package to do nearly any computing task imaginable. Python is useful for reading Earth‐observing satellite datasets, which can be difficult to use due to the volume of information that results from the multitude of sensors, platforms, and spatio‐temporal spacing. Python facilitates reading a variety of self‐describing binary datasets in which these observations are often encoded. Using the same software, one can complete the entirety of a research project and produce plots. Within a notebook environment, a scientist can document and distribute the code to other users, which can improve efficiency and transparency within the Earth sciences community.

      Satellite data often require some pre‐processing to make it usable, but which steps to take and why are not always clear. Data users often misinterpret concepts such as data quality, how to perform an atmospheric correction, or how to implement the complex gridding schemes necessary to compare data at different resolutions. Even to a technical user, the nuances can be frustrating and difficult to overcome. This book walks you through some of the considerations a user should make when working with satellite data.

      The primary goal of this text is to get the reader up to speed on the Python coding techniques needed to perform research and analysis using satellite datasets. This is done by adopting an example‐driven approach. It is light on theory but will briefly cover relevant background in a nontechnical manner. Rather than getting lost in the weeds, this book purposefully uses realistic examples to explain concepts. I encourage you to run the interactive code alongside reading the text. In this chapter, I will discuss a few of the satellites, sensors, and datasets covered in this book and explain why Python is a great tool for visualizing the data.

1.1 History of Computational Scientific Visualization

Scientific data visualizing used to be a very tedious process. Prior to the 1970s, data points were plotted by hand using devices such as slide rules, French curls, and graph paper. During the 1970s, IBM mainframes became increasingly available at universities and facilitated data analysis on the computer. For analysis, IBM mainframes required that a researcher write Fortran‐IV code, which was then printed to cards using a keypunch machine (Figure 1.1). The punch cards then were manually fed into a shared university computer to perform calculations. Each card is roughly one line of code. To make plots, the researcher could create a Fortran program to make an ASCII plot, which creates a plot by combining lines, text, and symbols. The plot could then be printed to a line‐printer or a teleprinter. Some institutions had computerized graphic devices, such as Calcomp plotters. Rather than create ASCII plots, the researcher could use a Calcomp plotting command library to control how data were visualized and store the code on computer tape. The scientist would then take the tape to a plotter, which was not necessarily (or usually) in the same area as the computer or keypunch machine. Any errors – such as bugs in the code, damaged punch cards, or damaged tape – meant the whole process would have to be repeated from scratch.

Figure 1.1 (a) An example of a Fortran punch card. Each vertical column represents a character and one card roughly one line of Fortran code. (b) 1979 photo of an IMSAI 8080 computer that could store up to 32 kB of the data, which could then be transferred to a keypunch machine to create punch cards. (c) an image created from the Hubble Space Telescope using a Calcomp printer, which was made from running punch cards and plotting commands through a card reader.

In the mid‐1980s, universities provided remote terminals that would eventually replace the keypunch and card reader machine system. This substantially improved data visualization processes, as scientists no longer had to share limited resources such as keypunch machines, card readers, or terminals. By the late 1980s, personal computers became more affordable for scientists. A typical PC, such as the IBM XT 286, had 640 Kb of random access memory, a 32 MB hard drive, and 5.25 inch floppy disks with 1.2 MB of disk storage (IBM, 1989). At this time, pen plotters became increasingly common for scientific visualization, followed later by the prevalence of ink‐jet printers in the 1990s. These technologies allowed researchers to process and visualize data conveniently from their offices. With the proliferation of user‐friendly person computers, printers eventually made their way into all homes and offices.

      Now with advances in computing and internet access, researchers no longer need to print their visualizations at all, but often keep data in digital form only. Plots can be created in various data formats that easily embed into digital presentations and documents. Scientists often do not ever print visualizations because computers and cloud storage can store many gigabytes of data. Information is created and consumed entirely in digital form. Programming languages, such as Python, can tap into high‐level plotting programs and can minimize the axis calculation and labeling requirements within a plot. Thus, the expanded access to computing tools and simplified processes have advanced scientific data visualization opportunities.

1.2 Brief Catalog of Current Satellite Products

In Figure 1.2, you can see that the international community has developed and launched a plethora of Earth‐observing satellites, each with several onboard sensors that have a range of capabilities. I am not able to discuss every sensor, dataset, and mission (a term coined by NASA to describe projects involving spacecraft). However, I will describe some that are relevant to this text, organized by subject area.

Figure 1.2 Illustration of current Earth, space weather, and environmental monitoring satellites from the World Meteorological Organization (WMO). Source: U.S. Department of Commerce / NOAA / Public Domain.

      1.2.1 Meteorological and Atmospheric Science

      Most Earth‐observing satellites orbit our planet either in either geostationary or low‐Earth orbiting patterns. These types of satellites tend to be managed and operated by large international government agencies, and the data are often freely accessible online:

       Geosynchronous