Preface

Last updated: 2024-05-20 11:46:01

---

Welcome

This book contains the materials of the 3-credit undergraduate course named Introduction to Spatial Data Programming with Python, given at the Department of Environmental, Geoinformatics and Urban Planning Sciences, Ben-Gurion University of the Negev, in Autumn 2023.

The structure of the book is as follows:

• This section (see Preface) covers the context (why learn Python for working with spatial data?) and technical details about the material (which packages and sample data are we going to use?)

• INTRODUCTION TO PYTHON (see Setting up the environment, Python basics, Conditionals and loops) explains how to set up and use the Python working environment, and introduces the basics of the Python language, which are prerequisites for the later chapters

• WORKING WITH DATA (see Arrays (numpy), Tables (pandas), Table reshaping and joins) covers the numpy and pandas packages for working with array and tabular data in Python, respectively

• VECTOR LAYERS (see Geometries (shapely), Vector layers (geopandas), Geometric operations) covers the shapely and geopandas packages for working with vector layers in Python

• RASTERS (see Rasters (rasterio) and Raster-vector interactions) covers the rasterio package for working with rasters in Python

• ARCPRO SCRIPTING (see ArcGIS Pro scripting (arcpy)) introduces the arcpy package for writing Python scripts to automate the ArcPro GIS software

What is Python?

Python (Fig. 1) is a general-purpose programming language. Python is used for a wide variety of purposes, such as:

• Web servers and web applications (e.g., https://en.wikipedia.org/wiki/Django_(web_framework))

• Scientific computing and data science (e.g., https://en.wikipedia.org/wiki/NumPy)

• Machine learning and AI (e.g., https://en.wikipedia.org/wiki/TensorFlow)

• Scripting language in other software (e.g., https://en.wikipedia.org/wiki/ArcGIS_Pro)

Fig. 1 Python website (https://www.python.org/)

Python is open-source, has an intuitive syntax, and it is very popular. Among programming language questions on StackOverflow, Python currently (2023) stands at 1^st place with ~14% of all questions (Fig. 2).

Fig. 2 Most popular programming languages, according to StackOverflow question proportions (https://insights.stackoverflow.com/trends)

Python was initially released in 1991. The present version, which we learn in this book, is Python 3, released in 2008.

Why choose Python for spatial data?

There are numerous reasons to choose Python for working with spatial data. These include general reasons for working through a Command Line Interface (CLI), as opposed to Graphical User Interfaces (GUI), that is, roughly speaking, writing code as opposed to clicking on menu buttons:

• Programming facilitates automation and reproducibility of our workflows. When programming, we interact with the computer through scripts. Therefore, the workflows we create can be repeated, adapted for other use cases in the future, and shared with other people who would like to accurately reproduce your workflow.

• Through programming, the user is “forced” to have a deeper understanding of the underlying data and the computational algorithms behind GIS workflows. Working through a CLI usually involves knowledge of lower-level details and requires us be specific about what we want to do.

There are also specific advantages of Python, over other CLI approaches:

• Python is a widespread and extremely popular language, in the GIS as well as other industries, and in academic research (Fig. 2). For example, according to the 2021 StackOverflow survey, Python was the 3^rd (after JavaScript and HTML) most popular programming technology, with 48.2% of respondents using it [1]. A recent FOSS4G (Free and Open Source Software for Geospatial) conference (FOSS4G 2021 Buenos Aires), Python was the major programming planguage in the Workshops, with three different 4-hour geospatial Python workshops (Fig. 3).

• Python and the packages we are going to learn (see What are we going to learn?) are free and open-source, which means that you can setup the workflows we learn at any place and time, at zero cost.

• The Python syntax was designed to be clear and straightforward. This means that Python has a (relatively) gentle learning curve, and that Python programs are often easy to read and understand, compared to other programming languages.

• In addition to being a standalone tool, Python is also used to automate GIS (and other) software, such as ArcGIS/ArcPro (arcpy) and QGIS (PyQGIS) (see ArcGIS Pro scripting (arcpy)). Often, Python is the main or only CLI interface of GIS software. Google Earth Engine, a polular cloud computing environment for working with big spatial data has an official Python API.

• Deep learning libraries, such as Keras/TensorFlow and PyTorch, are almost exclusively accessed through Python. Among other uses, deep learning is applicable to spatial analysis tasks such as object detection and image classification in remote sensing (Fig. 4).

Fig. 3 Doing Geospatial with Python workshop in the FOSS4G 2021 Buenos Aires conference

Fig. 4 Building segmentation result, using deep learning with Python (source: https://medium.com/@anthropoco/how-to-segment-buildings-on-drone-imagery-with-fast-ai-cloud-native-geodata-tools-ae249612c321)

Nevertheless, Python has disadvantages over other CLI approaches. For example, the R programming language can be considred as an alternative CLI tool for spatial analysis, with the following advantages over Python:

• Python is a general-purpose language, which means that it is not natively designed to work with data. For example, the Python standard library does not support basic data science concepts and data structures, such as “No Data” values, arrays, and tables. This means we almost always need to rely on third-party packages such as numpy (see Arrays (numpy)) and pandas (see Tables (pandas)) when working with data. In R, the standard library covers all of those data-related concepts and much more.

• Python’s spatial analysis “ecosystem” is more scattered, with numerous packages independently developed and not always inter-compatible. For example, although vector-based analysis is mostly unified into a single package called geopandas (see Vector layers (geopandas)), there are multiple packages for raster-based analysis, each with its own features, level of abstraction, advantages, and disadvantages, such as rasterio (see Rasters (rasterio) and Raster-vector interactions), xarray-spatial, rioxarray, earthpy, and geowombat. Consequently, vector-based and raster-based ecosystems are not well integrated. For example, basic operations such as zonal statistics may require using additional third party packages, such as a package called rasterstats (see Zonal statistics). In R there is much tighter integration between spatial analysis packages. For example, the pair of compatible R packages sf and stars cover most vector-based and raster-based analysis tasks, respectively.

What are we going to learn?

In this book, we are going to work with the Python programming language, using packages from the standard library (which is built-in with the Python installation), as well as several third-party Python packages (which need to be installed separately, see Installing packages).

By the end of this book, you will be able to write Python programs to automate processing and analysis of spatial data. You will be able to write Python scripts for spatial analysis workflows consisting of operations such as:

• Importing tables, vector layers, and rasters

• Filtering and aggregating the data

• Calculating new attributes, or reclassifying values to new categories

• Making spatial calculations, such as calculating distances, or creating buffered layers

• Creating simple plots and maps to inspect your data

• Exporting the results to a table, vector layer, or raster file

You will also have a strong background in the fundamental packages for data science in Python, namely numpy and pandas. This is a good starting point for learning data-related topics other than spatial analysis, such as:

• Data processing

• Statistical analysis

• Machine learning

• Image processing

The most important third-party packages for spatial analysis in Python, which we are going to cover in detail, are listed in order of appearance in Table 1. The package versions being used when compiling the book are specified in System information.

Table 1 Main third-party Python packages used in this book

Package	Functionality	Website
numpy	Arrays	https://numpy.org/
pandas	Tables	https://pandas.pydata.org/
shapely	Vector geometries	https://shapely.readthedocs.io/
geopandas	Vector layers	https://geopandas.org/
rasterio	Rasters	https://rasterio.readthedocs.io/

As we will see, these packages depend on one another. The major dependencies are depicted in Fig. 5.

Fig. 5 Main dependencies between the Python packages we are going to learn

Additionally, we are going to use the packages listed in Table 2 for specific tasks. Packages from the standard library (which we’ll explain in What are packages?) are marked with (*) in the table.

Table 2 Other Python packages used in this book. Packages from the standard library are marked with (*).

Package	Functionality	Website
csv (*)	Working with CSV files	https://docs.python.org/3/library/csv.html
math (*)	Mathematical functions	https://docs.python.org/3/library/math.html
matplotlib	Plots	https://matplotlib.org/
glob (*)	File search by pattern	https://docs.python.org/3/library/glob.html
rasterstats	Zonal statistics	https://pythonhosted.org/rasterstats/
richdem	Topographic raster calculations	https://richdem.readthedocs.io/en/latest/
scipy	Focal filtering	https://www.scipy.org/

Prerequisites

The book can be used as primary text for an introductory course on programmatic approach to spatial data analysis in geography departments, or by anyone who is interested in the topic. Each of the chapters is designed to be covered in a three-hour lecture, or through self-study. Short questions and exercises are given throughout the chapters to demonstrate the material from different angles and facilitate understanding.

Familiarity with basic concepts of geographic data and GIS (coordinate systems, projections, spatial layer file formats, etc.) is necessary for deeper understanding of some of the topics in the book. Readers who are not familiar with GIS can skip the theoretic considerations and still follow the material from the technical point of view.

The book assumes no background knowledge in programming, going through all necessary material from the beginning.

Sample data

Throughout the book, we are going to use several datasets for demonstrating the methods we learn. The data can be downloaded here:

• data.zip

Table 3 lists the datasets used in the book.

Table 3 Datasets used in the book

Dataset	Filename	Format	Accessed	Source
Python script	test.py	Python	2022	-
“Requirements” file	requirements.txt	TXT	2023	-
World cities	world_cities.csv	CSV	2021	R package maps
Carmel DEM	carmel.csv	CSV	2016	SRTM data, from EarthExplorer
Carmel DEM (low resolution)	carmel_lowres.csv	CSV	2016	SRTM data, from EarthExplorer
GISS global temperature	ZonAnn.Ts+dSST.csv	CSV	2024	NASA
Trees in Beer-Sheva	trees.csv	CSV	2023	Beer-Sheva Municipality
GTFS	gtfs/*[2]	CSV	2023	Israel Ministry of Transport
BGU logo	bgu.wkt	WKT	2021	BGU
Railway stations	RAIL_STAT_ONOFF_MONTH.shp[3]	Shapefile	2020	Israel Ministry of Transport
Railway lines	RAIL_STRATEGIC.shp	Shapefile	2020	Israel Ministry of Transport
Israel municipalities	muni_il.shp	Shapefile	2022	Israel Ministry of Interior
Statistical areas demography 2019	statisticalareas_demography2019.gdb	Geodatabase	2019	Israel Central Bureau of Statistics
Beer-Sheva aerial photo (2015)	BSV_res200-M.tif	GeoTIFF	2021	MAPI
Sentinel2 image	T36RXV_20201226T082249_B0*.jp2	JPEG2000	2020	Sentinel2, from EarthExplorer

In some code examples in the book we are also going to create new files, to be used in later chapters or to demonstrate file export using Python. You can create them on your own, by running the code examples. Alternatively, you can download them from the following link:

• output.zip

Table 4 lists the files that we are going to create in the book.

Table 4 Files created in the code examples in the book

Dataset	Filename	Format	Chapter
World cities	world_cities.shp	Shapefile	Setting up the environment
Packages	packages.csv	CSV	Writing CSV files
Railway stations	stations.csv	CSV	Tables (pandas)
Public transit routes	routes.shp	Shapefile	Vector layers (geopandas)
Public transit routes	routes.geojson	GeoJSON	Vector layers (geopandas)
Public transit routes	routes.gpkg	GeoPackage	Vector layers (geopandas)
Carmel DEM	carmel.tif	GeoTIFF	Rasters (rasterio)
Sentinel-2 stacked image	sentinel2.tif	GeoTIFF	Rasters (rasterio)
Carmel topographic aspect	carmel_aspect.tif	GeoTIFF	Rasters (rasterio)

System information

Python version used when rendering the book is:

3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]

Package (Table 1, Table 2) versions used when rendering the book are:

notebook==7.0.7
numpy==1.26.4
pandas==2.2.1
shapely==2.0.3
geopandas==1.0.0a1
rasterio==1.3.9
matplotlib==3.5.1
rasterstats==0.19.0
richdem==0.3.4
scipy==1.12.0

Hostname:

michael-desktop

Acknowledgements

I thank the authors of the Python language, and the authors of the numpy, pandas, shapely, geopandas, and rasterio packages which are used extensively in the book, for building these wonderful tools.

Recommended materials online

Overview

This section contains links to recommended online resources which are relevant to the scope of this book. All of the listed resources are freely available online.

Official tutorials

The official tutorials of the third-party packages are often the best place to start. Go to the home pages of the main packages we are going to learn (Table 1) to find the official tutorials of those packages.

Courses

Courses on general Python:

• Practical Data Science (Duke University)

• Geo-Python (University of Helsinki)

• Python Online (King’s College London)

• Programming in Python for Data Science (The University of British Columbia)

• Python Programming for Data Science (University of British Columbia)

Courses on geospatial Python:

• Automating GIS-processes (University of Helsinki)

• Geospatial Analysis with Python (University of Tartu)

• PyGIS - Open Source Geospatial Programming & Remote Sensing (The George Washington University)

• Geographic Data Science (University of Liverpool)

• Geographic Data Science for Applied Economists

• Python for GIS and Geoscience (Ghent University)

• Spatial data science for sustainable development (Aalto University)

• Geospatial Analysis and Representation for Data Science (University of Trento)

• Foundations of Spatial Data Science (University College London)

• Geospatial Data Science (University of Copenhagen)

• Python for Geospatial Analysis (University of British Columbia)

Tutorials

Tutorials on general Python:

• Python 3 tutorial (WikiBooks, in Hebrew)

• Python in a hurry

• Plotting and Programming in Python (Data Carpentry)

• Programming with Python (Data Carpentry)

• Data Analysis and Visualization with Python for Social Scientists (Data Carpentry)

• Data Analysis and Visualization in Python for Ecologists (Data Carpentry)

• Python for Atmosphere and Ocean Scientists (Data Carpentry)

• Calm Code tutorials

Tutorials on geospatial Python:

• Mapping and Data Visualization with Python (SpatialThoughts)

• Python Foundation for Spatial Analysis (SpatialThoughts)

• Introduction to Geospatial Raster and Vector Data with Python (Data Carpentry)

• Get Started With GIS in Open Source Python

• Python for Geosciences (post series)

• Geospatial Analysis Tutorials (Kaggle)

• Earth Data Science Tutorials in Python

• Pythia Foundations - A community learning resource for Python-based computing in the geosciences

Python tutorials on related topics:

• Image Processing with Python

Books

Books on general Python:

• Think Python (2nd Ed.)

• A Byte of Python

Books on working with data in Python:

• Python for Data Analysis

• Coding for Economists

Books on geospatial Python:

• Geocomputation with Python (in preparation)

• An Introduction to Earth and Environmental Data Science

• Introduction to Python for Geographic Data Analysis (in preparation)

• Geographic Data Science with PySAL and the PyData Stack

• Spatial Data Science (Python version)

---

[1]

https://insights.stackoverflow.com/survey/2021#most-popular-technologies-language

[2]

The GTFS dataset is composed of several .txt files (located in the gtfs folder), namely: agency.txt, calendar.txt, fare_attributes.txt, fare_rules.txt, routes.txt, shapes.txt, stops.txt, stop_times.txt, translations.txt, trips.txt.

[3]

By convention, Shapefile datasets are listed as .shp files. However, a Shapefile is actually composed of at least two more files (.shx, .dbf), and usually more, sharing the same prefix.