Preface#

Last updated: 2024-02-17 15:49:13

Welcome#

This book contains the materials of the 3-credit undergraduate course named Introduction to Spatial Data Programming with Python, given at the Department of Environmental, Geoinformatics and Urban Planning Sciences, Ben-Gurion University of the Negev, in Autumn 2023.

The structure of the book is as follows:

What is Python?#

Python (Fig. 1) is a general-purpose programming language. Python is used for a wide variety of purposes, such as:

_images/python_homepage.png

Fig. 1 Python website (https://www.python.org/)#

Python is open-source, has an intuitive syntax, and it is very popular. Among programming language questions on StackOverflow, Python currently (2023) stands at 1st place with ~14% of all questions (Fig. 2).

_images/stackoverflow_most_popular_languages.png

Fig. 2 Most popular programming languages, according to StackOverflow question proportions (https://insights.stackoverflow.com/trends)#

Python was initially released in 1991. The present version, which we learn in this book, is Python 3, released in 2008.

Why choose Python for spatial data?#

There are numerous reasons to choose Python for working with spatial data. These include general reasons for working through a Command Line Interface (CLI), as opposed to Graphical User Interfaces (GUI), that is, roughly speaking, writing code as opposed to clicking on menu buttons:

  • Programming facilitates automation and reproducibility of our workflows. When programming, we interact with the computer through scripts. Therefore, the workflows we create can be repeated, adapted for other use cases in the future, and shared with other people who would like to accurately reproduce your workflow.

  • Through programming, the user is “forced” to have a deeper understanding of the underlying data and the computational algorithms behind GIS workflows. Working through a CLI usually involves knowledge of lower-level details and requires us be specific about what we want to do.

There are also specific advantages of Python, over other CLI approaches:

  • Python is a widespread and extremely popular language, in the GIS as well as other industries, and in academic research (Fig. 2). For example, according to the 2021 StackOverflow survey, Python was the 3rd (after JavaScript and HTML) most popular programming technology, with 48.2% of respondents using it [1]. A recent FOSS4G (Free and Open Source Software for Geospatial) conference (FOSS4G 2021 Buenos Aires), Python was the major programming planguage in the Workshops, with three different 4-hour geospatial Python workshops (Fig. 3).

  • Python and the packages we are going to learn (see What are we going to learn?) are free and open-source, which means that you can setup the workflows we learn at any place and time, at zero cost.

  • The Python syntax was designed to be clear and straightforward. This means that Python has a (relatively) gentle learning curve, and that Python programs are often easy to read and understand, compared to other programming languages.

  • In addition to being a standalone tool, Python is also used to automate GIS (and other) software, such as ArcGIS/ArcPro (arcpy) and QGIS (PyQGIS) (see ArcGIS Pro scripting (arcpy)). Often, Python is the main or only CLI interface of GIS software. Google Earth Engine, a polular cloud computing environment for working with big spatial data has an official Python API.

  • Deep learning libraries, such as Keras/TensorFlow and PyTorch, are almost exclusively accessed through Python. Among other uses, deep learning is applicable to spatial analysis tasks such as object detection and image classification in remote sensing (Fig. 4).

_images/foss4g_workshop.png

Fig. 3 Doing Geospatial with Python workshop in the FOSS4G 2021 Buenos Aires conference#

_images/building_segmentation.png

Fig. 4 Building segmentation result, using deep learning with Python (source: https://medium.com/@anthropoco/how-to-segment-buildings-on-drone-imagery-with-fast-ai-cloud-native-geodata-tools-ae249612c321)#

Nevertheless, Python has disadvantages over other CLI approaches. For example, the R programming language can be considred as an alternative CLI tool for spatial analysis, with the following advantages over Python:

  • Python is a general-purpose language, which means that it is not natively designed to work with data. For example, the Python standard library does not support basic data science concepts and data structures, such as “No Data” values, arrays, and tables. This means we almost always need to rely on third-party packages such as numpy (see Arrays (numpy)) and pandas (see Tables (pandas)) when working with data. In R, the standard library covers all of those data-related concepts and much more.

  • Python’s spatial analysis “ecosystem” is more scattered, with numerous packages independently developed and not always inter-compatible. For example, although vector-based analysis is mostly unified into a single package called geopandas (see Vector layers (geopandas)), there are multiple packages for raster-based analysis, each with its own features, level of abstraction, advantages, and disadvantages, such as rasterio (see Rasters (rasterio) and Raster-vector interactions), xarray-spatial, rioxarray, earthpy, and geowombat. Consequently, vector-based and raster-based ecosystems are not well integrated. For example, basic operations such as zonal statistics may require using additional third party packages, such as a package called rasterstats (see Zonal statistics). In R there is much tighter integration between spatial analysis packages. For example, the pair of compatible R packages sf and stars cover most vector-based and raster-based analysis tasks, respectively.

What are we going to learn?#

In this book, we are going to work with the Python programming language, using packages from the standard library (which is built-in with the Python installation), as well as several third-party Python packages (which need to be installed separately, see Installing packages).

By the end of this book, you will be able to write Python programs to automate processing and analysis of spatial data. You will be able to write Python scripts for spatial analysis workflows consisting of operations such as:

  • Importing tables, vector layers, and rasters

  • Filtering and aggregating the data

  • Calculating new attributes, or reclassifying values to new categories

  • Making spatial calculations, such as calculating distances, or creating buffered layers

  • Creating simple plots and maps to inspect your data

  • Exporting the results to a table, vector layer, or raster file

You will also have a strong background in the fundamental packages for data science in Python, namely numpy and pandas. This is a good starting point for learning data-related topics other than spatial analysis, such as:

The most important third-party packages for spatial analysis in Python, which we are going to cover in detail, are listed in order of appearance in Table 1. The package versions being used when compiling the book are specified in System information.

Table 1 Main third-party Python packages used in this book#

Package

Functionality

Website

numpy

Arrays

https://numpy.org/

pandas

Tables

https://pandas.pydata.org/

shapely

Vector geometries

https://shapely.readthedocs.io/

geopandas

Vector layers

https://geopandas.org/

rasterio

Rasters

https://rasterio.readthedocs.io/

As we will see, these packages depend on one another. The major dependencies are depicted in Fig. 5.

_images/diagram_01_packages2.svg

Fig. 5 Main dependencies between the Python packages we are going to learn#

Additionally, we are going to use the packages listed in Table 2 for specific tasks. Packages from the standard library (which we’ll explain in What are packages?) are marked with (*) in the table.

Table 2 Other Python packages used in this book. Packages from the standard library are marked with (*).#

Package

Functionality

Website

csv (*)

Working with CSV files

https://docs.python.org/3/library/csv.html

math (*)

Mathematical functions

https://docs.python.org/3/library/math.html

matplotlib

Plots

https://matplotlib.org/

glob (*)

File search by pattern

https://docs.python.org/3/library/glob.html

rasterstats

Zonal statistics

https://pythonhosted.org/rasterstats/

richdem

Topographic raster calculations

https://richdem.readthedocs.io/en/latest/

scipy

Focal filtering

https://www.scipy.org/

Sample data#

Throughout the book, we are going to use several datasets for demonstrating the methods we learn. The data can be downloaded here:

Table 3 lists the datasets used in the book.

Table 3 Datasets used in the book#

Dataset

Filename

Format

Accessed

Source

Python script

test.py

Python

2022

-

“Requirements” file

requirements.txt

TXT

2023

-

World cities

world_cities.csv

CSV

2021

R package maps

Carmel DEM

carmel.csv

CSV

2016

SRTM data, from EarthExplorer

Carmel DEM (low resolution)

carmel_lowres.csv

CSV

2016

SRTM data, from EarthExplorer

GISS global temperature

ZonAnn.Ts+dSST.csv

CSV

2024

NASA

Trees in Beer-Sheva

trees.csv

CSV

2023

Beer-Sheva Municipality

GTFS

gtfs/*[2]

CSV

2023

Israel Ministry of Transport

BGU logo

bgu.wkt

WKT

2021

BGU

Railway stations

RAIL_STAT_ONOFF_MONTH.shp[3]

Shapefile

2020

Israel Ministry of Transport

Railway lines

RAIL_STRATEGIC.shp

Shapefile

2020

Israel Ministry of Transport

Israel municipalities

muni_il.shp

Shapefile

2022

Israel Ministry of Interior

Statistical areas demography 2019

statisticalareas_demography2019.gdb

Geodatabase

2019

Israel Central Bureau of Statistics

Beer-Sheva aerial photo (2015)

BSV_res200-M.tif

GeoTIFF

2021

MAPI

Sentinel2 image

T36RXV_20201226T082249_B0*.jp2

JPEG2000

2020

Sentinel2, from EarthExplorer

In some code examples in the book we are also going to create new files, to be used in later chapters or to demonstrate file export using Python. You can create them on your own, by running the code examples. Alternatively, you can download them from the following link:

Table 4 lists the files that we are going to create in the book.

Table 4 Files created in the code examples in the book#

Dataset

Filename

Format

Chapter

World cities

world_cities.shp

Shapefile

Setting up the environment

Packages

packages.csv

CSV

Writing CSV files

Railway stations

stations.csv

CSV

Tables (pandas)

Public transit routes

routes.shp

Shapefile

Vector layers (geopandas)

Public transit routes

routes.geojson

GeoJSON

Vector layers (geopandas)

Public transit routes

routes.gpkg

GeoPackage

Vector layers (geopandas)

Carmel DEM

carmel.tif

GeoTIFF

Rasters (rasterio)

Sentinel-2 stacked image

sentinel2.tif

GeoTIFF

Rasters (rasterio)

Carmel topographic aspect

carmel_aspect.tif

GeoTIFF

Rasters (rasterio)

System information#

Python version used when rendering the book is:

3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]

Package (Table 1, Table 2) versions used when rendering the book are:

notebook==7.0.6
numpy==1.26.3
pandas==2.2.0
shapely==2.0.2
geopandas==0.14.3
rasterio==1.3.9
matplotlib==3.5.1
rasterstats==0.19.0
richdem==0.3.4
scipy==1.12.0

Hostname:

dell14

Acknowledgements#

I thank the authors of the Python language, and the authors of the numpy, pandas, shapely, geopandas, and rasterio packages which are used extensively in the book, for building these wonderful tools.