Classification case: Assessing the performance of remote sensing models

Luciana Nieto & Adrian Correndo

2023-04-13

1. Introduction


The metrica package was developed to visualize and compute the level of agreement between observed ground-truth values and model-derived (e.g., mechanistic or empirical) predicted.

This package is intended to fit into the following workflow:

  1. a data set containing the observed values is used to train a model
  2. the trained model is used to generate predicted
  3. a data frame containing at least the observed and model-predicted values is created
  4. metrica package is used to compute and evaluate the classification model based on observed and predicted values
  5. metrica package is used to visualize model fit and selected fit metrics

This vignette introduces the functionality of the metrica package applied to observed and model-predicted values of a binary land cover classification scenario, where the two classes are vegetation (1) and non-vegetation (0)).

Let’s begin by loading the packages needed.
## Libraries

library(metrica)
library(dplyr)
library(purrr)
library(tidyr)

2. Example datasets

2.1. Kansas Land Cover data



Figure 1. This binary classification dataset corresponds to a Random Forest model using a 70:30 training:testing split to predict vegetation vs. other land coverage. This exercise focused on pixel level classification. The image is showing the classification map where yellow areas are associated to non-vegetation pixels (other), and the green areas to those classified as vegetation.

Now we load the binary land_cover data set already included with the metrica package. This data set contains two columns:

# Load
binary_landCover <- metrica::land_cover

# Printing first observations
head(binary_landCover)
#>   actual predicted
#> 1      0         0
#> 2      1         1
#> 3      1         1
#> 4      0         0
#> 5      0         0
#> 6      1         1

2.2. Maize Phenology



Figure 2. This multiclass classification dataset corresponds to a Random Forest model using a 70:30 training:testing split to predict maize vegetation vs. other land coverage. This exercise focused on field level classification. The image is showing, in dark grey shapes, the fields used as the ground-truth locations to develop the model.

Now we load the multinomial maize_phenology data set, which is also already included with the metrica package. This multiclass data set presents 16 different classes corresponding to phenological stages of the maize (Zea Mays (L.)) crop.

# Load
multi_maize_phen <- metrica::maize_phenology

# Printing first observations
head(multi_maize_phen)
#>   actual predicted
#> 1     R1       V18
#> 2     R2        R1
#> 3     R2        R1
#> 4     R2        R2
#> 5     R2        R2
#> 6     R2        R2

3 Visual assessment of agreement

3.1 Confusion matrix

The simplest way to visually assess agreement between observed and predicted classes is with a confusion matrix.

We can use the function confusion_matrix() from the metrica package.

The function requires specifying:

The output of the confusion_matrix() function is either a table (plot = FALSE) or a ggplot2 object (plot = TRUE) that can be further customized:

3.1. Binary

# a. Print
binary_landCover %>% confusion_matrix(obs = actual, pred = predicted, 
                                      plot = FALSE,
                                      unit = "count")
#>          OBSERVED
#> PREDICTED   0   1
#>         0 181   6
#>         1   6  92

# b. Plot
binary_landCover %>% confusion_matrix(obs = actual, pred = predicted, 
                                      plot = TRUE,
                                      colors = c(low="#ffe8d6" , high="#892b64"), 
                                      unit = "count")


# c. Unit = proportion
binary_landCover %>% confusion_matrix(obs = actual, pred = predicted, 
                                      plot = TRUE,
                                      colors = c(low="#f9dbbd" , high="#892b64"), 
                                      unit = "proportion")

3.2. Multiclass

# a. Print
multi_maize_phen %>% confusion_matrix(obs = actual, pred = predicted, 
                                      plot = FALSE, 
                                      unit = "count")
#>          OBSERVED
#> PREDICTED R1 R2 R3 R4 R5 R6 V10 V12 V13 V14 V15 V16 V18 V19 V9 VT
#>       R1   0  2  0  0  0  0   0   0   0   0   0   0   0   0  0  0
#>       R2   0  4  0  0  0  0   0   0   0   1   0   0   0   0  0  0
#>       R3   0  1  2  0  0  0   0   0   0   0   0   0   0   0  0  0
#>       R4   0  0  0  6  0  0   0   0   0   0   0   0   0   0  0  0
#>       R5   0  0  0  0 12  1   0   0   0   0   0   0   0   0  0  0
#>       R6   0  0  0  0  3 32   0   0   0   0   0   0   0   0  0  0
#>       V10  0  0  0  0  0  0   6   0   0   1   0   0   0   0  0  0
#>       V12  0  0  0  0  0  0   0   7   1   0   0   0   0   0  0  0
#>       V13  0  0  0  0  0  0   0   0   6   0   0   0   0   0  0  0
#>       V14  0  0  0  0  0  0   0   0   0   2   0   0   0   0  0  0
#>       V15  0  0  0  0  0  0   0   0   0   0   1   0   0   0  0  1
#>       V16  0  0  0  0  0  0   0   0   0   0   0   2   0   0  0  0
#>       V18  1  0  0  0  0  0   0   0   0   0   0   0   4   0  0  0
#>       V19  0  0  0  0  0  0   0   0   0   0   0   0   0   2  0  0
#>       V9   0  0  0  0  0  0   0   0   0   0   0   0   0   0  2  0
#>       VT   0  0  0  0  0  0   0   0   0   0   0   0   0   0  0  3

# b. Plot
multi_maize_phen %>% confusion_matrix(obs = actual, pred = predicted, 
                                      plot = TRUE, 
                                      colors = c(low="grey85" , high="steelblue"), 
                                      unit = "count")

4. Numerical assessment of agreement

The metrica package contains functions for 26 scoring rules to assess the agreement between observed and predicted values for classification data.

A list with all the the classification metrics including their name, definition, details, formula, and function name, please check here.

All of the metric functions take at least three arguments:

4.1. Single metrics

The user can choose to calculate a single metric, or to calculate all metrics at once.

To calculate a single metric, the metric function can be called. For example, to calculate \(accuracy\), we can use the accuracy() function:

# Binary
binary_landCover %>% accuracy(data = ., obs = actual, pred = predicted, tidy = TRUE)
#>    accuracy
#> 1 0.9578947

# Multiclass
maize_phenology %>% accuracy(data = ., obs = actual, pred = predicted, tidy = TRUE)
#>    accuracy
#> 1 0.8834951

Or considering imbalanced observations across classes we can call the balacc() function for balanced accuracy:

# Binary
binary_landCover %>% balacc(data = ., obs = actual, pred = predicted, tidy = TRUE)
#>     balacc
#> 1 0.953345

# Multiclass
maize_phenology %>% balacc(data = ., obs = actual, pred = predicted, tidy = TRUE)
#>      balacc
#> 1 0.9160466

Similarly, to calculate precision, we can use the precision() function:

# Binary
binary_landCover %>% precision(data = ., obs = actual, pred = predicted, tidy = TRUE)
#>   precision
#> 1 0.9387755

# Multiclass
maize_phenology %>% precision(data = ., obs = actual, pred = predicted, tidy = TRUE)
#>   precision
#> 1 0.8335108

4.2. Metrics summary

The user can also calculate all metrics at once using the function metrics_summary():


# Get all at once with metrics_summary()
# Binary
binary_landCover %>% metrics_summary(data = ., obs = actual, pred = predicted, type = "classification")
#>         Metric        Score
#> 1     accuracy   0.95789474
#> 2   error_rate   0.04210526
#> 3    precision   0.93877551
#> 4       recall   0.93877551
#> 5  specificity   0.96791444
#> 6       balacc   0.95334497
#> 7       fscore   0.93877551
#> 8          agf   0.93877551