---
title: "butterfly"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{butterfly}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

The goal of butterfly is to aid in the verification of continually updating timeseries data, where we expect new values over time, but want to ensure previous data remains unchanged, and timesteps remain continuous. 

Unnoticed changes in previous data could have unintended consequences, such as invalidating DOIs, or altering future predictions if used as input in forecasting models.

Other unnoticed changes could include a jump in time or measurement frequency, due to instrument failure or software updates.

This package provides functionality that can be used as part of a data pipeline, to check and flag changes to previous data to prevent changes going unnoticed.

You can provide butterfly with a timeseries dataset to check for continuity, with `timeline()`, and if it is not fully continuous as expected, split it into continuous chunks with `timeline_group()`. To check for changes to previous data, you can provide two versions of the *same* dataset, and `loupe()` will check if there are changes to matching rows, and tell you which rows are new. You can use `catch()` and `release()` to extract or remove rows with changes. Full examples of functionality are provided below.

## Data

This packages includes a small dummy dataset, `butterflycount`, which contains a list of monthly dataframes of butterfly counts for a given date. 

```{r butterfly_count}
library(butterfly)
butterflycount
```

This dataset is entirely fictional, and merely included to aid in demonstrating butterfly's functionality.

Another dummy dataset, `forestprecipitation`, also contains a list of monthly dataframes, but for fictional rainfall data. This dataset is intended to illustrate an instance of instrument failure leading to timesteps being recorded out of sync.

``` {r precipitation}
forestprecipitation
```

## Examining datasets: `loupe()`

We can use `butterfly::loupe()` to examine in detail whether previous values have changed.

```{r butterfly_example}
butterfly::loupe(
  butterflycount$february,
  butterflycount$january,
  datetime_variable = "time"
)

butterfly::loupe(
  butterflycount$march,
  butterflycount$february,
  datetime_variable = "time"
)
```

`butterfly::loupe()` uses `dplyr::semi_join()` to match the new and old objects using a common unique identifier, which in a timeseries will be the timestep. `waldo::compare()` is then used to compare these and provide a detailed report of the differences.

In addition to a report, `loupe()` also returns `TRUE` if there are no differences and `FALSE` when there are differences. This is especially useful when using butterfly in a pipeline that runs in a shell environment, a check for differences to fail gracefully.

`butterfly` follows the `waldo` philosophy of erring on the side of providing too much information, rather than too little. It will give a detailed feedback message on the status between two objects.

### Additional arguments from `waldo::compare()`

You have the flexibility to pass further arguments accepted by `waldo::compare()`to any of `loupe()`, `catch()` or `release()`. 

One such argument is `tolerance`. If we add a tolerance of 2 to the previous example, no differences should be returned:

```{r tolerance_example}
butterfly::loupe(
  butterflycount$march,
  butterflycount$february,
  datetime_variable = "time",
  tolerance = 2 # <- setting a tolerance of 2
)
```

See `?waldo::compare()` for the full list of arguments.

## Extracting unexpected changes: `catch()`

You might want to return changed rows as a dataframe, instead of returning `TRUE`/`FALSE`. For this `butterfly::catch()`is provided.

`butterfly::catch()` only returns rows which have **changed** from the previous version. It will not return new rows.

```{r butterfly_catch}
df_caught <- butterfly::catch(
  butterflycount$march,
  butterflycount$february,
  datetime_variable = "time"
)

df_caught
```

## Dropping unexpected changes: `release()`

Conversely, `butterfly::release()` drops all rows which have changed from the previous version. Note it retains new rows, as these were expected.

```{r butterfly_release}
df_released <- butterfly::release(
  butterflycount$march,
  butterflycount$february,
  datetime_variable = "time"
)

df_released
```

However, you do have the option to exclude new rows as well with the argument `include_new` set to `FALSE`.

```{r butterfly_release_no_new_rows}
df_release_without_new <- butterfly::release(
  butterflycount$march,
  butterflycount$february,
  datetime_variable = "time",
  include_new = FALSE
)

df_release_without_new

```

## Checking for continuity: `timeline()`
To check if a timeseries is continuous, `timeline()` and `timeline_group()` are
provided. Even if a timeseries does not contain obvious gaps, this does not
automatically mean it is also continuous.

Measuring instruments can have different behaviours when they fail. For
example, during power failure an internal clock could reset to "1970-01-01",
or the manufacturing date (e.g. "2021-01-01", this is common behaviour for Raspberry Pi's). This leads to unpredictable
ways of checking if a dataset is continuous.

```{r rain_gauge_data}
# A rain gauge which measures precipitation every day
butterfly::forestprecipitation$january

# In February there is a power failure in the instrument
butterfly::forestprecipitation$february
```

To check if a timeseries is continuous:

```{r check_continuity}
butterfly::timeline(
   forestprecipitation$january,
   datetime_variable = "time",
   expected_lag = 1
 )
```

As expected January is a continuous dataset, where there is no more than a difference
of 1 day between timesteps.

However, in February our imaginary rain gauge's onboard computer had a failure.

The timestamp was reset to `1970-01-01`:
 
```{r not_continuous}
forestprecipitation$february

butterfly::timeline(
  forestprecipitation$february,
   datetime_variable = "time",
   expected_lag = 1
 )
```

## Grouping distinct continuous sequences: `timeline_group()`

If we wanted to group chunks of our timeseries that are distinct, or broken up
in some way, but still continuous, we can use `timeline_group()`:

```{r timeline_group}
butterfly::timeline_group(
  forestprecipitation$february,
   datetime_variable = "time",
   expected_lag = 1
 )
```

We now have groups 1 & 2, which are both continuous sets of data, but there is
no continuity between them.

## Using `butterfly` in a data processing pipeline

If you would like to know more about using `butterfly` in an operational data processing pipeline, please refer to the article on  [using `butterfly` in an operational pipeline](https://docs.ropensci.org/butterfly/articles/butterfly_in_pipeline.html).

## A note on controlling verbosity

Although verbosity is mostly the purpose if this package, **should** you wish to silence messages and warnings, you can do so with `options(rlib_message_verbosity = "quiet")` and options `(rlib_warning_verbosity = "quiet")`.

## Rationale

There are a lot of other data comparison and QA/QC packages out there, why butterfly?

### Unexpected changes in models

This package was originally developed to deal with [ERA5](https://cds.climate.copernicus.eu/datasets/reanalysis-era5-single-levels?tab=documentation)'s initial release data, ERA5T. ERA5T data for a month is overwritten with the final ERA5 data two months after the month in question. 

Usually ERA5 and ERA5T are identical, but occasionally an issue with input data can (for example for [09/21 - 12/21](https://confluence.ecmwf.int/display/CKB/ERA5T+issue+in+snow+depth), and [07/24](https://forum.ecmwf.int/t/final-validated-era5-product-to-differ-from-era5t-in-july-2024/6685)) force a recalculation, meaning previously published data differs from the final product.

When publishing ERA5-derived datasets, and minting it with a DOI, it is possible to continuously append without invalidating that DOI. However, recalculation would overwrite previously published data, thereby forcing a new publication and DOI to be minted. 

We use the functionality in this package in an automated data processing pipeline to detect changes, stop data transfer and notify the user.

### Unexpected changes in data acquisition

Measuring instruments can have different behaviours when they have a power failure. For example, during power failure an internal clock could reset to "1970-01-01", or the manufacturing date (say, "2021-01-01"). If we are automatically ingesting and processing this data, it would be great to get a head's up that a timeseries is no longer continuous in the way we expect it to be. This could have consequences for any calculation happening downstream.

To prevent writing different ways of checking for this depending on the instrument, we wrote `butterfly::timeline()`.

### Variable measurement frequencies

In other cases, a non-continuous timeseries is intentional, for example when there is temporal variability in the measurements taken depending on events. At BAS, we collect data from a penguin weighbridge on Bird Island, South Georgia. This weighbridge measure weight on two different load cells (scales), one on the colony side and one on the ocean side, to determine penguin weight and the direction they came from. The idea is that we may be able to derive information on their diet, as they return from the ocean to the colony.

You can read about this work in more detail in [Afanasyev et al. (2015)](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0126292), but the important point here is that the weighbridge does not collect continuous measurements. In a remote, off-grid location this would drain batteries far too quickly.

Therefore, when no weight is detected on the load cells it only samples at 1hz, but as soon as any change in weight is detected it will start collecting data at 100hz. This is of course intentional, to reduce the sheer volume of data we need to process, but also has another benefit in isolating (or attempting to) individual crossings.

The individual crossings are the most valuables pieces of data, as these allow us to deduce information on weight, direction and ultimately, diet.

In this case separating distinct, but continuous segments of data is required. This is the reasoning behind `timeline_group()`. This function allows us to split our timeseries in groups of individual crossings.

### In summary 

This package has intentionally been generalised to accommodate other, but similar, use cases. Other examples could include a correction in instrument calibration, compromised data transfer or unnoticed changes in the parameterisation of a model.