---
title: "Why are the results not what I expect?"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Why are the results not what I expect?}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set (
    collapse = TRUE,
    comment = "#>"
)
```

This vignette presumes that you've read the "_How does pkgmatch work?_"
vignette, which initially explains that,

> The "pkgmatch" package package finds packages, as well as individual
> functions, which best match a given input. Inputs can be text descriptions,
> sections of code, or even entire R packages. "pkgmatch" finds the most
> closely matching packages using a combination of Language Models (LMs, or
> equivalently, "LLMs" for large language models), and traditional
> [token-frequency algorithms](https://en.wikipedia.org/wiki/Okapi_BM25).

This vignette digs more deeply into the question of why "pkgmatch" may
sometimes fail to produce expected results. In answering that question, it is
important to understand that LMs effectively rely on compressed representations
of input data. That compression is in the form of vectors of "embeddings" which
transform textual input to vectors of numeric values. The vectors here, and in
many LM-based systems, comprise 768 individual values. No matter how long an
input is, it will always be represented in the embedding space by a vector of
768 numeric elements. This representation is thus inherently "lossy", and
therefore inherently approximate. "pkgmatch" works by matching the embedding
vectors of any input to pre-computed data sets of embedding vectors from the
specific corpora. Because all embeddings are approximate, matching is also
unavoidably approximate. Any expected match may thus not necessarily appear as
the best-matched result from {pkgmatch}. Nevertheless, to the extent that the
approximations are accurate, expected matches should appear somewhere within
the first few matches. This vignette explores why even that approximate
expectation may sometimes fail to happen.

Note that all results in this vignette are pre-generated and hard-coded,
because they rely on complex and time-consuming outputs of locally-running
language models. While all code within the entire vignette may be run directly
as is, results may differ slightly from those shown here.

## Demonstration of unexpected results

To start, we need to load the package:

```{r library}
library (pkgmatch)
```

The entire vignette is based on results from a single prompt, for which the
expected result is [the `lubridate` package](https://lubridate.tidyverse.org/):

```{r}
input <- "Package that works with dates and times in tidy format"
```

By default, the object returned from [the `pkmgatch_similar_pkgs()`
function()](https://docs.ropensci.org/pkgmatch/reference/pkgmatch_similar_pkgs.html)
prints the top five matching packages:

```{r sim-pkgs1, eval = FALSE}
pkgmatch_similar_pkgs (input, corpus = "cran")
```
```{r sim-pkgs1-out, echo = FALSE}
c ("datefixR", "fpp3", "rebus.datetimes", "sweep", "datetime")
```

And while expected results may not necessarily always be in the first position,
the absence of [`lubridate`](https://lubridate.tidyverse.org/) from the top
five matches is indeed unexpected. Understanding why this arises requires
diving into the code used by {pkgmatch} to obtain those matches. The following
sub-sections re-generate full packages embeddings for the five packages listed
above plus the expected result of `lubridate`, in order to compare results for
that expected package with those of the five others.

### Obtain package source

The following code downloads and extracts tarballs of source code for the five
packages listed above, plus `lubridate`.

```{r dl-pkgs}
pkgs <- c (
    "datefixR", "fpp3", "rebus.datetimes", "sweep", "datetime", "lubridate"
)
path <- utils::download.packages (
    pkgs,
    destdir = fs::path_temp (),
    repos = "https://cloud.r-project.org"
)
chk <- lapply (path [, 2], function (p) {
    utils::untar (p, exdir = fs::path_temp (), tar = "internal")
})
pkg_paths <- fs::path (fs::path_temp (), pkgs)
stopifnot (all (fs::dir_exists (pkg_paths)))
```

### Embeddings from package source

Embeddings used in {pkgmatch} are generated from [the
`pkgmatch_embeddings_from_pkgs()`
function](https://docs.ropensci.org/pkgmatch/reference/pkgmatch_embeddings_from_pkgs.html).
This function accepts one main input parameter specifying paths to one or more
local directories containing source code.

```{r embeddings, eval = FALSE}
emb <- pkgmatch_embeddings_from_pkgs (pkg_paths)
str (emb)
```

```{r embeddings-output, echo = FALSE}
message ("Generating text embeddings [1 / 2] ...")
message ("Generating text embeddings [2 / 2] ...")
message ("Generating code embeddings ...")

text_with_fns <- array (runif (768 * 6, -1, 1), dim = c (768, 6))
text_wo_fns <- array (runif (768 * 6, -1, 1), dim = c (768, 6))
code <- array (runif (768 * 6, -1, 1), dim = c (768, 6))
colnames (text_with_fns) <- colnames (text_wo_fns) <- colnames (code) <- pkgs
emb <- list (
    text_with_fns = text_with_fns,
    text_wo_fns = text_wo_fns,
    code = code
)
str (emb)
```

Those three list items in the returned object are embeddings from the full
package text including text of all function documentation entries
(`text_with_fns`); the equivalent text without function documentation entries
(`text_wo_fns`), and the entire package source code represented as a single
character vector (`code`).

### Generating matches from those embeddings

The embedding vector corresponding to the input is then compared to these
embedding vectors in order to find the best-matched package. Similarities
between embeddings are measured here, as in the vast majority of LM
applications, using [cosine
similarity](https://en.wikipedia.org/wiki/Cosine_similarity).

```{r similarities, eval = FALSE}
emb_input <- get_embeddings (input, code = FALSE)
simil_with_fns <- cosine_similarity (emb_input, emb$text_with_fns)
simil_wo_fns <- cosine_similarity (emb_input, emb$text_wo_fns)
names (simil_with_fns) [2] <- "simil_with_fns"
names (simil_wo_fns) [2] <- "simil_wo_fns"
similarities <- dplyr::left_join (simil_with_fns, simil_wo_fns, by = "package")
print (similarities)
```

```{r similarities-output, echo = FALSE}
p <- c ("rebus.datetimes", "datetime", "datefixR", "lubridate", "sweep", "fpp3")
wi <- c (0.8304852, 0.8202274, 0.8194707, 0.7905832, 0.7714603, 0.7071283)
wo <- c (0.8186134, 0.8351010, 0.8186624, 0.7940536, 0.7984862, 0.8093774)
similarities <- data.frame (package = p, simil_with_fns = wi, simil_wo_fns = wo)
print (similarities)
```


That generates two vectors of similarities, with values for each package
generated from embeddings calculated both with and without text from function
documentation. Matching in "pkgmatch" also uses additional data from ["BM25"
values](https://en.wikipedia.org/wiki/Okapi_BM25) for inverse word frequencies.
These are derived from calculating word frequencies across the entire corpora,
weighting each word in an input by the inverse of these frequencies, and then
matching the result to word frequency vectors for each target package. The
following lines load the pre-computed "inverse document frequencies" for the
corpus, and then use those to calculate BM25 scores for the input. Since the
"idfs" include data for the entire CRAN corpus, this code also filters final
scores down to our selection of six packages only, noting that the "package"
column contains full names of tarballs.

```{r idf-bm25, eval = FALSE}
idfs <- pkgmatch_load_data (what = "idfs", corpus = "cran")
bm25 <- pkgmatch_bm25 (input = input, idfs = idfs, corpus = "cran") |>
    dplyr::mutate (package = gsub ("\\_.*$", "", package)) |>
    dplyr::filter (package %in% pkgs)
similarities <- dplyr::left_join (similarities, bm25, by = "package")
print (similarities)
```
```{r idf-bm25-output, echo = FALSE}
similarities$bm25_with_fns <-
    c (12.7782429, 12.6079534, 13.5863247, 0.2140736, 15.6956661, 12.2372817)
similarities$bm25_wo_fns <-
    c (13.1704148, 12.6606973, 16.3750351, 0.6645225, 15.0961380, 14.8849026)
print (similarities)
```

Finally, these similarities are combined using a ["reranking"
function](https://www.anthropic.com/news/contextual-retrieval). For "pkgmatch",
this function by default takes only similarities excluding function
definitions, so only two of the four columns shown above, and combines them
using an `lm_proportion` parameter with a default value of 0.5 for equal
contributions of similarities from embeddings and BM25 values.

```{r rank-output, eval = FALSE}
pkgmatch_rerank (similarities)
```
```{r rank-output-output, echo = FALSE}
data.frame (
    package = c (
        "datefixR", "datetime", "sweep", "rebus.datetimes", "fpp3",
        "lubridate"
    ),
    rank = 1:6
)
```

And as expected, `lubridate` is the lowest-ranked of the six packages. Note
also the the BM25 scores for `lubridate` are much lower than for any of the
other packages. This indicates that the actual words used in the input prompt
are only a poor match for the actual words used within the text of the
`lubridate` package, whereas they are a much better match for the other five
top-matched packages. Nevertheless, `lubridate` is also the worst-matched
package in terms of similarities without function documentation text. Thus,
regardless of how LM and BM25 values are combined, `lubridate` is never
included in the top five matches.

While BM25 values are derived from empirical word frequencies, and so are
effectively fixed, the LM similarity metrics are derived from embeddings, and
as stated at the output, these are inherently approximate. The next section
describes how {pkgmatch} might be refined to improve the accuracy of these
approximate representations.

---

## Modifying embeddings through chunking

One common strategy to improve the accuracy of LM embeddings is "chunking",
which refers to taking different "chunks" of an input that generally extends
beyond the admissible "context window" of the model. The models used here have
context windows of 8096 tokens, which is often enough to enter the entire code
or text of a package. Chunking effects can nevertheless be achieved through
permuting components of the input in different orders. The embedding for "This
plus that. Then those." will differ to that for "Then those. This plus
that.":

```{r ex-embeddings, eval = FALSE}
head (get_embeddings ("This plus that. Then those."))
head (get_embeddings ("Then those. This plus that."))
```
```{r ex-embeddings-out, echo = FALSE}
matrix (
    c (-0.5418470, 0.1983769, 1.1284385, 0.2710975, 0.1558345, 0.0710893),
    ncol = 1L
)


matrix (
    c (-0.57014912, 0.22628722, 1.09075117, 0.24393138, 0.11764651, 0.09092502),
    ncol = 1L
)
```

Those two sets of embeddings are quite similar, and yet not identical. Chunking
is generally used to generate several embeddings for any given input, with
similarity metrics averaged over these several embeddings then providing a more
accurate approximation to the underlying textual similarities. In other words,
chunking reduces the noise inherent in approximate representation through
embeddings.

We will now demonstrate how chunking can be applied within the general
"pkgmatch" workflow shown above. The following code modifies the procedure used
within the `get_embeddings()` function, to create differently permuted chunks
of the package input. (This requires calling two non-exported functions using
the "three-dots" notation, `:::`, in contrast to the standard "two dots" for
exported functions.)

```{r}
txt_with_fns <- vapply (pkg_paths, function (p) pkgmatch:::get_pkg_text (p), character (1L))
txt_wo_fns <- pkgmatch:::rm_fns_from_pkg_txt (txt_with_fns)
```

The `get_pkg_text()` function inserts markdown-formatted section headers. These
can then be used to break the text into sections which can then be randomly
rearranged to form new chunks. The following code generates embeddings from
differently-ordered chunks (excluding text from function documentation).

```{r, eval = FALSE}
n_permutations <- 5L
permute_text <- function (text_input) {
    txt <- strsplit (text_input, "#+") [[1]]
    index <- order (runif (length (txt)))
    paste0 (txt [index], collapse = "\\n")
}

embeddings <- lapply (pkg_paths, function (p) {
    txt_with_fns <- pkgmatch:::get_pkg_text (p)
    txt_wo_fns <- pkgmatch:::rm_fns_from_pkg_txt (txt_with_fns) [[1]]

    do.call (cbind, lapply (
        seq_len (n_permutations),
        function (j) { get_embeddings (permute_text (txt_wo_fns)) }
    ))
})
```

Calculate similarities with those embeddings, and generate average similarities
across the chunks for each package:

```{r simil-with-chunks, eval = FALSE}
input_emb <- get_embeddings (input) [, 1]
similarities <- vapply (embeddings, function (emb) {
    colnames (emb) <- letters [seq_len (ncol (emb))]
    simil <- cosine_similarity (input_emb, emb, fns = FALSE)
    mean (simil$simil)
}, numeric (1L))
similarities <- data.frame (package = pkgs, similarity = similarities) |>
    dplyr::arrange (dplyr::desc (similarity))
print (similarities)
```
```{r similar-with-chunks-output, echo = FALSE}
data.frame (
    package = c (
        "datetime", "rebus.datetimes", "datefixR", "lubridate",
        "sweep", "fpp3"
    ),
    similarity = c (
        0.8379673, 0.8235914, 0.8203636, 0.7936824, 0.7849108,
        0.7724631
    )
)
```

And averaging similarities across random chunks have increased the similarity
between the input text and the `lubridate` package from last place up to fourth
out of six. That is sufficient for `lubridate` to appear in the default list of
top five matches seen when printing the output of "pkgmatch" functions.

## Why not use chunking in `pkgmatch`?

Chunking like that demonstrated immediately above is very commonly used in many
LM applications, notably including by almost all commercial API providers. The
previous section also demonstrates that it can and does improve the results of
"pkgmatch". And yet it is not used here, for reason we now explain.

The current procedures generate three embedding vectors for every R package in
both the rOpenSci and CRAN corpora, with these embeddings updated every day.
The embeddings are stored with [a GithUb release of this
package](https://github.com/ropensci-review-tools/pkgmatch/releases/tag/v0.4.0).
Clicking on that link shows all files of pre-calculated data needed in this
package, and importantly shows their sizes as well. The data currently total
over 300MB, with the largest single file being the embedding vectors for all
CRAN packages, with a size of 227MB. Each permutation of package text and code
would require an additional embedding vector, so that using five random
permutations like the code above would increase the size of that file to well
over one gigabtye. Anybody using this package must wait for these data to be
downloaded. Downloading 227MB of data can already be very time consuming.
The additional burden of forcing users to wait to download over 1GB of data in
order to generate results would be a strong disincentive to using this package.

Equally importantly, these data are updated on a daily basis, and their
generation requires energy. Each additional permutation of input chunks
requires the energy used for current daily updates to be used again, with five
chunks requiring five times the current energy usage.

In short: chunking may be effective in increasing the accuracy of LM outputs,
but it also increases demands on memory, on data transfer, and on energy
consumption. To remain generally useable by as many people as possible, and to
reduce as far as possible the energy, bandwidth, and storage requirements of
this package, chunking is not used here.

## Recommendations for improving results

This entire vignette has shown how the numerical representations used within
LMs are approximate, and demonstrated the consequences of those approximations.
While chunking is an effective approach to _numerically_ overcoming some of the
negative effects of these approximations, it can be just as effective to
provide more accurately descriptive textual input, as illustrated by the
following code which uses `lm_proportion = 1` to display match results from
LM output only:

```{r longer-input, eval = FALSE}
input <- paste0 (
    "package that works with dates and times in a tidy format and ",
    "overcomes unintuitive frustrating quirks with daylight savings, ",
    "time zones, and leap years"
)
pkgmatch_similar_pkgs (input, corpus = "cran", lm_proportion = 1)
```

```{r long-input-out, echo = FALSE}
c ("timechange", "neatRanges", "fpp3", "datetime", "lubridate")
```

And `lubridate` appears in the top five matched packages after tweaking the
input to more accurately match some of the "quirks" of the text of that package
(which includes that word).
