--- title: "d1 - Installing R and Python packages in a Nix environment" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{d1-installing-r-packages-in-a-nix-environment} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>" ) ``` ```{r, include=FALSE} library(rix) ``` ## Introduction You now know how to declare and build reproducible development environments using `{rix}` and Nix. This vignette will explain how to install specific versions of CRAN packages and how to install packages from GitHub. ## A word of caution It is important at this stage to understand that you should not call `install.packages()` from a running Nix environment. Doing so will raise an error to avoid issues. If you want to add packages while analyzing data, you need to add it the `default.nix` expression and rebuild the environment. The same goes for installing packages from GitHub; use the method described in this vignette instead of using something like `remotes::install_github()`. We recommend you create a script called `create_env.R` or similar, and add the call to `rix()` there: ``` library(rix) rix(r_ver = "4.4.0", r_pkgs = c("dplyr", "ggplot2"), system_pkgs = NULL, git_pkgs = NULL, ide = "code", project_path = path_default_nix, overwrite = TRUE, print = TRUE) ``` Then, add the packages you need to `r_pkgs` and run the script again. Then, build the environment using `nix-build` again, and drop into it using `nix-shell`. Calling `install.packages()` is a bad idea for several reasons: - it goes against the idea of defining an environment in a declarative way. If you were able to add packages using `install.packages()`, your environment would end up in a state where the `default.nix` definition of the environment and the actual environment don't match anymore. - using `install.packages()` would likely simply not work, and if it would work, it would cause issues. For example, if you call `install.packages("ggplot2")` from one Nix shell, it will not install `{ggplot2}` "inside" the Nix shell, but will install it on your user's system library of packages. This is because the Nix shell cannot be changed at run-time, and so, R will instead install the packages in the user's library. This version of `{ggplot2}`, because it is in that system-wide library of packages, will be available to any other Nix shells. If you call `install.packages("ggplot2")` again from another Nix shell, say 6 months later, this will replace the first version of `{ggplot2}` with the latest version. Ideally, you should only manage R versions and R packages using Nix, and uninstall any system-managed version of R and R packages. But if you do wish to keep a system-managed version of R and R packages, `rix::rix()` also runs `rix::rix_init()` automatically which generates an `.Rprofile` file that avoids any clashes between your global user library and Nix-managed libraries of R packages. ## Installing old packages archived on CRAN It is possible to install an arbitrary version of a package that has been archived on CRAN: ```{r, eval = F} path_default_nix <- tempdir() rix( r_ver = "4.2.1", r_pkgs = c("dplyr@0.8.0", "janitor@1.0.0"), system_pkgs = NULL, git_pkgs = NULL, ide = "none", project_path = path_default_nix, overwrite = TRUE ) ``` ``` #> let #> pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/79b3d4bcae8c7007c9fd51c279a8a67acfa73a2a.tar.gz") {}; #> #> git_archive_pkgs = [ #> (pkgs.rPackages.buildRPackage { #> name = "dplyr"; #> src = pkgs.fetchzip { #> url = "https://cran.r-project.org/src/contrib/Archive/dplyr/dplyr_0.8.0.tar.gz"; #> sha256 = "sha256-f30raalLd9KoZKZSxeTN71PG6BczXRIiP6g7EZeH09U="; #> }; #> propagatedBuildInputs = builtins.attrValues { #> inherit (pkgs.rPackages) #> assertthat #> glue #> magrittr #> pkgconfig #> R6 #> Rcpp #> rlang #> tibble #> tidyselect #> BH #> plogr; #> }; #> }) #> #> (pkgs.rPackages.buildRPackage { #> name = "janitor"; #> src = pkgs.fetchzip { #> url = "https://cran.r-project.org/src/contrib/Archive/janitor/janitor_1.0.0.tar.gz"; #> sha256 = "sha256-3NJomE/CNbOZ+ohuVZJWe2n1RVGUm8x8a0A0qzLmdN4="; #> }; #> propagatedBuildInputs = builtins.attrValues { #> inherit (pkgs.rPackages) #> dplyr #> tidyr #> snakecase #> magrittr #> purrr #> rlang; #> }; #> }) #> ]; #> #> system_packages = builtins.attrValues { #> inherit (pkgs) #> R #> glibcLocales #> nix; #> }; #> #> in #> #> pkgs.mkShell { #> LOCALE_ARCHIVE = if pkgs.system == "x86_64-linux" then "${pkgs.glibcLocales}/lib/locale/locale-archive" else ""; #> LANG = "en_US.UTF-8"; #> LC_ALL = "en_US.UTF-8"; #> LC_TIME = "en_US.UTF-8"; #> LC_MONETARY = "en_US.UTF-8"; #> LC_PAPER = "en_US.UTF-8"; #> LC_MEASUREMENT = "en_US.UTF-8"; #> #> buildInputs = [ git_archive_pkgs system_packages ]; #> #> } ``` The above expression will install R version 4.2.1, and `{dplyr}` at version 0.8.0 and `{janitor}` at version 1.0.0. This can be useful, especially for packages that have been archived, but otherwise, this feature should ideally be used sparingly, for two reasons. First, if you want to reconstruct an environment as it was around 2019, use the version of R that was current at that time using the `date` argument in `rix()`. This will ensure that every package that gets installed is at a version compatible with that version of R, which might not be the case if you need to install a very old version of one particular package. Second, doing so will install the package from source. For packages that don't require compilation, this should be fine, but packages that require compilation will likely fail to compile successfully. We are working on handling this better for future versions of `{rix}`. ## Installing packages from GitHub It is also possible to install packages from GitHub: ```{r, eval = F} path_default_nix <- tempdir() rix( r_ver = "4.2.1", r_pkgs = c("dplyr", "janitor"), git_pkgs = list( list( package_name = "housing", repo_url = "https://github.com/rap4all/housing/", commit = "1c860959310b80e67c41f7bbdc3e84cef00df18e" ), list( package_name = "fusen", repo_url = "https://github.com/ThinkR-open/fusen", commit = "d617172447d2947efb20ad6a4463742b8a5d79dc" ) ), ide = "none", project_path = path_default_nix, overwrite = TRUE ) ``` ```{r parsermd-chunk-4, echo = F} #> # This file was generated by the {rix} R package v0.7.1 on 2024-07-01 #> # with following call: #> # >rix(r_ver = "79b3d4bcae8c7007c9fd51c279a8a67acfa73a2a", #> # > r_pkgs = c("dplyr", #> # > "janitor"), #> # > git_pkgs = list(list(package_name = "housing", #> # > repo_url = "https://github.com/rap4all/housing/", #> # > commit = "1c860959310b80e67c41f7bbdc3e84cef00df18e"), #> # > list(package_name = "fusen", #> # > repo_url = "https://github.com/ThinkR-open/fusen", #> # > commit = "d617172447d2947efb20ad6a4463742b8a5d79dc")), #> # > ide = "none", #> # > project_path = path_default_nix, #> # > overwrite = TRUE) #> # It uses nixpkgs' revision 79b3d4bcae8c7007c9fd51c279a8a67acfa73a2a for reproducibility purposes #> # which will install R version 4.2.1. #> # Report any issues to https://github.com/ropensci/rix #> let #> pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/79b3d4bcae8c7007c9fd51c279a8a67acfa73a2a.tar.gz") {}; #> #> rpkgs = builtins.attrValues { #> inherit (pkgs.rPackages) #> dplyr #> janitor; #> }; #> #> git_archive_pkgs = [ #> (pkgs.rPackages.buildRPackage { #> name = "housing"; #> src = pkgs.fetchgit { #> url = "https://github.com/rap4all/housing/"; #> rev = "1c860959310b80e67c41f7bbdc3e84cef00df18e"; #> sha256 = "sha256-s4KGtfKQ7hL0sfDhGb4BpBpspfefBN6hf+XlslqyEn4="; #> }; #> propagatedBuildInputs = builtins.attrValues { #> inherit (pkgs.rPackages) #> dplyr #> ggplot2 #> janitor #> purrr #> readxl #> rlang #> rvest #> stringr #> tidyr; #> }; #> }) #> #> #> (pkgs.rPackages.buildRPackage { #> name = "fusen"; #> src = pkgs.fetchgit { #> url = "https://github.com/ThinkR-open/fusen"; #> rev = "d617172447d2947efb20ad6a4463742b8a5d79dc"; #> sha256 = "sha256-TOHA1ymLUSgZMYIA1a2yvuv0799svaDOl3zOhNRxcmw="; #> }; #> propagatedBuildInputs = builtins.attrValues { #> inherit (pkgs.rPackages) #> attachment #> cli #> desc #> devtools #> glue #> here #> magrittr #> parsermd #> roxygen2 #> stringi #> tibble #> tidyr #> usethis #> yaml; #> }; #> }) #> ]; #> #> system_packages = builtins.attrValues { #> inherit (pkgs) #> R #> glibcLocales #> nix; #> }; #> #> in #> #> pkgs.mkShell { #> LOCALE_ARCHIVE = if pkgs.system == "x86_64-linux" then "${pkgs.glibcLocales}/lib/locale/locale-archive" else ""; #> LANG = "en_US.UTF-8"; #> LC_ALL = "en_US.UTF-8"; #> LC_TIME = "en_US.UTF-8"; #> LC_MONETARY = "en_US.UTF-8"; #> LC_PAPER = "en_US.UTF-8"; #> LC_MEASUREMENT = "en_US.UTF-8"; #> #> buildInputs = [ git_archive_pkgs rpkgs system_packages ]; #> #> } ``` This will install two packages from GitHub: the `{housing}` package and more specifically the code as it is in the `fusen` branch; however the branch name is not required, as the commit is enough to pin the exact version of the code needed. The `{fusen}` package is also installed, as of commit `d617172447d`. If you want to install a package from GitHub, which store the R package in a subfolder, you should specify the subfolder in the `repo_url` argument. For example, if you want to install the R version of the package `{BPCells}`: ```{r, eval = F} path_default_nix <- tempdir() rix( r_ver = "4.5.0", r_pkgs = c("dplyr"), system_pkgs = NULL, git_pkgs = list( package_name = "BPCells", repo_url = "https://github.com/bnprks/BPCells/r", commit = "16faeade0a26b392637217b0caf5d7017c5bdf9b" ), ide = "none", project_path = ".", overwrite = TRUE, print = TRUE ) ``` ## Installing local archives It is also possible to install packages from a local `tar.gz` file. For this, place the package in the same folder where the `default.nix` will be generated, and write something like this: ```{r, eval = F} rix( r_ver = "4.3.1", local_r_pkgs = c("chronicler_0.2.1.tar.gz", "knitr_1.43.tar.gz"), overwrite = TRUE ) ``` This assumes that both `chronicler_0.2.1.tar.gz` and `knitr_1.43.tar.gz` have been downloaded beforehand. If you want to include a local package that you are developing, make sure to first build it using `devtools::build()` to build the `.tar.gz` archive, but if you can, consider uploading the source code to your package on GitHub and installing it from GitHub instead. ## Installing Python packages It is also possible to add Python packages to an environment, by passing a list of two elements to the `py_conf` argument of `rix()`. This list needs to first specify a Python version, and then an atomic character vector of Python packages: ```{r, eval = F} rix( date = "2025-02-17", r_pkgs = "ggplot2", py_conf = list( py_version = "3.12", py_pkgs = c("polars", "great-tables") ), overwrite = TRUE ) ``` This will install Python 3.12, the `polars` and `great-tables` packages, but also `ipykernel` and `pip` to ensure the Python interpreter works correctly with IDEs such as Positron. ## Installing Python packages not available via nixpkgs (impure) Not all Python packages can be installed through Nix; unlike CRAN, Pypi doesn’t get automatically mirrored and individual packages fixed by volunteers. Instead, specific Python packages get packaged individually for Nix. Thus, it could very well be the case that a specific Python package (or version of a Python package) that you need for a project is not available via nixpkgs. In this case, it is still possible to use Python-specific package managers, like `uv`, to install packages. This is also useful if you work on a project with colleagues that use `uv` and that don’t want (yet) to use Nix. `uv`, is 10-100x faster than `pip` and also generates a lock file for improved reproducibility. The idea is to install `uv` in your shell (but not any Python nor Python packages): ```{r, eval = F} rix( ... system_pkgs = c("uv"), ... ) ``` And then use `uv` from your shell as you would usually. We recommend specifying Python packages in a `requirements.txt` file, and specifying explicit versions (e.g., `scanpy==1.11.4`). Finally, we also recommend setting a shell hook to set up the virtual environment and install the packages from the `requirements.txt` when entering the shell (mind the quotes): ```r rix( ... system_pkgs = c("uv"), shell_hook = " if [ ! -f pyproject.toml ]; then uv init --python 3.13.5 # or whichever Python version you need fi uv add --requirements requirements.txt # Create alias so python uses uvs environment alias python='uv run python' ", ) ``` After running `nix-shell`, `uv` should initalize a Python project with the specified Python version and install the packages listed in `requirements.txt` within the nix environment. This will take place each time `nix-shell` is called, however this will be cached and not installed each time. To make sure everything works fine, you could simply start a Python interpreter and try to load `numpy`. This should work fine, but if it doesn’t, the following error could be raised: ```bash ImportError: IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE! Importing the numpy C-extensions failed. This error can happen for many reasons, often due to issues with your setup or how NumPy was installed. We have compiled some common reasons and troubleshooting tips at: https://numpy.org/devdocs/user/troubleshooting-importerror.html Please note and check the following: * The Python version is: Python3.13 from "/home/user/projects/rix_uv/.venv/bin/python3" * The NumPy version is: "2.2.6" and make sure that they are the versions you expect. Please carefully study the documentation linked above for further help. Original error was: libstdc++.so.6: cannot open shared object file: No such file or directory ``` That is an issue when using *wheels* (wheels are binaries of Python packages that get installed by default using `uv`). These wheels expect certain libraries to be in certain places. One way to solve this is to add the following to your shell hook: ```r shellHook = '' # Export LD_LIBRARY is required for python packages that dynamically load libraries, such as numpy export LD_LIBRARY_PATH="${pkgs.lib.makeLibraryPath (with pkgs; [ zlib gcc.cc glibc stdenv.cc.cc ])}":LD_LIBRARY_PATH; ... ``` Your environment should now work. If this seems complicated: yes, and that is actually exactly the type of problems that Nix aims to solve. However, there are just too many Python packages to automate their inclusion into nixpkgs like how it’s done for R. If you can, prefer using the Python packages included in nixpkgs. ## Installing Julia packages Unlike with R packages, date‑specific snapshots of the Julia registry only become available starting on the 2025-09-04. If you choose an earlier date, Nixpkgs will fall back to whatever registry snapshot was bundled at that time—which will be an older version, and it may not install successfully. So if you need Julia, use a date after the 2025-09-04. To add Julia packages, you can use the `jl_conf` argument of `rix()`. This list needs to first specify a Julia version, and then an atomic character vector of packages: ```{r, eval = F} rix( date = "2025-09-04", r_pkgs = "ggplot2", jl_conf = list( jl_version = "1.11", jl_pkgs = c("TidierData", "GLM") ), project_path = ".", overwrite = TRUE ) ``` This will install Julia 1.11, the `TidierData` and `GLM` packages. You can also install the long-term support version of Julia by using `"lts"` as the `jl_version`. ## Converting from an renv.lock file `{rix}` also includes an `renv2nix()` function that converts an `renv.lock` file into a valid Nix expression. Read the vignette `vignette("f-renv2nix")` to learn more. ## A complete example This example shows how to install packages from CRAN, from the CRAN archives and from GitHub: ```{r, eval = F} path_default_nix <- tempdir() rix( r_ver = "4.2.1", r_pkgs = c("dplyr", "janitor", "AER@1.2-8"), git_pkgs = list( list( package_name = "housing", repo_url = "https://github.com/rap4all/housing/", commit = "1c860959310b80e67c41f7bbdc3e84cef00df18e" ), list( package_name = "fusen", repo_url = "https://github.com/ThinkR-open/fusen", commit = "d617172447d2947efb20ad6a4463742b8a5d79dc" ) ), ide = "none", project_path = path_default_nix, overwrite = TRUE ) ``` ``` #> # This file was generated by the {rix} R package v0.7.1 on 2024-07-01 #> # with following call: #> # >rix(r_ver = "79b3d4bcae8c7007c9fd51c279a8a67acfa73a2a", #> # > r_pkgs = c("dplyr", #> # > "janitor", #> # > "AER@1.2-8"), #> # > git_pkgs = list(list(package_name = "housing", #> # > repo_url = "https://github.com/rap4all/housing/", #> # > commit = "1c860959310b80e67c41f7bbdc3e84cef00df18e"), #> # > list(package_name = "fusen", #> # > repo_url = "https://github.com/ThinkR-open/fusen", #> # > commit = "d617172447d2947efb20ad6a4463742b8a5d79dc")), #> # > ide = "none", #> # > project_path = path_default_nix, #> # > overwrite = TRUE) #> # It uses nixpkgs' revision 79b3d4bcae8c7007c9fd51c279a8a67acfa73a2a for reproducibility purposes #> # which will install R version 4.2.1. #> # Report any issues to https://github.com/ropensci/rix #> let #> pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/79b3d4bcae8c7007c9fd51c279a8a67acfa73a2a.tar.gz") {}; #> #> rpkgs = builtins.attrValues { #> inherit (pkgs.rPackages) #> dplyr #> janitor; #> }; #> #> git_archive_pkgs = [ #> (pkgs.rPackages.buildRPackage { #> name = "housing"; #> src = pkgs.fetchgit { #> url = "https://github.com/rap4all/housing/"; #> rev = "1c860959310b80e67c41f7bbdc3e84cef00df18e"; #> sha256 = "sha256-s4KGtfKQ7hL0sfDhGb4BpBpspfefBN6hf+XlslqyEn4="; #> }; #> propagatedBuildInputs = builtins.attrValues { #> inherit (pkgs.rPackages) #> dplyr #> ggplot2 #> janitor #> purrr #> readxl #> rlang #> rvest #> stringr #> tidyr; #> }; #> }) #> #> #> (pkgs.rPackages.buildRPackage { #> name = "fusen"; #> src = pkgs.fetchgit { #> url = "https://github.com/ThinkR-open/fusen"; #> rev = "d617172447d2947efb20ad6a4463742b8a5d79dc"; #> sha256 = "sha256-TOHA1ymLUSgZMYIA1a2yvuv0799svaDOl3zOhNRxcmw="; #> }; #> propagatedBuildInputs = builtins.attrValues { #> inherit (pkgs.rPackages) #> attachment #> cli #> desc #> devtools #> glue #> here #> magrittr #> parsermd #> roxygen2 #> stringi #> tibble #> tidyr #> usethis #> yaml; #> }; #> }) #> #> (pkgs.rPackages.buildRPackage { #> name = "AER"; #> src = pkgs.fetchzip { #> url = "https://cran.r-project.org/src/contrib/Archive/AER/AER_1.2-8.tar.gz"; #> sha256 = "sha256-OqxXcnUX/2C6wfD5fuNayc8OU+mstI3tt4eBVGQZ2S0="; #> }; #> propagatedBuildInputs = builtins.attrValues { #> inherit (pkgs.rPackages) #> car #> lmtest #> sandwich #> survival #> zoo #> Formula; #> }; #> }) #> ]; #> #> system_packages = builtins.attrValues { #> inherit (pkgs) #> R #> glibcLocales #> nix; #> }; #> #> in #> #> pkgs.mkShell { #> LOCALE_ARCHIVE = if pkgs.system == "x86_64-linux" then "${pkgs.glibcLocales}/lib/locale/locale-archive" else ""; #> LANG = "en_US.UTF-8"; #> LC_ALL = "en_US.UTF-8"; #> LC_TIME = "en_US.UTF-8"; #> LC_MONETARY = "en_US.UTF-8"; #> LC_PAPER = "en_US.UTF-8"; #> LC_MEASUREMENT = "en_US.UTF-8"; #> #> buildInputs = [ git_archive_pkgs rpkgs system_packages ]; #> #> } ``` The next vignette, `vignette("d2-installing-system-tools-and-texlive-packages-in-a-nix-environment")`, explains how you can install tools such as text editors, git, Quarto, TexLive packages, and any other tool available through `nixpkgs` for your development environments. ## Package installation issues Some R packages are quite difficult to install: that is usually not an issue for most users that use either Windows or macOS as their operating systems, because when calling `install.packages()` a compiled binary gets downloaded from CRAN and installed in a matter of seconds. On Ubuntu, likely the most popular Linux distribution, binary packages for R packages are also available via the [r2u](https://eddelbuettel.github.io/r2u/) repository. However, if you need to install old packages, these instead will need to be installed from source, as binaries for old packages are not kept. For most packages, this is not an issue, but some packages require compilation and this is where issues start. Nix solves this, because all packages must have their dependencies also declared, so installing old packages should not be an issue. However, it can happen that one particular package that you want to install may not build. This can happen because, even though we spend a lot of time making sure R packages work flawlessly with Nix, there are many R packages (almost 30'000 between CRAN and Bioconductor) and there's not many of us (R contributors for Nix). Should you have trouble installing a package, feel free to open an issue and we'll do our best to fix it! We also made sure that old packages would work by backporting many fixes, and actually building many old versions of popular packages for all the dates included in `available_dates()`. Here is the list of packages that were built and tested (but keep in mind that this list doesn't show all the dependencies of all the packages that also have to work, and that just because a package isn't listed, doesn't mean it's not going to work!):
Click to show package list ``` DBI R6 RColorBrewer RCurl RSQLite Rcpp RcppEigen arrow askpass backports base64enc bit bit64 blob broom bslib cachem callr cellranger cli clipr collapse colorspace cpp11 crayon curl data_table dbplyr devtools digest dplyr duckdb evaluate fansi farver fastmap fontawesome forcats fs gargle generics ggplot2 glue gtable haven highr hms htmltools htmlwidgets httr icosa igraph isoband jquerylib jsonlite kit knitr labeling languageserver later lifecycle lubridate magrittr memoise mime modelr munsell nloptr openssl openxlsx pillar pkgconfig prettyunits processx progress promises ps purrr rJava ragg rappdirs readr readxl rematch rematch2 rlang rmarkdown rprojroot rstan rstudioapi rvest sass scales selectr Seurat sf shiny stars stringi stringr sys systemfonts terra textshaping tibble tidyr tidyselect tidyverse timechange tinytex tzdb utf8 vctrs viridisLite withr xfun xlsx xml2 yaml zoo ```