This vignette describes how to run
roreviewapi on a Digital Ocean droplet, including the
editor volunteer search feature introduced in v0.2. Endpoints for that
feature are described in the editor
search vignette.
It covers two cases:
A few terms used throughout this vignette:
ssh <user>@<droplet-ip>) or
by using the web console button from Digital Ocean interface.docker-compose.yml.reviewbot.ropensci.org into the droplet’s IP address.
rOpenSci’s DNS is managed in Cloudflare, which also
sits between visitors and the droplet.docker-compose.yml defines two containers:
plumber is the API itself. It is built
from the repository’s Dockerfile, and listens on port 8000,
which is only reachable from the other container.nginx answers on ports 80 (HTTP) and
443 (HTTPS), handles HTTPS with the certificate, redirects HTTP to
HTTPS, and passes requests on to plumber.So a request travels: visitor → Cloudflare → nginx (port
443) → plumber (port 8000).
Follow this section to set up a new droplet. It also applies when a droplet is rebuilt: Digital Ocean’s “Rebuild” reinstalls the operating system and erases the whole disk. Everything that is not in the Git repository is then lost, in particular:
.env file of secrets (see Environment variables),/srv/roreviewapi/data/, which holds
the database and the Gmail credential files,/etc/ssl/cloudflare/,You need:
sudo
rights.ropensci.org domain, or someone who has it.All commands below are run on the droplet, unless stated otherwise.
Git downloads the code from GitHub. Install it with:
Install Docker Engine and its Compose plugin by following Docker’s instructions for Ubuntu. Then check that both work:
Docker must start automatically when the droplet boots, otherwise the service stays down after every reboot:
Git records a name and an email address with every commit. Set them to the bot account, for root (which runs the weekly redeploy, see Cron job) and for your own user:
sudo git config --global user.name "ropensci-review-bot"
sudo git config --global user.email "[email protected]"
git config --global user.name "ropensci-review-bot"
git config --global user.email "[email protected]"The project lives in /home/shared/roreviewapi. Download
it there with:
sudo mkdir -p /home/shared
cd /home/shared
git clone https://github.com/ropensci-review-tools/roreviewapi.git
cd roreviewapiThe remaining commands that refer to files such as .env
or docker-compose.yml are run from this directory.
Secrets are not stored in the
Dockerfile or anywhere else in the Git repository. They
live in a .env file that exists only on the droplet. It is
listed in .gitignore and .dockerignore, and is
passed to the plumber container when it starts, via
env_file: .env in docker-compose.yml. Because
the file is not tracked by Git, git pull never overwrites
it, and the values never end up in an image.
Create .env in the project directory, with one
KEY=value entry per line (no quotes and no
export), and make it readable only by its owner:
| Variable | Description |
|---|---|
GITHUB_PAT |
GitHub personal access token (also used as
GITHUB_TOKEN) |
PKGCHECK_TOKEN |
pkgcheck authentication token: a secret password shared by the Digital Ocean droplet and buffy |
GMAIL_SENDER |
Sender address used for outgoing emails |
GMAIL_OAUTH_CLIENT_SECRET |
Path to the Google OAuth client secret JSON file,
e.g. /data/email/<client_secret>.json |
GMAIL_TOKEN_CACHE |
Path to the cached Gmail token,
e.g. /data/email/gmailr_token |
GMAIL_AUTH_EMAIL |
Google account used to authenticate with Gmail |
AIRTABLE_API_KEY |
AirTable personal access token |
AIRTABLE_BASE_ID |
ID of the AirTable base containing the EiC rotation table |
ROREVIEWAPI_BASE_URL |
Public HTTPS base URL of the deployed API
(https://reviewbot.ropensci.org) |
When the container starts, docker-entrypoint.sh writes
these values to ~/.Renviron (setting
GITHUB_TOKEN from GITHUB_PAT, and
ROREV_CONTAINER=true), so that R can read them, and then
starts the API.
The two GMAIL_* file paths must be inside
/data/email/. That is the container’s view of the data
directory (see Data directory). Place the
corresponding files in /srv/roreviewapi/data/ on the
droplet.
data.records:read scope on the relevant base. Set this as
AIRTABLE_API_KEY.https://airtable.com/appXXXXXXXX/...). Set this as
AIRTABLE_BASE_ID.The editor-in-chief-rotation table within that base must
have period_start, period_end, and
acting_eic_email fields.
The data directory holds the SQLite database, the notify-email cache file, and the Gmail OAuth client secret and token cache files. It lives on the droplet, outside the containers, so that it survives when the containers are rebuilt every week. Create it with:
docker-compose.yml makes it available inside the
plumber container as /data/email. The
container must be able to write to it, because the Gmail token cache is
updated during authentication.
After a rebuild, restore its contents, and .env, from a
backup (see Backing up the
data and secrets). Without a backup, the database starts empty, and
the Gmail authorization must be redone, following the editor
search vignette.
The domain’s DNS is managed in Cloudflare. It needs an A
record pointing the subdomain to the droplet’s IP address, for
example:
reviewbot.ropensci.org. IN A <droplet-ip>The record must be proxied (orange cloud in the Cloudflare dashboard), not “DNS only”. The TLS certificate below is a Cloudflare Origin Certificate, which only Cloudflare trusts, so visitors must reach the droplet through Cloudflare.
A rebuilt droplet keeps its IP address, so the record does not change. A new droplet has a new IP address, so the record must be updated.
TLS uses a Cloudflare Origin Certificate. It secures the connection between Cloudflare and the droplet, is valid for up to 15 years, and needs no renewal on the droplet.
In the Cloudflare dashboard for the domain, go to SSL/TLS
→ Origin Server → Create Certificate. Keep the default key
type, list the hostname (reviewbot.ropensci.org), and
choose a validity period.
Cloudflare shows the certificate and the private key. Cloudflare
does not store the private key, so it is only shown once: save it
straight away. A copy of the current key is stored in rOpenSci’s
1Password. On the droplet, save them as
/etc/ssl/cloudflare/origin.pem and
/etc/ssl/cloudflare/origin.key.
Create the directory and the certificate file. tee waits
for input: paste the certificate, press Enter, then
Ctrl+D to save:
Do the same for the private key, then make it readable only by root:
sudo tee is needed because in
sudo echo ... > file the redirection runs as the current
user, not as root. Sending the output to /dev/null keeps
the pasted key from being printed back to the terminal.
Check that both files are valid:
sudo openssl x509 -in /etc/ssl/cloudflare/origin.pem -noout -subject -enddate
sudo openssl pkey -in /etc/ssl/cloudflare/origin.key -noout && echo "key OK"The first command shows the hostname and the expiry date. The second prints “key OK” if the key can be read.
In SSL/TLS → Overview, set the encryption mode to Full (strict).
After a rebuild, there is no need to create a new certificate. Copy
the key back from 1Password, and download the certificate again from the
Cloudflare dashboard (SSL/TLS → Origin Server). Save
both with the tee commands above.
From the project directory, build the images and start the containers in the background:
The first build takes a while. nginx only starts once
plumber reports itself as healthy, which can take a few
minutes.
The compose file:
/srv/roreviewapi/data into the
plumber container at /data/email, and sets
ROREVIEWAPI_EMAIL_DB=/data/email/searches.sqlite, so that
the SQLite database persists across rebuilds..env via env_file./etc/ssl/cloudflare read-only into the
nginx container, so that the certificate is available.NGINX_SERVER_NAME), which is
substituted into nginx.conf when the container starts.restart.sh downloads new commits and rebuilds the
service (see Weekly redeploy). A cron job
runs it every week. One authorized admin of the droplet needs to set it
up.
The job must run as a user that can run docker compose
(either root or a member of the docker group). The
following uses root’s crontab, so that the log file below can be written
to /var/log.
Check whether a job already exists (see Viewing, changing and checking the cron job), to avoid creating a duplicate.
Open the crontab for editing. If prompted, choose an editor
(nano is the easiest):
Add the following line:
This runs the script at 00:00 every Tuesday. The five leading fields are minute, hour, day of month, month, and day of week (0–7, where both 0 and 7 are Sunday). crontab.guru is useful for checking a schedule. The trailing redirect appends all output, including errors, to a log file.
Save and exit. Cron picks up the change automatically; no restart of the cron service is needed.
An absolute path is required because cron does not run in the project
directory. restart.sh finds the project directory (and thus
.env) from its own location, so it needs no other
configuration.
Both containers should be running, with plumber marked
healthy, and curl should print
200. Otherwise, see Troubleshooting.
Finally, set up the uptime alert if it does not already exist.
The Software Peer Review Lead should set up a regular alert to check that the service is up. This bash script prints a message and fails if anything is wrong:
#!/usr/bin/env bash
URL="https://reviewbot.ropensci.org"
status=$(curl -s -o /dev/null -w "%{http_code}" "$URL")
if [[ "$status" != "404" ]]; then
echo -e "reviewbot is down"
exit 1
fiIt should then be run regularly, for example from a
~/.bashrc file, or from a cron job.
The cron job runs restart.sh
every Tuesday. The script:
git pull --ff-only,So changes pushed to GitHub reach the droplet within a week, and the images pick up updated R packages. The service is down while the images rebuild.
To redeploy straight away, for example after pushing a fix, run the script by hand:
List the jobs in root’s crontab with:
Omit sudo to list the current user’s own crontab
instead. Each user has a separate crontab, so if nothing appears, check
the other user (or root) that may have been used to set the job up.
To change the job, re-open the crontab with
sudo crontab -e, edit the line (for example, change the
schedule fields), and save. To disable the job without deleting it,
comment the line out by prefixing it with #. To remove it,
delete the line. Confirm the result with
sudo crontab -l.
To check that the job ran, read the end of its log file:
and confirm that the containers are up with
docker compose ps.
To restart the containers without rebuilding them, for example after
changing .env or installing a new certificate, run from the
project directory:
To change a secret, edit .env on the droplet and
restart. No change to the Dockerfile is needed.
The data directory (/srv/roreviewapi/data/) and the
.env file are the only things on the droplet that cannot be
recreated from GitHub or Cloudflare. Keep a copy of them off the
droplet, and refresh it before risky changes such as an operating system update, and
after changing .env or redoing the Gmail authorization.
The backup contains secrets, so store it somewhere safe, such as rOpenSci’s 1Password, rather than in email, chat or a shared drive.
On the droplet, stop the API briefly, so that the database is not being written to while it is copied:
Pack the data directory and .env into one archive in
your home directory, make it yours so that it can be downloaded, and
start the API again:
sudo tar -czf ~/roreviewapi-backup.tar.gz -C / srv/roreviewapi/data home/shared/roreviewapi/.env
sudo chown $USER ~/roreviewapi-backup.tar.gz
docker compose start plumber-C / makes the paths in the archive start from the root
of the disk, so that they can be restored to the same places.
On your own computer, download the archive with scp,
which copies files over SSH. <user> and
<droplet-ip> are the same as for
ssh:
Back on the droplet, delete the archive, so that no extra copy of the secrets is left behind:
To check what an archive contains, without unpacking it, run
tar -tzf roreviewapi-backup.tar.gz.
On a rebuilt droplet, restore the backup after getting the code, and before starting the service.
On your own computer, upload the archive to your home directory on the droplet:
On the droplet, unpack it, which puts the files back in
/srv/roreviewapi/data/ and
/home/shared/roreviewapi/.env, then delete the archive:
Unpacking with sudo keeps the original owners and
permissions, including the restricted permissions of .env
and of the data directory.
If the service is already running, restart it so that it picks up the restored files.
Update the droplet’s operating system every 2 years. Ubuntu publishes a new long-term support (LTS) release every 2 years, and older releases eventually stop receiving security fixes. Check the current release with:
.env first, because rebuilding
erases the whole disk.Create a new certificate when the current one is about to expire,
when the key is lost, or when it may have been exposed. Nothing renews
the certificate automatically, so check its expiry date with the
openssl x509 command in TLS
certificate. The current certificate expires on 16 October 2041.
Create a new certificate and install it on the droplet, following steps 1 and 2 of TLS certificate. Replace the key in 1Password with the new one.
Restart nginx so that it loads the new files, and
check that the site responds:
Revoke the old certificate in SSL/TLS → Origin Server, so that the old key can no longer be used.
Cloudflare’s Origin CA documentation describes creating, installing and revoking origin certificates in more detail.
Cloudflare reports problems with the droplet as status codes:
| Status | Likely cause |
|---|---|
| 521 | Nothing answers on port 443: nginx or the whole stack is down, or nginx cannot find the certificate files and exits on start |
| 522 | The droplet is unreachable, for example a firewall blocks port 443 |
| 526 | The certificate is invalid, expired, or does not match the hostname |
| 502 | nginx is up but plumber is not |
For 521 or 502, check the containers and their logs on the droplet:
docker compose ps
docker compose logs --tail=50 nginx
docker compose logs --tail=50 plumber
sudo ls /etc/ssl/cloudflare/For 522, there are two independent firewalls to check.
Droplet firewall (ufw):
If ufw is active and port 443 is not listed, open it:
Digital Ocean cloud firewall: in the Digital Ocean console under Networking → Firewalls, confirm there is an inbound rule allowing TCP port 443. The cloud firewall sits in front of the droplet and blocks traffic before it reaches ufw, so both layers must allow it.
Optionally, Authenticated Origin Pulls make nginx accept only connections that come through Cloudflare.
Maintenance falls into two parts with very different transfer costs:
.env changes.When the new person only deploys and debugs, and the existing authorizing account stays active:
sudo access on the droplet.roreviewapi GitHub repository.ro-auto-email Google Cloud project’s
IAM..env
securely, for example via a password manager rather than email or
chat.This is required if the outgoing maintainer’s accounts will become
unusable, for example on leaving the organization. The values in
.env that are tied to a person are:
| Variable | Tied to | Action on transfer |
|---|---|---|
GITHUB_PAT |
GitHub account that created it | Replace with a token created by the new maintainer |
AIRTABLE_API_KEY |
AirTable account that created it | Replace with a token created by the new maintainer (see AirTable for the required scope) |
GMAIL_AUTH_EMAIL, GMAIL_TOKEN_CACHE |
Google account that authorized gmailr |
Replace, following the Gmail transfer steps in the editor search vignette |
GMAIL_OAUTH_CLIENT_SECRET,
GMAIL_SENDER |
Google Cloud project and the [email protected]
alias |
Unchanged if the same Cloud project and alias are kept |
AIRTABLE_BASE_ID, ROREVIEWAPI_BASE_URL,
PKGCHECK_TOKEN |
Not tied to a person | Unchanged unless the base, URL or pkgcheck service changes |
The steps are then:
The new maintainer creates their own GitHub and AirTable personal access tokens, and completes the Gmail steps in the editor search vignette.
Update .env on the droplet with the new
values.
Restart the service so that
the container picks up the new .env and
credentials.
Verify that the editor search workflow still sends and receives emails (see the editor search vignette).
Once the new setup works, the outgoing maintainer should revoke their old GitHub and AirTable tokens, and the old Google grant.
git config lines in
the Dockerfile hold the user.name and
user.email used for commits made inside the container.