Moving a simple httpd web server from Rancher 1 to Rancher 2¶

Below we describe the steps taken to migrate a simple Apache httpd web server from Rancher 1 to Rancher 2.

Background¶

NERSC maintains the website https://docs-dev.nersc.gov for documenting usage of testbeds and non-production resources at NERSC. The content is hosted in a git repository on GitLab; it is based on mkdocs, and is cloned from the structure of the primary NERSC documentation website. One runs mkdocs build to render the mkdocs content in the repository into HTML viewable from a web browser.

Why host the site in Rancher instead of on gitlab.io?¶

The simplest way to host the website whose content is stored on GitLab and which uses a markdown-like language to store its content is directly from GitLab itself, which provides a rudimentary website hosting service via gitlab.io. The problem with this approach is that GitLab’s webpage rendering service does not support HSTS, which is required for websites hosted under the .gov domain, including, by extension, anything hosted under the nersc.gov domain. GitLab is aware of the limitation, but has made little progress toward resolving it.

By contrast, Rancher automatically enables HSTS when an appropriate SSL or TLS certificate is provided for a website hosting service running in Rancher.

Migrating from Rancher 1 to Rancher 2¶

Historically the https://docs-dev.nersc.gov website was hosted in Rancher 1. But in November 2020 it was migrated to Rancher 2, which simplified considerably the workflow of keeping the website up to date; the git repository itself remains in the same place on gitlab.com.

Scalable deployment and CronJob configuration¶

The Rancher 2 configuration for the website consists of two Rancher Workloads. One Workload is a CronJob which runs once per hour. It is a container which uses a Docker ENTRYPOINT command to clone the latest version of the git repository from gitlab.com, and then execute the mkdocs build command, storing the resulting rendered HTML into a persistent NFS volume in Spin. This Workload must be a CronJob and not a Scalable deployment because Rancher's Kubernetes orchestrator tries to keep all Scalable deployments alive - if the container exits or returns, the orchestrator will immediately try to restart it. In contrast, a Job or CronJob is allowed to exit or return, and the orchestrator will not try to restart it (unless it is a Cron Job, in which case it will execute again at the next scheduled time). The idea behind this CronJob approach is to mimic a continuous deployment workflow, athough it is not a true CD solution. This approach is not as immediate as true CI/CD, but it is never more than 1 hour behind the most recent changes to the website.

The second Workload contains a simple Apache web server, cloned from the httpd:latest image on Dockerhub, and configured as a Scalable deployment of 1 Pod. In Rancher 2, the persistent NFS volume where the rendered HTML is stored from the CronJob is mounted into the httpd container.

CNAME record request¶

We then filed a request to the CNAME record authority for the nersc.gov domain to change the existing CNAME record for https://docs-dev.nersc.gov from the old URL running in Rancher 1 to the new URL running in Rancher 2. We also added an ingress rule in Rancher 2 to add the URL https://docs-dev.nersc.gov to the httpd container; we forgot to do this initially after completing the CNAME record change, and it resulted in 404 errors when trying to navigate a web browser to the URL https://docs-dev.nersc.gov.

TLS certificate generation¶

Finally, we ran into an issue that our new httpd server running in Rancher 2 did not have a proper TLS certificate, so HTTPS access to the website was not available. This caused most browsers (Safari, Firefox, Chrome) to either warn that the site was insecure, or to refuse to load the site altogether. So we generated a signed certificate for this new website. First, we generated an RSA key by running the command¹

openssl genrsa -out docs-dev.nersc.gov.key 2048

Next, we used that key to generate a certificate signing request (CSR):

openssl req -new -sha256 -key ./docs-dev.nersc.gov.key -out ./docs-dev.nersc.gov.csr

After generating the RSA key and CSR, we then submitted these to the Certificate Authority for the nersc.gov domain, who provided a signed certficiate for the website, which we then attached to the httpd container running in Rancher 2. This resolved the security warnings that several browsers showed earlier, and now the website works without any problem. Attaching the certificate in Rancher 2 also enables HSTS automatically.