Then the packages.
Two halves of the same job. One rebuilds and tests the approved package list inside the workspace analysts actually use; the other builds that workspace from a written recipe first. Both end in binaries plus per-package evidence, so a claim that these packages work here is a documented fact rather than a hope.
The software. Builds compilers, R, Python and the libraries into the foundation, as versioned modules you load on demand.
The extractions
Files in the repo. Pull it and run - nothing to unpack.Package build
package-build
7 files
57 KB
extracted from
eks.regeneron.ndexr.io
A quality-control kit for the R and Python packages analysts rely on. It takes the approved package list, rebuilds every package from its original source code inside the exact workspace people use, tests each one there, and writes down what passed and what failed - so "these packages work here" is a documented fact, not a hope.
A manifest-driven R package pipeline - fetch, check, build - as a plugin: estate-agnostic naming throughout, so it drops into anyone's CI or runs by hand inside the container it validates, in the working directory. Same stages, cache layout, check flags and evidence CSV as the source pipeline; gone is the machinery around them - no podman, no mounts, no registry, no arch matrix. One run validates this container, this architecture, one R version.
build-packages.sh manifest pipeline README.md run.R walkthrough.qmd
- The container to validate, with R at the requested version (/opt/R/<ver> or PATH)
- A compiler toolchain (gcc, g++, gfortran, make) - preflight names anything missing
- Network for fetch and check (the dependency closure installs live, as in CI)
- ~20 GB free in the working directory
# from R, via the rxedn catalogue
rxedn::rxedn_install_module("repo.extract.ndexr.io")
# the extraction lands at <domains_dir>/repo.extract.ndexr.io/package-build/
No archive to unpack and no checksum to compare: rxedn publishes each
repo as an immutable
<host>-<sha7>.tar.gz
and records its sha256 in the catalogue index, so the commit is the
identity. Pull it, and the files are simply there.
cd package-build
# from R (Workbench session) - session-core-aware, R version = the session's R:
# source('run.R'); kickoff('preflight'); kickoff('all', background = TRUE)
./build-packages.sh --r-version 4.6.1 --stage preflight # shell lane - seconds
./build-packages.sh --r-version 4.6.1 --stage all # hours; PKG_REPO=<dated snapshot> to pin
Output stays put, in
./pkg-cache/bin/<arch>/<rver>/ (evidence in ./pkg-cache/evidence/)
- nothing is copied out.
Every file - 7
- build-packages.sh
- manifest/manifest.dcf
- manifest/sysreqs-rhel9.txt
- pipeline/gitlab-ci.package-build.yml
- README.md
- run.R
- walkthrough.qmd
RHEL Posit container + package pipeline
rhel-posit-pipeline
17 files
75 KB
extracted from
eks.regeneron.ndexr.io
A factory for the analyst workspace itself. From a written recipe it builds the standard RStudio/Posit environment data scientists log into, then runs every approved package through that environment as proof. The result is a workspace that can be recreated identically anywhere, and packages with evidence that they work in it.
The session container and the pipeline that qualifies R packages through it: a RHEL 9 Posit Workbench/Connect session image (four R, three Pythons, the full system-library floor) plus a builder container that compiles the declared set from source against exactly that image - fetch, install, check, test, build - ending in binaries and per-package evidence. A session and source builder only: no Kubernetes, no Helm, no orchestration, no registry assumptions, no credentials.
build-image.sh containers manifest pipeline README.md run-stage.sh run.R
- A build host with podman or docker - image builds cannot happen inside a Workbench session
- ~30 GB free (the session image carries four R + three Python runtimes, TeX, chromium)
- Network for the image build (Posit CDN, dnf) and for fetch/install (PKG_REPO)
- Hours for the first image build; the package stages resume from ./pkg-cache
# from R, via the rxedn catalogue
rxedn::rxedn_install_module("repo.extract.ndexr.io")
# the extraction lands at <domains_dir>/repo.extract.ndexr.io/rhel-posit-pipeline/
No archive to unpack and no checksum to compare: rxedn publishes each
repo as an immutable
<host>-<sha7>.tar.gz
and records its sha256 in the catalogue index, so the commit is the
identity. Pull it, and the files are simply there.
cd rhel-posit-pipeline
# from R (RStudio on the build host) - session-core-aware:
# source('run.R'); build_image(); kickoff('all', background = TRUE)
./build-image.sh # the two containers
./run-stage.sh --r-version 4.6.1 --stage fetch # then install, check, test, build
Output stays put, in
./pkg-cache/bin/<arch>/<rver>/ (evidence in ./pkg-cache/evidence/) + the session image in the local engine
- nothing is copied out.
Every file - 17
- build-image.sh
- containers/package-builder/build-packages.sh
- containers/package-builder/Dockerfile
- containers/session/Dockerfile
- manifest/environments/r-full/bioc_packages.txt
- manifest/environments/r-full/cran_packages.txt
- manifest/environments/r-full/git_repos.dcf
- manifest/environments/r-full/manifest.yaml
- manifest/environments/r-minimal/bioc_packages.txt
- manifest/environments/r-minimal/cran_packages.txt
- manifest/environments/r-minimal/git_repos.dcf
- manifest/environments/r-minimal/manifest.yaml
- manifest/sysreqs-rhel9.txt
- pipeline/gitlab-ci.rhel-posit-pipeline.yml
- README.md
- run-stage.sh
- run.R
Declared inputs
What each extraction will build.Package build
Carried in
manifest/
, verbatim - the declared set
this extraction will build. It travels in the repo, so a copy taken
to a machine with no route back here still knows the full list.
| File | Records | Size |
|---|---|---|
manifest.dcf
|
197 records | 10 KB |
sysreqs-rhel9.txt
|
31 entries | 1 KB |
RHEL Posit container + package pipeline
Carried in
manifest/
, verbatim - the declared set
this extraction will build. It travels in the repo, so a copy taken
to a machine with no route back here still knows the full list.
| File | Records | Size |
|---|---|---|
environments/r-full/bioc_packages.txt
|
empty | 0 KB |
environments/r-full/cran_packages.txt
|
159 entries | 2 KB |
environments/r-full/git_repos.dcf
|
3 records | 1 KB |
environments/r-full/manifest.yaml
|
12 entries | 1 KB |
environments/r-minimal/bioc_packages.txt
|
empty | 0 KB |
environments/r-minimal/cran_packages.txt
|
87 entries | 1 KB |
environments/r-minimal/git_repos.dcf
|
3 records | 1 KB |
environments/r-minimal/manifest.yaml
|
12 entries | 1 KB |
sysreqs-rhel9.txt
|
31 entries | 1 KB |