HPC Thread Management and Scheduler Quota Detection

Source: R/hpc_threads.R

Functions to dynamically query, configure, and scope the worker thread allocation used by the Rust high-performance computing backend in hySpc.hpc.

Details

In multi-tenant HPC cluster environments (Slurm, PBS, Grid Engine) or shared multi-core cloud instances, multiple R worker processes frequently execute on the same physical node. Setting environment variables like RAYON_NUM_THREADS after Rayon has already initialized has no effect in pure Rust.

hySpc.hpc solves this via a dedicated thread pool manager in Rust:

  • set_hpc_threads() dynamically adjusts the active Rayon worker pool size at runtime without restarting the R process.

  • get_hpc_threads() inspects the currently allocated thread budget.

  • detect_hpc_cores() automatically parses scheduler environment variables (SLURM_CPUS_PER_TASK, PBS_NCPUS, NSLOTS) to determine the exact CPU allocation granted to the current job.

  • with_hpc_threads() provides an RAII-style scoped context manager that temporarily executes code under a specific thread budget and restores the previous configuration upon exit.