Replication package for "ORCAID: Oblique Rule-Based Continuous-Action Interpretation for Deep RL Policies"

Replication package for "ORCAID: Oblique Rule-Based Continuous-Action Interpretation for Deep RL Policies" (NeurIPS 2026)

ORCAID extracts interpretable rule-based surrogate models from deep reinforcement learning policies with continuous actions and mixed continuous-discrete states. It learns oblique decision trees with linear models in their leaves, then simplifies them into a small set of rules. This package contains the code and run configurations needed to reproduce all experiments of the paper.

Contents
- Python implementation of ORCAID and of the baselines (decision trees, Cubist, RuleFit), all wrapped in a DAgger loop.
- Scripts for RL training (DDPG, Stable-Baselines3), rule learning, evaluation, policy improvement through rule mirroring (RQ3), LLM-as-a-Judge interpretability evaluation (RQ4), and an interactive tool to explore learned rules.
- Run lists for all experiments (scripts/experiments_*.csv, one row per run, including all seeds), plus the SLURM job scripts generated from them with generate_scripts.py.
- analyse_results.py, which aggregates the results into the tables and figures of the paper, and tools/recompute_model_sizes.py.
- tools/make_cubist_local.py, which creates the modified Cubist file used by the Cubist baseline from the installed cubist package (SHA-256 verified).
- A README with installation instructions, all command-line options, an example run on Mountain Car with plots, a table mapping each table and figure of the paper to the scripts that produce it, and the licensing information.

Not included: trained RL policies, collected data and learned surrogate models. They are created when running the scripts.

Environments: nine Gymnasium tasks: Mountain Car, Pendulum, Lunar Lander, Inverted Pendulum, Inverted Double Pendulum, Reacher, Swimmer, Hopper and Half Cheetah.

Requirements: tested on Ubuntu 24.04 with Python 3.12.3 and CUDA. The pinned dependencies are in requirements.txt and are installed in a virtual environment with pip install --no-binary cubist -r requirements.txt, which builds Cubist from source (requires gcc 13). Lunar Lander also needs the system package swig. The LLM-as-a-Judge evaluation needs an OpenRouter API key (not included).

Compute: the experiments were run as SLURM jobs on a cluster, each with one NVIDIA A40 GPU and a walltime of up to 72 hours.

Reproduction:
1. Set up the environment as described in the README (system packages, virtual environment, pip install --no-binary cubist -r requirements.txt) and run python tools/make_cubist_local.py once.
2. Run the job scripts in scripts/ (or regenerate them with python generate_scripts.py).
3. Run analyse_results.py to produce the tables and figures.

License: MIT. The package contains no third-party code. The Cubist baseline uses a slightly modified version of cubist.py from the Python package cubist 0.1.4 (GPLv3, https://github.com/pjaselin/Cubist); this file is not included but created locally from the installed package with tools/make_cubist_local.py (see README).

Citation: please cite the paper (see CITATION.cff). Until the NeurIPS proceedings are published:
Lopez-Miguel, I. D., Bartocci, E., Eiter, T., Tappler, M. ORCAID: Oblique Rule-Based Continuous-Action Interpretation for Deep RL Policies. NeurIPS 2026. arXiv:2607.07235, https://doi.org/10.48550/arXiv.2607.07235