# Parallel SDO and SDOclust: Evaluation Experiments **Paper Artifact** -- Aug 2026 --- ## Warning The experiments were tested in the following computational environments. Some experiments are resource-intensive and may fail or require substantially longer execution times on machines with fewer CPU cores or less available memory. **(a) SLURM cluster.** A coordinator job (1 node, 2 CPUs, 32 GB RAM, 24 h walltime) orchestrated the experiments and instantiated the Dask scheduler/client. Dask-based methods provisioned their workers dynamically via `dask_jobqueue`: 16 independent single-core SLURM jobs were requested, each with 1 CPU, 16 GB RAM, and 24 h walltime. The resulting 16 workers provided 16 worker cores in total and were scheduled by SLURM across the available physical nodes. **(b) Local machine.** Intel Xeon E5-2620 v4 CPU (8 physical cores, 16 threads) with 160 GB RAM. All methods were executed on CPU; distributed variants used a local Dask cluster to emulate distributed execution. ## Documentation - **docs/sdo.md**: SDO / SDOclust algorithm, parameters, NumPy vs Dask backends. - **docs/comparison_tests.md**: what `comparison_tests.py` measures and how results are structured. - **docs/scaling.md**: what `scaling.py` measures (weak/strong scaling) and how results are structured. - **docs/stream_test.md**: what `stream_test.py` measures (incremental batch streaming) and how results are structured. ## Experiment config vs. execution config Every script takes two things: - **an experiment config** (what to run -- algorithm, dataset, parameters). For `stream_test.py`, these live in `code/stream_config/*.json`. `scaling.py` and `comparison_tests.py` don't need one (they generate their own parameter grid). - **an execution config** (where/how to run it -- local machine or SLURM cluster, how many workers). These live in `code/infrastructure_config/local.json` and `code/infrastructure_config/slurm.json`. **This is the only file you need to edit for your own environment** (partition name, cores, memory, email). All three modes below (a/b/c) use this same pair of files; only *how you invoke Python* changes. --- ## (a) Without containers Requires the exact package versions in `code/requirements.txt`. ```bash python3 -m venv .venv && source .venv/bin/activate pip install -r code/requirements.txt cd code bash run_all.sh infrastructure_config/local.json ``` For a SLURM cluster without a container, the same venv must be reachable from every compute node (e.g. on a shared filesystem): ```bash cd code bash run_all.sh infrastructure_config/slurm.json ``` This is the least portable option -- Apptainer (mode c) avoids the "same venv on every node" requirement entirely. --- ## (b) With Docker (local machine only -- Docker is not used on the SLURM cluster, see mode c) Build once (or directly load the pre-compiled image: `reproducible-env.tar.gz`): ```bash bash build_docker.sh ``` Run: ```bash cd docker make run # runs run_all.sh infrastructure_config/local.json inside the container ``` or alternatively: ```bash docker compose -f docker/docker-compose.yml run --rm reproducible-env bash run_all.sh infrastructure_config/local.json ``` --- ## (c) With Apptainer (single machine or SLURM -- this is the mode used on HPC) ### Build the `.sif` (or directly load the pre-compiled image: `reproducible-env.sif`) ```bash apptainer build reproducible-env.sif docker-daemon://reproducible-env ``` ### Run on a single machine ```bash apptainer exec --pwd /workspace \ --bind packetdata:/workspace/packetdata:ro \ --bind results:/workspace/results \ reproducible-env.sif bash run_all.sh infrastructure_config/local.json ``` `--pwd /workspace` is required. ### Run distributed on the SLURM cluster 1. Copy `reproducible-env.sif` to the cluster, and edit your local copy of `run_in_slurm/slurm.json` with your partition (`queue`) and, under `"container"`, the absolute paths the `.sif` and `packetdata`/`results` will have on the cluster filesystem. 2. Edit the placeholders in `run_in_slurm/submit_manager_distrib.sh` (`YOUR_EMAIL@example.com`, `--partition`) to match your account/cluster. 3. Export `BASE` -- the root path of your copy of this project on the cluster filesystem -- and submit: ```bash export BASE=/path/to/your/copy/of/this/project cd run_in_slurm sbatch submit_manager_distrib.sh ``` The manager **and** every Dask/SLURM worker run inside the same `reproducible-env.sif` -- the `"container"` block in `slurm.json` is what wires the workers to it (see `build_client()` / `build_cluster()` in the code). `slurm.json` is bind-mounted at run time (not baked into the image), so editing it never requires rebuilding the `.sif`. **Troubleshooting:** if the job fails with `sbatch: command not found` or a `munge` authentication error, the container needs explicit access to the host's SLURM client and `munge` socket. Bind them in `submit_manager_distrib.sh` (paths depend on the cluster's SLURM installation): ```bash --bind /opt/slurm:/opt/slurm \ --bind /etc/slurm:/etc/slurm \ --bind /var/run/munge:/var/run/munge \ --env PATH=/opt/slurm/bin:$PATH ``` Useful SLURM commands while a job is running: ```bash squeue -u $USER # your jobs scontrol show job # details on one job watch -n 60 squeue -u $USER # auto-refresh every 60s ``` Note: `sbatch` jobs keep running after you log out, so `tmux` is not required for the workflow above. It's only useful for interactive work outside `sbatch` (e.g. debugging mode (a) directly, or leaving a `watch squeue` running): `tmux new -s session_name` to start, `Ctrl+B` then `d` to detach, `tmux attach -t session_name` to reattach. --- ## Output All results (CSV files, logs, temporary Dask worker files) are written under `results/` -- this is the only writable path inside the containers, bind-mounted in all three modes. ## Real data `packetdata/` contains a parquet version of the network traffic packet data provided in: Brenner, B., Fabini, J., Offermanns, M., Semper, S., and Zseby, T. (2024). *Malware communication in smart factories: A network traffic data set*. Computer Networks, 255, 110804. https://doi.org/10.1016/j.comnet.2024.110804 Associated dataset: Brenner, B., Fabini, J., Offermanns, M., Semper, S., and Zseby, T. (2025). *Dataset of Publication "Malware Communication in Smart Factories: A Network Traffic Data Set"*. TU Wien. https://doi.org/10.48436/ghdc6-45k78 --- ## Tree and Files . ├── build_docker.sh ├── code │   ├── comparison_tests.py │   ├── datagen.py │   ├── fast_metrics.py │   ├── infrastructure_config │   │   └── local.json │   ├── requirements.txt │   ├── run_all.sh │   ├── scaling.py │   ├── sdobase.py │   ├── sdoclust │   │   └── clust.py │   ├── stream_config │   │   ├── stream_ad_iforest-frozen.json │   │   ├── stream_ad_iforest.json │   │   ├── stream_ad_kmeans_ddask-frozen.json │   │   ├── stream_ad_kmeans_ddask.json │   │   ├── stream_ad_kmeans_ndask-frozen.json │   │   ├── stream_ad_kmeans_ndask.json │   │   ├── stream_ad_sdo_dask.json │   │   ├── stream_ad_sdo-frozen_dask.json │   │   ├── stream_ad_sdo-frozen_numpy.json │   │   ├── stream_ad_sdo_numpy.json │   │   ├── stream_clust_fast-hdbscan.json │   │   ├── stream_clust_kmeans_dask.json │   │   ├── stream_clust_mbkmeans.json │   │   ├── stream_clust_sdoclust_dask.json │   │   └── stream_clust_sdoclust_numpy.json │   └── stream_test.py ├── docker │   ├── docker-compose.yml │   ├── Dockerfile │   └── Makefile ├── docs │   ├── comparison_tests.md │   ├── scaling.md │   ├── sdo.md │   └── stream_test.md ├── packetdata │   ├── ... ├── README.md ├── reproducible-env.sif ├── reproducible-env.tar.gz ├── results ├── results_local_8cores_160GbRAM_Aug2026.zip ├── results_SlurmCluster_16workers_Aug2026.zip ├── run_in_slurm    ├── slurm.json    └── submit_manager_distrib.sh | File / directory | Description | | ------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | | `build_docker.sh` | Script for building the Docker image used to reproduce the software environment. | | `code/` | Main source code, experiment scripts, and configuration files. | | `code/comparison_tests.py` | Runs comparative experiments between the implemented methods and reference approaches. | | `code/datagen.py` | Synthetic dataset generation and dataset storage utilities. | | `code/fast_metrics.py` | Efficient implementations and utilities for evaluating clustering and anomaly detection results. | | `code/infrastructure_config/` | Infrastructure-specific execution configurations. | | `code/infrastructure_config/local.json` | Configuration for local execution using a local Dask cluster. | | `code/requirements.txt` | Python package dependencies required to reproduce the experiments. | | `code/run_all.sh` | Main execution script that runs all stream analysis, scaling, and comparison experiments. | | `code/scaling.py` | Performs strong and weak scaling experiments. | | `code/sdobase.py` | Core implementation and utilities for the SDO-based methods. | | `code/sdoclust/` | Package containing the clustering implementation. | | `code/sdoclust/clust.py` | Implementation of the SDO-based clustering algorithm. | | `code/stream_config/` | Configuration files defining the different streaming clustering and anomaly detection experiments. | | `code/stream_config/stream_ad_iforest-frozen.json` | Configuration for the frozen Isolation Forest anomaly detection experiment. | | `code/stream_config/stream_ad_iforest.json` | Configuration for the Isolation Forest anomaly detection experiment. | | `code/stream_config/stream_ad_kmeans_ddask-frozen.json` | Configuration for the frozen distributed Dask-based K-Means anomaly detection experiment. | | `code/stream_config/stream_ad_kmeans_ddask.json` | Configuration for the distributed Dask-based K-Means anomaly detection experiment. | | `code/stream_config/stream_ad_kmeans_ndask-frozen.json` | Configuration for the frozen non-distributed K-Means anomaly detection experiment using the Dask backend. | | `code/stream_config/stream_ad_kmeans_ndask.json` | Configuration for the non-distributed K-Means anomaly detection experiment using the Dask backend. | | `code/stream_config/stream_ad_sdo_dask.json` | Configuration for the Dask-based SDO anomaly detection experiment. | | `code/stream_config/stream_ad_sdo-frozen_dask.json` | Configuration for the frozen Dask-based SDO anomaly detection experiment. | | `code/stream_config/stream_ad_sdo-frozen_numpy.json` | Configuration for the frozen NumPy-based SDO anomaly detection experiment. | | `code/stream_config/stream_ad_sdo_numpy.json` | Configuration for the NumPy-based SDO anomaly detection experiment. | | `code/stream_config/stream_clust_fast-hdbscan.json` | Configuration for the Fast-HDBSCAN streaming clustering experiment. | | `code/stream_config/stream_clust_kmeans_dask.json` | Configuration for the Dask-based K-Means streaming clustering experiment. | | `code/stream_config/stream_clust_mbkmeans.json` | Configuration for the MiniBatch K-Means streaming clustering experiment. | | `code/stream_config/stream_clust_sdoclust_dask.json` | Configuration for the Dask-based SDOclust streaming clustering experiment. | | `code/stream_config/stream_clust_sdoclust_numpy.json` | Configuration for the NumPy-based SDOclust streaming clustering experiment. | | `code/stream_test.py` | Main script for executing configurable streaming clustering and anomaly detection experiments. | | `docker/` | Docker and Docker Compose files used to create and run the reproducible software environment locally. | | `docker/docker-compose.yml` | Docker Compose configuration for local execution, including data and results directory mounts. | | `docker/Dockerfile` | Definition of the reproducible Python 3.11.2 software environment and its dependencies. | | `docker/Makefile` | Convenience commands for building, running, and accessing the Docker environment. | | `docs/` | Documentation describing the experimental scripts and implemented methods. | | `docs/comparison_tests.md` | Documentation for the comparative experiments. | | `docs/scaling.md` | Documentation for the strong and weak scaling experiments. | | `docs/sdo.md` | Documentation of the SDO-based methodology and implementation. | | `docs/stream_test.md` | Documentation for the streaming experiments and their configuration. | | `packetdata/` | Input packet datasets used in the streaming experiments. | | `README.md` | General repository documentation, installation instructions, and reproduction workflow. | | `reproducible-env.sif` | Apptainer/Singularity image containing the reproducible software environment for HPC and SLURM execution. | | `reproducible-env.tar.gz` | Docker image archive containing the reproducible software environment. | | `results/` | Output directory where newly generated experimental results are stored. | | `results_local_8cores_160GbRAM_Aug2026.zip` | Archived results obtained from local execution on a machine with 8 CPU cores and 160 GB RAM. | | `results_SlurmCluster_16workers_Aug2026.zip` | Archived results obtained from distributed execution on a SLURM cluster using 16 workers. | | `run_in_slurm/` | Files required for reproducing the distributed experiments on a SLURM cluster. | | `run_in_slurm/slurm.json` | SLURM/Dask infrastructure configuration used for distributed execution. | | `run_in_slurm/submit_manager_distrib.sh` | SLURM submission script that launches the distributed experimental workflow. |