Parallel SDO and SDOclust: Evaluation Experiments

Parallel SDO and SDOclust: Evaluation Experiments

Context and methodology

This repository accompanies the paper Closing the Effectiveness–Scalability Gap in Distributed Unsupervised Learning and contains the experimental data and results used to evaluate parallel versions of SDO and SDOclust (and alternatives) in anomaly detection and clustering under controlled conditions. Experiments include: comparisons under different scaling factors (cardinality, dimensionality, number of clusters, and rate of anomalies) with synthetic data, strong and weak scaling experiments, and tests with real network data in a stream data large-batch incremental setting.

Closing the Effectiveness–Scalability Gap in Distributed Unsupervised Learning, by Félix Iglesias, Julian Lampert, Tanja Zseby, and Arthur Zimek. Currently under review

Experiments are prepared for running in both local and distributed environment. They were tested in:

1. SLURM cluster. A coordinator job (1 node, 2 CPUs, 32 GB RAM, 24 h walltime) orchestrated the experiments and instantiated the Dask scheduler/client. Dask-based methods provisioned their workers dynamically via `dask_jobqueue`: 16 independent single-core SLURM jobs were requested, each with 1 CPU, 16 GB RAM, and 24 h walltime. The resulting 16 workers provided 16 worker cores in total and were scheduled by SLURM across the available physical nodes.
2. Local machine. Intel Xeon E5-2620 v4 CPU (8 physical cores, 16 threads) with 160 GB RAM. All methods were executed on CPU; distributed variants used a local Dask cluster to emulate distributed execution.

Technical details

Real datasets are stored in the `packetdata/` directory. Eperimental results are provided in the `results_SlurmCluster_16workers_Aug2026.zip` and `results_local_8cores_160GbRAM_Aug2026.zip` files. The artifact includes Python scripts for data generation and experiment execution, together with configuration and metadata files.

The experiments require Python 3.9.6 and the dependencies listed in `code/requirements.txt`. A pre-built Docker and Apptainer images and the corresponding configurations are also provided to facilitate reproducibility.

Updated parallel SDO and SDOclust codes can be found in: https://github.com/CN-TU/parallel-SDO

Further details

See the `README.md` (within `scripts.zip`) for detailed instructions on how to reproduce the experiments.

Artifact packed files

•  code_and_docs.zip:  Source code, scripts, and documentation required to run and analyze the project.

• packetdata.zip:  Input network packet data used in the experiments.

• reproducible-env.sif`: Apptainer image containing the software environment required to reproduce the experiments. 

• reproducible-env.tar.gz: Compressed archive of the reproducible software environment and its required dependencies.

• results_SlurmCluster_16workers_Aug2026.zip: Experimental results obtained on a Slurm cluster using 16 workers in August 2026.

• results_local_8cores_160GbRAM_Aug2026.zip: Experimental results obtained in a local environment with 8 CPU cores and 160 GB of RAM in August 2026. 

• FIRST_OF_ALL.txt: File containing initial instructions and important information for reproducing the experiments.

• third_party_LICENSE.md:  License and attribution information for Malware Communication in Smart Factories: A Network Traffic Data Set (CC BY 4.0)

Licenses

• The Malware Communication in Smart Factories: A Network Traffic Data Set dataset is licensed under CC BY 4.0 
• The results files (contained in `results_SlurmCluster_16workers_Aug2026.zip` and `results_local_8cores_160GbRAM_Aug2026.zip`) are licensed under CC BY 4.0 
• The rest of material and software files are licensed under MIT