# Scaling Tests for Distributed Clustering

This repository includes a scalability benchmarking script for the SDOclust algorithm under distributed execution with Dask.
The main script is `scaling.py`, which evaluates weak and strong scaling behavior, resource usage, and clustering quality.

> **How to run this** (locally, with Docker, or with Apptainer/SLURM): see the root `README.md`. This document only covers what the script measures and how results are structured.

Aug 2026

---

## 1) Overview

The script evaluates SDOclust under different parallel configurations:

* Number of workers (`p`)
* Scaling mode (`weak` vs `strong`)
* Dataset size (`N`)
* Fixed dataset characteristics (dimensionality, clusters, outliers)

Metrics:

| Category           | Metrics                   |
| ------------------ | -------------------------- |
| Performance        | Fit time, prediction time |
| Clustering quality | ARI, AMI                  |
| Resources          | Peak memory per worker    |
| Parallel scaling   | Speedup, efficiency       |

---

## 2) Experimental Design

### Fixed parameters

| Parameter      | Value       |
| -------------- | ----------- |
| `N_ref`        | 100000      |
| `d`            | 25          |
| `k`            | 15          |
| `outlier_frac` | 0.01        |
| `dataset_type` | cardinality |
| `rseed`        | 42          |

### Worker configurations

```python
workers_list = [1, 2, 4, 8, 16]
```

### Scaling modes

#### Weak scaling

Dataset size increases with the number of workers:

```text
N_total = N_ref × p
```

#### Strong scaling

Dataset size is fixed:

```text
N_total = N_ref × 10
```

A fixed reference sample (`N_ref`) is used for model training.

---

## 3) Dataset Generation

Datasets are generated using `datagen.generate_dataset`, stored temporarily as Parquet files under `results/`, and deleted after each experiment.

---

## 4) Distributed Execution

The number of workers per iteration (`workers_list`) is combined with the rest of the cluster infrastructure (queue, cores, memory, walltime...) from the execution config passed on the command line — see the root README for `local.json` vs `slurm.json`.

The dataset is loaded as a Dask array:

```python
X_dask = df.drop("label", axis=1).to_dask_array(lengths=True)
y = df["label"].compute()
```

The model is trained on a reference sample and applied to the full dataset:

```python
model = sdo.SDOclust(backend="dask").fit(Xs)
labels = model.predict(X_dask)
```

---

## 5) Output Files

Results are stored in the `results/` folder:

| File                 | Description             |
| -------------------- | ------------------------ |
| `weak_scaling.csv`   | Weak scaling results   |
| `strong_scaling.csv` | Strong scaling results |

### Example output columns

| Column          | Meaning                    |
| --------------- | --------------------------- |
| scaling         | `weak` / `strong`          |
| workers         | Number of workers          |
| N               | Total dataset size         |
| N_per_worker    | Samples per worker         |
| time_fit        | Training time               |
| time_predict    | Prediction time            |
| peak_mem_mb     | Peak memory per worker     |
| ARI / AMI       | Clustering quality         |
| speedup         | Strong scaling speedup     |
| efficiency      | Strong scaling efficiency  |
| weak_efficiency | Weak scaling efficiency    |

---

## 6) Reproducibility

* Random seeds are controlled via `rseed`
* A fixed-size sample (`N_ref`) ensures comparable training cost across scaling configurations.
* Thread counts are fixed via environment variables:

```python
OMP_NUM_THREADS=1
MKL_NUM_THREADS=1
OPENBLAS_NUM_THREADS=1
```

---

## 7) Notes on Interpretation

* Weak scaling: prediction time should remain approximately constant as `p` increases.
* Strong scaling: prediction time should decrease with diminishing returns.
* ARI/AMI should remain stable across configurations.