# SDO & SDOclust

**Scalable Anomaly Detection and Clustering Based on Sparse Data Observers**

## 1. Overview

This module provides two main components:

* **SDO**: Scalable outlier detection based on observers.
* **SDOclust**: Clustering built on top of SDO.

Both algorithms support two execution backends:

* `numpy`: in-memory computation (joblib).
* `dask`: scalable distributed.

Check the official repository for updated information: [https://github.com/CN-TU/parallel-SDO](https://github.com/CN-TU/parallel-SDO)

---

## 2. Core Idea

* SDO works by selecting a subset of representative points called **observers**.
* Observers approximate the data distribution, i.e. coreset (low-density model)
* Outliers are points far from observers.
* Clusters emerge from the structure of observers taken as a graph (Connected Components).

---

## 3. Installation & Dependencies

Required:

* numpy
* scipy
* scikit-learn
* joblib

Optional:

* dask (for distributed backend)
* faiss or pynndescent (for fast nearest neighbors)

---

## 4. Quick Start

### 4.1 Outlier Detection

        # NumPy (joblib)        
        from sdobase import SDO
        model = SDO()
        # X is a NumPy array
        scores = model.fit_predict(X) 
    
        # Dask (joblib only for fit)        
        from sdobase import SDO
        model = SDO(backend="dask")
        # X is a NumPy array, Dask array o Dask dataframe
        scores = model.fit_predict(X) 

---

### 4.2 Clustering 

        # NumPy (joblib)        
        from sdobase import SDOclust
        model = SDOclust()
        # X1 and X2 are NumPy arrays
        labels_X2 = model.fit_predict(X1) 
        labels_X1 = model.update_predict(X2) 
            
        # Dask (joblib only for fit and update)        
        from sdobase import SDOclust
        model = SDOclust(backend="dask")
        # X1 and X2 are a NumPy arrays, Dask arrays o Dask dataframes
        labels_X2 = model.fit_predict(X1) 
        labels_X1 = model.update_predict(X2) 

---

### 4.2 Backends, methods, inputs and algorithms 

| Backend    | Method to call   | X is                |  Algorithm |
| ---------- | ---------------- | ------------------- | ---------- |
| numpy/dask | fit(X)           | numpy array         | SDO/SDOclust |
| dask       | fit_from_dask(X) | dask array/df       | SDO/SDOclust |
| numpy/dask | predict(X)       | numpy/dask array/df | SDO/SDOclust |
| numpy/dask | fit_predict(X)   | numpy/dask array/df | SDO/SDOclust |
| numpy/dask | update(X)        | numpy array         | SDOclust |
| dask       | update_from_dask(X) | dask array/df    | SDOclust |
| numpy/dask | update_predict(X)| numpy/dask array/df | SDOclust |

---

## 5. NumPy vs Dask backends

Use Numpy backend when...

* Dataset fits in memory.
* Maximum speed is required.
* Simplicity is preferred.

Use Dask backend when...

* Dataset is very large.
* Data is already a Dask array or DataFrame.
* Distributed computation is needed.

---

## 6. Main Parameters

### 6.1 SDO 

| Parameter   | Meaning                        | Effect                                |
| ----------- | ------------------------------ | ------------------------------------- |
| `x`         | number of nearest observers    | locality of the score                 |
| `qv`        | quantile for pruning observers | model sparsity                        |
| `k`         | initial number of observers    | model size                            |
| `chunksize` | block size                     | memory/performance trade-off          |
| `method`    | neighbor search method         | `"brute"`, `"faiss"`, `"pynndescent"` |

#### Practical Guidelines

* Default parameters work well in most cases
* Increase `x` → smoother scores, less sensitivity to noise.
* Increase `qv` → fewer observers, faster but less precise.
* Increase `k` → more stable model, higher cost.

---


### 6.2 SDOclust

The same as in SDO, and additionally...

| Parameter     | Meaning                                        |
| ------------- | ---------------------------------------------- |
| `zeta`        | local-global cutoff threshold weight           |
| `chi`         | external adjustment of `chi`                   |
| `chi_min`     | absolute minimum `chi` when self-tuned         |
| `chi_prop`    | factor that sets `chi` proportional to the observers model size |
| `xc`          | observers used for label propagation           |
| `cons_factor` | fraction of observers preserved during updates |
| `e`           | minimum cluster size (of observers)            |

#### Practical Guidelines

* Default parameters work well in most cases
* `chi` defines the neighborhood size when clustering the observers space. If not externaly given, it's set as the maximum between `chi_min` and `chi_prop`*O.shape[0]
* Increase `chi`, `chi_min` or `chi_prop` → bigger clusters, less fragmented, mode dense.
* Decrease `chi`, `chi_min` or `chi_prop` → smaller clusters, more fragmented, more noise-sensitive.
* High `zeta` → local criteria: better when clusters show different densities.
* Low `zeta` → global criteria: better for avoiding close clusters merge and clustering noise.
* Increase `xc` → smoother cluster boundaries.
* Increase `cons_factor` → more inertia/persistence, less drift/adaptivity in the model.

---