Home Communities My dashboard

We are currently working on a new workflow for the record publication reviews! 🕵️📑
Are you interested? Please try it on this test system and give us some feedback!

Published February 27, 2024 | Version 1.0.0
Other Open

MYFix: Automated Fixation Annotation of Eye-Tracking Videos (Python Code and Sample Data)

Creators

  • 1. ROR icon TU Wien

Contributors

Data collector:

  • 1. TU Wien

Description

How to cite?

Alinaghi, N., Hollendonner, S., & Giannopoulos, I. (2024). MYFix: Automated Fixation Annotation of Eye-Tracking Videos. Sensors, 24(9), 2666.

Abstract

In mobile eye-tracking research, the automatic annotation of fixation points is an important yet difficult task, especially in varied and dynamic environments such as outdoor urban landscapes. This complexity is increased by the constant movement and dynamic nature of both the observer and their environment in urban spaces. This paper presents a novel approach that integrates the capabilities of two foundation models, YOLOv8 and Mask2Former, as a pipeline to automatically annotate fixation points without requiring additional training or fine-tuning. Our pipeline leverages YOLO’s extensive training on the MS COCO dataset for object detection and Mask2Former’s training on the Cityscapes dataset for semantic segmentation. This integration not only streamlines the annotation process but also improves accuracy and consistency, ensuring reliable annotations, even in complex scenes with multiple objects side by side or at different depths. Validation through two experiments showcases its efficiency, achieving 89.05% accuracy in a controlled data collection and 81.50% accuracy in a real-world outdoor wayfinding scenario. With an average runtime per frame of 1.61 ± 0.35 s, our approach stands as a robust solution for automatic fixation annotation.

How to use the code?

To create the environment execute the following command:  

`conda env create -f environment.yml`

Dependencies (which migth require extra attention):

- torch: pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

- YOLO: conda install -c conda-forge ultralytics

- transformers: conda install -c conda-forge transformers

 

To avoid problems with torch and numpy, install torch (which includes numpy anyway) first.

To annotate your eye-tracking data automatically, execute the `annotate.ipynb` file, and reference the data folder in the variable `base_path`, as well as the name of the gaze-data-file in `gaze_file` and the video-file in `video_file`. The following folder structure is mandatory:

```bash

.base_path

├── gaze_file              

└── video_file

````

The execution of the script will create files in the following folder structure:

```bash

.base_path

├── extracted_frames

│   ├── frame_0.jpg

│   ├── frame_3.jpg

│   └── ...

│

├── outputs

│   ├── saved_frames_semSeg_yolo

│   │   ├── frame_0.jpg

│   │   ├── frame_3.jpg

│   │   └── ...

│   │

│   ├── confusion_matrix.png  

│   ├── labeled_data_semSeg_yolo.csv

│   └── stitched_video.mp4

│

├── video_file

├── fixation_gaze_positions.csv

├── saccades_gaze_positions.csv

└── gaze_file

```

To receive evealuation results, a manual labeling for each frame is necessary. If no manual labeling is provided the script will return an error. Enter this information in the file `labeled_data_semSeg_yolo.csv` in the column `manual`.

Files

submission.zip

Files (683.7 MiB)

Name Size
md5:1369e8ff9dec2fd61e181148846d35c1
683.7 MiB Preview Download

Additional details

Dates

Submitted
2024-02-27