ORCAS-I

Description

ORCAS-I is an annotated version of ORCAS dataset (Craswell et al., 2020) annotated with user intents using weak supervision. It allows you to train your algorithm on various types of user intents. Those intents are initially taken from Broder's (2002) classification: informational, navigational and transactional. We also refined this classification and added two subcategories inside the informational category: factual and instrumental. If the intent did not get any label inside the informational category it was classified as abstain.
 

ORCAS-I consists of the following files:

a) ORCAS-I-18M.tsv

A complete ORCAS data set which contains 18 million unique query-urls pairs.

dataset size: 18,823,602
unique queries: 10,405,339
unique URLs: 1,422,029
unique domains: 241,199

b) ORCAS-I-2M.tsv

A 2M subset of ORCAS-I-18M.tsv that we used for our experiments with different machine learning algorithms.

dataset size: 2,000,000
unique queries: 1,796,652
unique URLs: 618,679
unique domains: 126,001

Both ORCAS-I-18M and ORCAS-I-2M contain the following columns:

1. qid: the id of the query
2. query: the text of the query
3. url: the url that the user clicked
4. did: the document from TREC deep learning track that the url leads to
5. level_1: first level of annotation which has three top level categories: informational, navigational and transactional
6. level_2: second level of annotation (only classifies according to factual and instrumental categories, so all the other intents in the column are classified as abstain)
7. label: final intent label. Provides the annotation for informational, navigational and transactional categories and also for factual, instrumental and abstain subcategories
8. data_split: either 'train' or 'validation' that corresponds to split used during the original experiments

You can train your classifier either on the 3 top level categories (column 'level_1') or on the full taxonomy (column 'label').

c) ORCAS-I-gold.tsv

This is a test file that contains 1000 randomly selected queries from the full dataset (they are excluded from the 2M sample). These queries were manually annotated by two IR specialists. 

dataset size: 1,000
unique queries: 1,000
unique URLs: 995
unique domains: 700

ORCAS-I-gold contains the following columns:

1. qid: the id of the query
2. query: the text of the query
3. url: the url that the user clicked
4. did: the document from TREC deep learning track that the url leads to
5. label_manual - manually annotated intent
6. data_split: always equal to 'test'

Files

Additional details