Dry Run Topics
Dry run topics are sample topics provided to help participants familiarize themselves with the task format and evaluation procedures. These topics are designed to be representative of the actual test topics and can be used for system development and testing. Note: There is no formal dry run phase or submission period.
Note: Dry run topics are only available for semantic access tasks (LSAT for Lifelog and CSAT for CASTLE). Other tasks such as CAST-Seg and Recipe Generation do not have dry run topics.
Lifelog Semantic Access Task (LSAT) Dry Run Topics
Dry run topics for the Lifelog Semantic Access Task (LSAT) are provided to help participants test their systems and understand the expected output format before submitting formal runs.
Dry Run Topics File: The Lifelog dry-run topics are available in XML format: dry-run-topics.xml
The dry-run topics include 26 topics covering two types:
- Known-Item Search (KIS): 13 topics (KIS001-KIS013) - Topics with one (or few) relevant moments in the collection.
- Ad-hoc Queries (AD): 13 topics (AD001-AD013) - Topics that may have many moments in the collection that are relevant.
Topic Format: Each topic in the XML file includes:
- ID: Unique topic identifier
- Type: knownitem or adhoc
- Title: Brief topic title
- Description: Short description of what to find
- Narrative: Detailed explanation of relevance criteria and what should/should not be considered relevant
Availability: Dry run topics have been released and are now available. These topics are provided for testing purposes only and do not require formal submission.
CASTLE Semantic Access Task (CSAT) Dry Run Topics
Dry run topics for the CASTLE Semantic Access Task (CSAT) are provided to help participants test their systems and understand the expected output format before submitting formal runs.
CSAT Dry Run Topics: Sample CSAT topics for the CASTLE dataset are available in XML format: castle-semantic-topics.xml
The CASTLE semantic topics include 10 topics (CSAT001-CSAT010) covering various semantic queries such as "Find instances of a Checkmate", "Find instances of bringing trash bags out", and other semantic queries.
Topic Format: Each topic in the XML file includes:
- ID: Unique topic identifier (CSAT001-CSAT010)
- Type: semantic
- Title: Brief topic title
- Description: Short description of what to find
- Narrative: Detailed explanation of relevance criteria and what should/should not be considered relevant
Note: The castle-semantic-topics.xml file is specifically for the CASTLE dataset and contains CSAT semantic access topics.
Other Tasks
CAST-Seg and Recipe Generation: These tasks do not have dry run topics. Participants should refer to the task guidelines and dataset documentation for these tasks.
Using Dry Run Topics
Participants are encouraged to use dry run topics to:
- Test and develop their systems
- Understand the expected output format
- Familiarize themselves with the evaluation procedures
- Identify and fix potential issues before formal submissions
Note: Dry run topics are provided for testing and development purposes only. There is no formal submission or evaluation for dry run topics. Participants are encouraged to use these topics to test their systems and understand the expected output format before submitting formal runs.
Access: Dry run topics are now available. Download the topic files directly from this page or access them through the task registration portal.