NTCIR19-Lifelog Task

Lifelog Subtasks

LSAT - Lifelog Semantic Access subTask

The Lifelog Semantic Access Task (LSAT) is a known-item search task that can be undertaken in an interactive or automatic manner. A text, visual, or multimodal query is given, and participants must retrieve specific moments based on the query in a lifelogger's life. Moments are defined as semantic events or activities.

Task Characteristics:

• Query Types: Textual, visual, or multimodal queries
• Retrieval Target: Specific moments (semantic events or activities) in a lifelogger's life
• Modes: Interactive or automatic
• Topic Types: Known-item search (KIS) and Ad-hoc queries (AD)
• Dataset: LSC'22-25 dataset (same as NTCIR18-Lifelog6)

Evaluation: LSAT uses post-evaluation methodology with precision, recall, and other standard information retrieval metrics. Relevance judgments are created through a pooled approach with binary relevance assessments.

Dry Run Topics: Available for testing. See the Dry Run Topics page for details.

CASTLE Subtasks

CSAT - CASTLE Semantic Access subTask

The CASTLE Semantic Access Task (CSAT) is a search-based task where participants retrieve key interactions or events from multi-modal data (e.g., video, audio, and speech transcripts) based on a semantic query. This task focuses on understanding and retrieving meaningful events from collaborative session data.

Task Characteristics:

• Data Modalities: Video, audio, speech transcripts, and other multimodal collaborative session data (plus 6DoF IMU, GPS, and biometric data)
• Retrieval Target: Key interactions or events from collaborative sessions
• Query Type: Semantic queries describing activities, interactions, or events
• Dataset: CASTLE dataset (available on Hugging Face)

Evaluation: CSAT evaluation focuses on the accuracy of retrieving relevant interactions or events based on semantic understanding of the query and multimodal data.

Dry Run Topics: Available for testing. See the Dry Run Topics page for details.

CAST-Seg - CASTLE Conversation Segmentation subTask

The CASTLE Conversation Segmentation subTask (CAST-Seg) requires participants to segment collaborative sessions into meaningful conversational units, such as tasks, discussions, or decisions, to enable better analysis and retrieval.

Task Characteristics:

• Objective: Identify meaningful boundaries in collaborative sessions
• Segmentation Units: Tasks, discussions, decisions, or other meaningful conversational units
• Data: Multimodal collaborative session data (video, audio, speech transcripts)
• Dataset: CASTLE dataset

Evaluation: CAST-Seg evaluation assesses the accuracy of segmentation boundaries and the meaningfulness of identified conversational units. There is no pre-defined ground truth; human assessors review each submission for coherence. Validated formal-run segmentations will be pooled and released as a community benchmark for future research on long-form conversation segmentation over the CASTLE 2024 recordings.

Why submit: CAST-Seg has no query topics to prepare; you only segment the sessions, and can tackle the task with transcripts or audio alone. Each submission helps build a shared reference dataset for the research community and gives your group visibility in the benchmark release. Up to 10 runs per group are accepted.

Note: CAST-Seg does not have dry run topics. Participants should refer to the Formal Run, Submission, and Evaluation pages for full guidelines.

Recipe Generation

The Recipe Generation subtask challenges participants to generate personal recipes or cooking procedures using multimodal collaborative session data from the CASTLE dataset. This task focuses on understanding cooking activities, meal planning, and recipe creation from collaborative cooking sessions.

Task Characteristics:

• Input Data: Multimodal CASTLE dataset including video, audio, speech transcripts, and collaborative session data from cooking activities
• Data Sources: The CASTLE dataset includes menu schedules showing meal assignments and a Castle Cookbook containing recipes for various dishes (e.g., Irish Breakfast, Pineapple Fried Rice, Bò Kho, Lẩu Gà, Italian Spinach Lasagne, Pumpkin Risotto, Tagliatelle, Chicken Curry, Zürchergeschnetzeltes & Rösti, Cookies, Shepherds Pie)
• Output: Personal recipes or cooking procedures generated from the collaborative session data
• Approach: Creative modelling of lifestyle-driven meal preparation based on multimodal collaborative cooking sessions
• Dataset: CASTLE dataset (multimodal collaborative session data including cooking activities, menu schedules, and recipe information)

Evaluation: Recipe Generation evaluation assesses the quality, relevance, and creativity of generated recipes based on the provided multimodal collaborative session data from the CASTLE dataset.

Note: Recipe Generation does not have dry run topics. Participants should refer to task guidelines and dataset documentation. The CASTLE dataset includes menu schedules and cookbook information that can be used as reference for recipe generation.