NTCIR19-Lifelog Task

CASTLE 2024 Dataset

The CASTLE 2024 dataset is a large-scale, multimodal dataset designed for advancing research in lifelogging, human activity recognition, and multimodal retrieval. It provides a rich collection of time-aligned sensor and video data for analysis and benchmarking.

Dataset Link: https://huggingface.co/CASTLE-Dataset. Two repositories are available:

• CASTLE-Dataset/CASTLE2024: the full multimodal dataset (video, audio, sensor streams).
• CASTLE-Dataset/CASTLE2024_Audio_only: an audio-only subset, useful for the audio-based CSAT topics (e.g., laughing, singing, complimenting the food).

Characteristics:

• Captured over four days in a controlled environment
• 10 participants engaged in natural activities
• 15 video streams (10 egocentric, 5 static perspectives)
• Over 600 hours of UHD 50fps video with audio
• Includes 6DoF IMU, GPS, and biometric data, plus automatically generated speech transcripts
• 8.22 TB total size

Supported Sub-tasks: CSAT (CASTLE Semantic Access Task), CAST-Seg (Conversation Segmentation), Recipe Generation

For CAST-Seg, formal-run segmentations submitted by participants will be assessor-validated and released as a community benchmark for future conversation-segmentation research. See the Formal Run page for details.

Lifelog Dataset (LSC'22-25 Dataset)

NTCIR19-Lifelog reuses the same dataset as NTCIR18-Lifelog6: the LSC'22-25 dataset, which is a multimodal dataset that spans 18 months from one active lifelogger. This comprehensive dataset supports research on semantic access, recipe generation, and personal information retrieval.

Dataset Access: Available after registration and completion of required agreement forms. Please email ntcir.lifelog.dcu@gmail.com with completed agreement forms as described below.

Dataset Components: The dataset consists of three password-protected files:

• Core Image Dataset: 18 months of wearable camera images, fully redacted and anonymized in 1024 x 768 resolution, captured using a Narrative Clip device. These images were collected during 2019-2020. All faces and readable text have been removed, as well as certain scenes and activities manually filtered out to respect local privacy requirements.
• Metadata: Textual metadata representing time, locations, and other contextual information for the collection.
• Visual Concepts: Extracted from the non-redacted version of the visual dataset, including:
  • attribute_top{i}: The attribute of the scene detected automatically from the image.
  • category_top{i}: The category of the scene detected automatically from the image.
  • category_top{i}_score: The confidence score of the scene prediction output.
  • concept_class{i}: Objects detected automatically from the image (using the object category list of 2014-2017 COCO datasets with 80 labels).
  • concept_score_top{i}: The confidence score of the object detection output.
  • concept_bbox_top{i}: The bounding box of the detected object in the format of {top_x top_y bottom_x bottom_y}.

Supported Sub-tasks: LSAT (Lifelog Semantic Access Task)

Access Requirements: NTCIR-Lifelog participants are required to sign two forms to access the datasets:

1. Organisation Agreement Form: To be signed by the organisation to which the participants belong. This form must be signed and sent by email to NTCIR-Lifelog organisers (ntcir.lifelog.dcu@gmail.com) in PDF format.
   Download Organisation Agreement Form (PDF)
2. Individual Agreement Form: To be signed by each individual researcher wishing to use the NTCIR-Lifelog data collection. This form must be filed by the participating organisation, but it does not need to be sent to the lifelog organisers unless requested at a later date.
   Download Individual Agreement Form (PDF)

Upon completion of this process, participants will be sent details of how to access the dataset. Please note that participants are also expected to register on the NTCIR-19 Website in order to participate.