Behavioral recordings

Example datasets

Datasets containing behavioral data can be found in the BIDS examples repository and can be used as helpful guidance when curating new datasets.

Template:

sub-<label>/
    [ses-<label>/]
        beh/
            sub-<label>[_ses-<label>]_task-<label>[_acq-<label>][_run-<index>]_beh.json
            sub-<label>[_ses-<label>]_task-<label>[_acq-<label>][_run-<index>]_beh.tsv
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_split-<index>][_recording-<label>]_audio.flac
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_split-<index>][_recording-<label>]_audio.json
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_split-<index>][_recording-<label>]_audio.mp3
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_split-<index>][_recording-<label>]_audio.ogg
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_split-<index>][_recording-<label>]_audio.wav
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_split-<index>][_recording-<label>]_video.avi
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_split-<index>][_recording-<label>]_video.json
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_split-<index>][_recording-<label>]_video.mkv
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_split-<index>][_recording-<label>]_video.mp4
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_split-<index>][_recording-<label>]_audiovideo.avi
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_split-<index>][_recording-<label>]_audiovideo.json
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_split-<index>][_recording-<label>]_audiovideo.mkv
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_split-<index>][_recording-<label>]_audiovideo.mp4
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_recording-<label>]_image.jpg
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_recording-<label>]_image.json
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_recording-<label>]_image.png
            sub-<label>[_ses-<label>]_task-<label>[_acq-<label>][_run-<index>]_events.json
            sub-<label>[_ses-<label>]_task-<label>[_acq-<label>][_run-<index>]_events.tsv
            sub-<label>[_ses-<label>]_task-<label>[_acq-<label>][_run-<index>][_recording-<label>]_physio.json
            sub-<label>[_ses-<label>]_task-<label>[_acq-<label>][_run-<index>][_recording-<label>]_physio.tsv.gz
            sub-<label>[_ses-<label>]_task-<label>[_acq-<label>][_run-<index>][_recording-<label>]_stim.json
            sub-<label>[_ses-<label>]_task-<label>[_acq-<label>][_run-<index>][_recording-<label>]_stim.tsv.gz

Legend:

For more information about filename elements (for example, entities, suffixes, extensions), follow the links embedded in the filename template.
<matches> is a placeholder to denote an arbitrary (and valid) sequence of entities and labels at the beginning of the filename (only BIDS "raw").
<source-entities> is a placeholder to denote an arbitrary sequence of entities and labels at the beginning of the filename matching a source file from which the file derives (only BIDS-Derivatives).
Filename entities or directories between square brackets (for example, [_ses-<label>]) are OPTIONAL.
Some entities may only allow specific values, in which case those values are listed in <>, separated by |.
_<suffix> means that there are several (>6) valid suffixes for this filename pattern.
.<extension> means that there are several (>6) valid extensions for this file type.
[.gz] means that both the unzipped and gzipped versions of the extension are valid.

The beh directory MAY store behavioral recordings such as audio (_audio.*), video (_video.*), combined audio-video (_audiovideo.*), and still image (_image.*) recordings, physiological (_physio.*) recordings, and other continuous recordings (_stim.tsv.gz, _stim.json). Audio, video, audio-video, and image recordings MAY be of subjects performing tasks, resting-state behavior, or recordings of stimuli being presented to the subject. Audio/video recordings MAY occur simultaneously with other recordings, such as BOLD or EEG. Relative timing between files may be determined by consulting the scans.tsv file. If no scans.tsv file is present, the alignment is undefined. The beh directory MAY also contain event timing files (_events.tsv) and their associated metadata (_events.json) for behavioral experiments that do not have corresponding neuroimaging or functional data.

Additionally, events files that do not include the mandatory onset and duration columns MAY be included, but MUST be labeled _beh.tsv rather than _events.tsv.

The following OPTIONAL columns are pre-defined for behavioral data files:

Column name	Requirement Level	Data type	Description
trial_type	OPTIONAL	string	Primary categorisation of each trial to identify them as instances of the experimental conditions. For example: for a response inhibition task, it could take on values `go` and `no-go` to refer to response initiation and response inhibition experimental conditions.
response_time	OPTIONAL	number	Response time measured in seconds. A negative response time can be used to represent preemptive responses and `n/a` denotes a missed response.
HED	OPTIONAL	string	Hierarchical Event Descriptor (HED) tags. See the HED Appendix for details.
stim_file	OPTIONAL	string	Represents the location of the stimulus file (such as an image, video, or audio file) presented at the given onset time. There are no restrictions on the file formats of the stimuli files, but they should be stored in the `/stimuli/` directory (under the root directory of the dataset; with OPTIONAL subdirectories). The values under the `stim_file` column correspond to a path relative to `/stimuli`. For example `images/cat03.jpg` will be translated to `/stimuli/images/cat03.jpg`.
Additional Columns	OPTIONAL	`n/a`	Additional columns are allowed.

Sidecar JSON (`*_beh.json`)

In addition to the metadata that is either:

RECOMMENDED for sidecar JSON files for tabular data, or
REQUIRED for some data that can be found in the beh directory (for example SamplingFrequency and StartTime for *_<physio|stim>.tsv.gz files),

it is RECOMMENDED to add the following metadata to the JSON files of this directory.

Task information

Key name	Requirement Level	Data type	Description
TaskName	RECOMMENDED	string	Name of the task. No two tasks should have the same name. The task label included in the filename MAY be derived from this `"TaskName"` field by removing all non-alphanumeric or `+` characters (that is, all except those matching `[0-9a-zA-Z+]`), and potentially replacing spaces with `+` to ease readability. For example `"TaskName"` `"faces n-back"` or `"head nodding"` could correspond to task labels `faces+n+back` or `facesnback` and `head+nodding` or `headnodding`, respectively.
Instructions	RECOMMENDED	string	Text of the instructions given to participants before the recording.
TaskDescription	RECOMMENDED	string	Longer description of the task.
CogAtlasID	RECOMMENDED	string	URI of the corresponding Cognitive Atlas Task term.
CogPOID	RECOMMENDED	string	URI of the corresponding CogPO term.

Institution information

Key name	Requirement Level	Data type	Description
InstitutionName	RECOMMENDED	string	The name of the institution in charge of the equipment that produced the measurements.
InstitutionAddress	RECOMMENDED	string	The address of the institution in charge of the equipment that produced the measurements.
InstitutionalDepartmentName	RECOMMENDED	string	The department in the institution in charge of the equipment that produced the measurements.

Audio, video, and audio-video recordings and images

Audio and video recordings of behaving subjects MAY be stored in the beh directory using the _audio, _video, and _audiovideo suffixes. The _audio suffix is for audio-only recordings, _video for video-only recordings, and _audiovideo for recordings that contain both audio and video streams. These recordings are typically used to capture vocalizations, speech, facial expressions, body movements, or other behavioral aspects during experimental tasks or rest periods.

Still images captured during behavioral experiments MAY be stored in the beh directory using the _image suffix. These images are typically used for training frames for pose estimation, snapshots of behavioral setups, or individual frames extracted from video recordings.

Privacy and personally identifiable information

Audio and video recordings and images of human subjects often contain personally identifiable information (PII) such as faces, voices, and other identifying features. Data curators MUST take special care to ensure compliance with applicable privacy regulations (such as HIPAA in the United States, GDPR in the European Union, or other local data protection laws) when handling these recordings.

These recordings are generally more suitable for internal use or for sharing non-human subject data, unless appropriate privacy protections are implemented.

File formats

Audio recordings MUST use one of the following extensions:

.flac - Free Lossless Audio Codec
.mp3 - MPEG Audio Layer III
.ogg - Ogg Vorbis
.wav - Waveform Audio File Format

Video and audio-video recordings MUST use one of the following extensions:

.mp4 - MPEG-4 Part 14
.mkv - Matroska video container
.avi - Audio Video Interleave

Image files MUST use one of the following extensions:

.jpg - JPEG image
.png - Portable Network Graphics

Entities

Audio and video files MAY use the following entities:

task - OPTIONAL for audio and video recordings
acq - OPTIONAL, can distinguish different recording setups
run - OPTIONAL, for multiple recordings with identical parameters
recording - OPTIONAL, to differentiate simultaneous recordings from different angles, locations, or devices
split - OPTIONAL, for continuous recordings split into multiple files

Examples

└─ sub-01/
   └─ beh/
      ├─ sub-01_task-rest_video.mp4 
      ├─ sub-01_task-rest_video.json 
      ├─ sub-01_task-interview_audiovideo.mp4 
      ├─ sub-01_task-interview_audiovideo.json 
      ├─ sub-01_task-stroop_recording-face_video.mp4 
      ├─ sub-01_task-stroop_recording-face_video.json 
      ├─ sub-01_task-stroop_recording-room_video.mp4 
      ├─ sub-01_task-stroop_recording-room_video.json 
      ├─ sub-01_task-rest_image.jpg 
      ├─ sub-01_task-rest_image.json 
      ├─ sub-01_task-vocalization_audio.wav 
      └─ sub-01_task-vocalization_audio.json

For continuous recordings split into multiple files:

└─ sub-01/
   └─ ses-01/
      └─ beh/
         ├─ sub-01_ses-01_task-freeplay_run-01_split-001_video.mp4 
         ├─ sub-01_ses-01_task-freeplay_run-01_split-002_video.mp4 
         ├─ sub-01_ses-01_task-freeplay_run-01_split-003_video.mp4 
         └─ sub-01_ses-01_task-freeplay_run-01_video.json

Sidecar JSON for audio, video, audio-video recordings, and images

The following metadata fields are available for audio, video, audio-video recordings, and images:

Key name	Requirement Level	Data type	Description
Device	OPTIONAL	string	Free-form description of the device used to record the data (for example, `"iPhone 12"`, `"Canon EOS R5"`).
DeviceSerialNumber	OPTIONAL	string	The serial number of the equipment that produced the measurements. A pseudonym can also be used to prevent the equipment from being identifiable, so long as each pseudonym is unique within the dataset.
License	OPTIONAL	string	The license for the dataset. The use of license name abbreviations is RECOMMENDED for specifying a license (see Licenses). The corresponding full license text MAY be specified in an additional `LICENSE` file.

Licensing for recordings containing participants

Audio, video, and image recordings of participants may have different licensing restrictions than the main dataset due to privacy considerations. The optional License field can be used to specify different terms for individual recordings that contain identifiable participant data. If not specified, the recording inherits the license from dataset_description.json.

Key name	Requirement Level	Data type	Description
Duration	OPTIONAL	number	Total duration of the audio or video recording in seconds. Must be a number greater than 0.

The following fields are available for audio recordings (_audio) and audio-video recordings (_audiovideo):

Key name	Requirement Level	Data type	Description
AudioChannelCount	OPTIONAL	integer	Number of audio channels in the recording (for example, `2` for stereo). Must be a number greater than or equal to 1.
AudioSampleRate	OPTIONAL	number	Sample rate of the audio recording in Hertz (for example, `44100`). Must be a number greater than 0.
AudioBitDepth	OPTIONAL	integer	Number of bits per sample in the audio recording. Common values include `16`, `24`, or `32`. Must be a number greater than or equal to 1.

The following fields are available for video recordings (_video) and audio-video recordings (_audiovideo):

Key name	Requirement Level	Data type	Description
FrameRate	OPTIONAL	number	Frame rate of the video recording in frames per second (for example, `30.0`). Must be a number greater than 0.
Height	OPTIONAL	integer	Height of the video in pixels (for example, `1080`). Must be a number greater than or equal to 1.
Width	OPTIONAL	integer	Width of the video in pixels (for example, `1920`). Must be a number greater than or equal to 1.
CameraPosition	OPTIONAL	string	Free-form description of the camera placement relative to the subject or scene. Examples include "front", "profile-left", "ceiling", "room-corner", or "overhead".

The following fields are available for image files (_image):

Key name	Requirement Level	Data type	Description
Device	OPTIONAL	string	Free-form description of the device used to record the data (for example, `"iPhone 12"`, `"Canon EOS R5"`).
DeviceSerialNumber	OPTIONAL	string	The serial number of the equipment that produced the measurements. A pseudonym can also be used to prevent the equipment from being identifiable, so long as each pseudonym is unique within the dataset.
License	OPTIONAL	string	The license for the dataset. The use of license name abbreviations is RECOMMENDED for specifying a license (see Licenses). The corresponding full license text MAY be specified in an additional `LICENSE` file.

Key name	Requirement Level	Data type	Description
Height	OPTIONAL	integer	Height of the video in pixels (for example, `1080`). Must be a number greater than or equal to 1.
Width	OPTIONAL	integer	Width of the video in pixels (for example, `1920`). Must be a number greater than or equal to 1.
CameraPosition	OPTIONAL	string	Free-form description of the camera placement relative to the subject or scene. Examples include "front", "profile-left", "ceiling", "room-corner", or "overhead".

Example audio-video sidecar JSON

For an audio-video file containing both video and audio streams:

{
  "TaskName": "RestingState",
  "Device": "Sony FDR-AX53",
  "AudioChannelCount": 2,
  "AudioSampleRate": 48000,
  "FrameRate": 30.0,
  "Height": 1080,
  "Width": 1920,
  "Duration": 600.5
}

Example video sidecar JSON

For a video-only recording:

{
  "TaskName": "RestingState",
  "Device": "Sony FDR-AX53",
  "FrameRate": 30.0,
  "Height": 1080,
  "Width": 1920,
  "Duration": 600.5
}

Example audio sidecar JSON

For an audio-only recording:

{
  "TaskName": "Vocalization",
  "Device": "Zoom H6 Handy Recorder",
  "AudioChannelCount": 2,
  "AudioSampleRate": 44100,
  "Duration": 300.2
}

Example image sidecar JSON

For a still image:

{
  "TaskName": "Reaching",
  "Device": "GoPro Hero 10",
  "Height": 1080,
  "Width": 1920,
  "CameraPosition": "overhead"
}

Annotations and events

Behavioral annotations or event markers for audio and video recordings SHOULD be stored in accompanying _events.tsv files following the standard events file format. These events files use the same filename entities as the audio/video file they describe, but with the _events suffix.

For example:

└─ sub-01/
   └─ beh/
      ├─ sub-01_task-speech_audio.wav 
      ├─ sub-01_task-speech_audio.json 
      ├─ sub-01_task-speech_events.tsv 
      └─ sub-01_task-speech_events.json

Example `_beh.tsv`

trial	response	response_time	stim_file
congruent	red	1.435	images/word-red_color-red.jpg
incongruent	red	1.739	images/word-red_color-blue.jpg

In the accompanying JSON sidecar, the trial column might be documented as follows:

{
   "TaskName": "Stroop",
   "trial": {
      "LongName": "Trial name",
      "Description": "Indicator of the type of trial",
      "Levels": {
         "congruent": "Word and font color match.",
         "incongruent": "Word and font color do not match."
      }
   }
}