Behavioral recordings

Example datasets

Datasets containing behavioral data can be found in the BIDS examples repository and can be used as helpful guidance when curating new datasets.

Template:

sub-<label>/
    [ses-<label>/]
        beh/
            sub-<label>[_ses-<label>]_task-<label>[_acq-<label>][_run-<index>]_beh.json
            sub-<label>[_ses-<label>]_task-<label>[_acq-<label>][_run-<index>]_beh.tsv
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_split-<index>][_recording-<label>]_audio.flac
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_split-<index>][_recording-<label>]_audio.json
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_split-<index>][_recording-<label>]_audio.mp3
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_split-<index>][_recording-<label>]_audio.ogg
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_split-<index>][_recording-<label>]_audio.wav
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_split-<index>][_recording-<label>]_video.avi
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_split-<index>][_recording-<label>]_video.json
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_split-<index>][_recording-<label>]_video.mkv
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_split-<index>][_recording-<label>]_video.mp4
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_split-<index>][_recording-<label>]_audiovideo.avi
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_split-<index>][_recording-<label>]_audiovideo.json
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_split-<index>][_recording-<label>]_audiovideo.mkv
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_split-<index>][_recording-<label>]_audiovideo.mp4
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_recording-<label>]_image.jpg
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_recording-<label>]_image.json
            sub-<label>[_ses-<label>][_task-<label>][_acq-<label>][_run-<index>][_recording-<label>]_image.png
            sub-<label>[_ses-<label>]_task-<label>[_acq-<label>][_run-<index>]_events.json
            sub-<label>[_ses-<label>]_task-<label>[_acq-<label>][_run-<index>]_events.tsv
            sub-<label>[_ses-<label>]_task-<label>[_acq-<label>][_run-<index>][_recording-<label>]_physio.json
            sub-<label>[_ses-<label>]_task-<label>[_acq-<label>][_run-<index>][_recording-<label>]_physio.tsv.gz
            sub-<label>[_ses-<label>]_task-<label>[_acq-<label>][_run-<index>][_recording-<label>]_physioevents.json
            sub-<label>[_ses-<label>]_task-<label>[_acq-<label>][_run-<index>][_recording-<label>]_physioevents.tsv.gz
            sub-<label>[_ses-<label>]_task-<label>[_acq-<label>][_run-<index>][_recording-<label>]_stim.json
            sub-<label>[_ses-<label>]_task-<label>[_acq-<label>][_run-<index>][_recording-<label>]_stim.tsv.gz

Legend:

For more information about filename elements (for example, entities, suffixes, extensions), follow the links embedded in the filename template.
<matches> is a placeholder to denote an arbitrary (and valid) sequence of entities and labels at the beginning of the filename (only BIDS "raw").
<source-entities> is a placeholder to denote an arbitrary sequence of entities and labels at the beginning of the filename matching a source file from which the file derives (only BIDS-Derivatives).
Filename entities or directories between square brackets (for example, [_ses-<label>]) are OPTIONAL.
Some entities may only allow specific values, in which case those values are listed in <>, separated by |.
_<suffix> means that there are several (>6) valid suffixes for this filename pattern.
.<extension> means that there are several (>6) valid extensions for this file type.
[.gz] means that both the unzipped and gzipped versions of the extension are valid.

The beh directory MAY store behavioral recordings such as audio (_audio.*), video (_video.*), combined audio-video (_audiovideo.*), and still image (_image.*) recordings, physiological (_physio.*) recordings, and other continuous recordings (_stim.tsv.gz, _stim.json). Audio, video, audio-video, and image recordings MAY be of subjects performing tasks, resting-state behavior, or recordings of stimuli being presented to the subject. Audio/video recordings MAY occur simultaneously with other recordings, such as BOLD or EEG. Relative timing between files may be determined by consulting the scans.tsv file. If no scans.tsv file is present, the alignment is undefined. The beh directory MAY also contain event timing files (_events.tsv) and their associated metadata (_events.json) for behavioral experiments that do not have corresponding neuroimaging or functional data.

Additionally, events files that do not include the mandatory onset and duration columns MAY be included, but MUST be labeled _beh.tsv rather than _events.tsv.

The following OPTIONAL columns are pre-defined for behavioral data files:

Column name	Requirement Level	Data type	Description
trial_type	OPTIONAL	string	Primary categorisation of each trial to identify them as instances of the experimental conditions. For example: for a response inhibition task, it could take on values `go` and `no-go` to refer to response initiation and response inhibition experimental conditions.
response_time	OPTIONAL	number	Response time measured in seconds. A negative response time can be used to represent preemptive responses and `n/a` denotes a missed response.
HED	OPTIONAL	string	Hierarchical Event Descriptor (HED) tags. See the HED Appendix for details.
stim_file	OPTIONAL	string	Represents the location of the stimulus file (such as an image, video, or audio file) presented at the given onset time. There are no restrictions on the file formats of the stimuli files, but they should be stored in the `/stimuli/` directory (under the root directory of the dataset; with OPTIONAL subdirectories). The values under the `stim_file` column correspond to a path relative to `/stimuli`. For example `images/cat03.jpg` will be translated to `/stimuli/images/cat03.jpg`.
Additional Columns	OPTIONAL	`n/a`	Additional columns are allowed.

Sidecar JSON (`*_beh.json`)

In addition to the metadata that is either:

RECOMMENDED for sidecar JSON files for tabular data, or
REQUIRED for some data that can be found in the beh directory (for example SamplingFrequency and StartTime for *_<physio|stim>.tsv.gz files),

it is RECOMMENDED to add the following metadata to the JSON files of this directory.

Task information

Key name	Requirement Level	Data type	Description
TaskName	RECOMMENDED	string	Name of the task. No two tasks should have the same name. The task label included in the filename MAY be derived from this `"TaskName"` field by removing all non-alphanumeric or `+` characters (that is, all except those matching `[0-9a-zA-Z+]`), and potentially replacing spaces with `+` to ease readability. For example `"TaskName"` `"faces n-back"` or `"head nodding"` could correspond to task labels `faces+n+back` or `facesnback` and `head+nodding` or `headnodding`, respectively.
Instructions	RECOMMENDED	string	Text of the instructions given to participants before the recording.
TaskDescription	RECOMMENDED	string	Longer description of the task.
CogAtlasID	RECOMMENDED	string	URI of the corresponding Cognitive Atlas Task term.
CogPOID	RECOMMENDED	string	URI of the corresponding CogPO term.

Institution information

Key name	Requirement Level	Data type	Description
InstitutionName	RECOMMENDED	string	The name of the institution in charge of the equipment that produced the measurements.
InstitutionAddress	RECOMMENDED	string	The address of the institution in charge of the equipment that produced the measurements.
InstitutionalDepartmentName	RECOMMENDED	string	The department in the institution in charge of the equipment that produced the measurements.

Audio, video, and audio-video recordings and images

Audio and video recordings of behaving subjects MAY be stored in the beh directory using the _audio, _video, and _audiovideo suffixes. The _audio suffix is for audio-only recordings, _video for video-only recordings, and _audiovideo for recordings that contain both audio and video streams. These recordings are typically used to capture vocalizations, speech, facial expressions, body movements, or other behavioral aspects during experimental tasks or rest periods.

Still images captured during behavioral experiments MAY be stored in the beh directory using the _image suffix. These images are typically used for training frames for pose estimation, snapshots of behavioral setups, or individual frames extracted from video recordings.

Privacy and personally identifiable information

Audio and video recordings and images of human subjects often contain personally identifiable information (PII) such as faces, voices, and other identifying features. Data curators MUST take special care to ensure compliance with applicable privacy regulations (such as HIPAA in the United States, GDPR in the European Union, or other local data protection laws) when handling these recordings.

These recordings are generally more suitable for internal use or for sharing non-human subject data, unless appropriate privacy protections are implemented.

File formats

Audio recordings MUST use one of the following extensions:

.flac - Free Lossless Audio Codec
.mp3 - MPEG Audio Layer III
.ogg - Ogg Vorbis
.wav - Waveform Audio File Format

Video and audio-video recordings MUST use one of the following extensions:

.mp4 - MPEG-4 Part 14
.mkv - Matroska video container
.avi - Audio Video Interleave

Image files MUST use one of the following extensions:

.jpg - JPEG image
.png - Portable Network Graphics

Entities

Audio and video files MAY use the following entities:

task - OPTIONAL for audio and video recordings
acq - OPTIONAL, can distinguish different recording setups
run - OPTIONAL, for multiple recordings with identical parameters
recording - OPTIONAL, to differentiate simultaneous recordings from different angles, locations, or devices
split - OPTIONAL, for continuous recordings split into multiple files

Examples

└─ sub-01/
   └─ beh/
      ├─ sub-01_task-rest_video.mp4 
      ├─ sub-01_task-rest_video.json 
      ├─ sub-01_task-interview_audiovideo.mp4 
      ├─ sub-01_task-interview_audiovideo.json 
      ├─ sub-01_task-stroop_recording-face_video.mp4 
      ├─ sub-01_task-stroop_recording-face_video.json 
      ├─ sub-01_task-stroop_recording-room_video.mp4 
      ├─ sub-01_task-stroop_recording-room_video.json 
      ├─ sub-01_task-rest_image.jpg 
      ├─ sub-01_task-rest_image.json 
      ├─ sub-01_task-vocalization_audio.wav 
      └─ sub-01_task-vocalization_audio.json

For continuous recordings split into multiple files:

└─ sub-01/
   └─ ses-01/
      └─ beh/
         ├─ sub-01_ses-01_task-freeplay_run-01_split-001_video.mp4 
         ├─ sub-01_ses-01_task-freeplay_run-01_split-002_video.mp4 
         ├─ sub-01_ses-01_task-freeplay_run-01_split-003_video.mp4 
         └─ sub-01_ses-01_task-freeplay_run-01_video.json

Sidecar JSON for audio, video, audio-video recordings, and images

The following metadata fields are available for audio, video, audio-video recordings, and images:

Key name	Requirement Level	Data type	Description
Device	OPTIONAL	string	Free-form description of the device used to record the data (for example, `"iPhone 12"`, `"Canon EOS R5"`).
DeviceSerialNumber	OPTIONAL	string	The serial number of the equipment that produced the measurements. A pseudonym can also be used to prevent the equipment from being identifiable, so long as each pseudonym is unique within the dataset.
DevicePosition	OPTIONAL	string	Free-form description of the recording device's placement relative to the subject or scene. This applies to any recording device, such as a camera or microphone. Examples include "front", "profile-left", "ceiling", "room-corner", or "overhead".
License	OPTIONAL	string	The license for the dataset. The use of license name abbreviations is RECOMMENDED for specifying a license (see Licenses). The corresponding full license text MAY be specified in an additional `LICENSE` file.

Licensing for recordings containing participants

Audio, video, and image recordings of participants may have different licensing restrictions than the main dataset due to privacy considerations. The optional License field can be used to specify different terms for individual recordings that contain identifiable participant data. If not specified, the recording inherits the license from dataset_description.json.

The general stream properties of these files (duration, codecs, sample rate, dimensions, and frame rate) are described by the common media file metadata (see Common media file metadata):

Key name	Requirement Level	Data type	Description
RecordingDuration	RECOMMENDED	number	Length of the recording in seconds (for example, `3600`).

The following fields are available for audio recordings (_audio) and audio-video recordings (_audiovideo):

Key name	Requirement Level	Data type	Description
AudioCodec	RECOMMENDED	string	The audio codec used to encode the audio stream, expressed as an FFmpeg codec name (for example, `"aac"`, `"mp3"`, `"opus"`, `"flac"`, `"pcm_s16le"`). This value can be auto-extracted using `ffprobe -v quiet -print_format json -show_streams`.
AudioSampleRate	RECOMMENDED	number	Sampling frequency of the audio stream, in Hz (for example, `44100`, `48000`, `96000`). Must be a number greater than 0.
AudioChannelCount	RECOMMENDED	integer	Number of audio channels in the audio or audio-video file (for example, `1` for mono, `2` for stereo). Must be a number greater than or equal to 1.
AudioBitDepth	OPTIONAL	integer	Number of bits per sample in the audio stream (for example, `16`, `24`, or `32`). Typically reported for uncompressed or losslessly compressed audio. Must be a number greater than or equal to 1.
AudioCodecRFC6381	OPTIONAL	string	The audio codec expressed as an RFC 6381 codec string (for example, `"mp4a.40.2"` for AAC-LC). This representation is useful for web and broadcast interoperability.

The following fields are available for video recordings (_video) and audio-video recordings (_audiovideo):

Key name	Requirement Level	Data type	Description
VideoCodec	RECOMMENDED	string	The video codec used to encode the video stream, expressed as an FFmpeg codec name (for example, `"h264"`, `"hevc"`, `"vp9"`, `"av1"`). This value can be auto-extracted using `ffprobe -v quiet -print_format json -show_streams`.
VideoFrameRate	RECOMMENDED	number	The video frame rate of the video stream, in Hz (for example, `24`, `25`, `29.97`, `30`, `60`). For variable rate videos, this value should be the nominal frame rate. Must be a number greater than 0.
VideoFrameCount	RECOMMENDED	integer	Total number of frames in the video stream. For constant frame rate video this can be derived from `VideoFrameRate` and `RecordingDuration`, but for variable frame rate (VFR) video the derivation is undefined, so an explicit value is needed. Also useful as an integrity check to detect truncated or corrupted files. Must be a number greater than or equal to 1.
VideoCodecRFC6381	OPTIONAL	string	The video codec expressed as an RFC 6381 codec string (for example, `"avc1.640028"` for H.264 High Profile Level 4.0). This representation is useful for web and broadcast interoperability.
ImageWidth	RECOMMENDED	integer	Width of the video frame or image, in pixels. Corresponds to the number of columns in the stored pixel grid as captured, without applying any orientation correction that may be reported by container metadata (for example, the EXIF `Orientation` tag). Must be a number greater than or equal to 1.
ImageHeight	RECOMMENDED	integer	Height of the video frame or image, in pixels. Corresponds to the number of rows in the stored pixel grid as captured, without applying any orientation correction that may be reported by container metadata (for example, the EXIF `Orientation` tag). Must be a number greater than or equal to 1.
ImagePixelFormat	OPTIONAL	string	The pixel format of the video frame or image, as reported by FFmpeg's `pix_fmt` field (for example, `"yuv420p"`, `"yuv420p10le"`, `"gray16le"`, `"rgb24"`). A single `pix_fmt` value encodes the color model, channel count, chroma subsampling, and bit depth, and can be extracted automatically with `ffprobe`.
ImageBitDepth	OPTIONAL	integer	Bit depth per channel of the stored pixel data of the video frame or image (for example, `8`, `10`, `12`, `16`). For multi-channel data this is the depth of each individual channel. When `ImagePixelFormat` is also provided, this field is redundant with the bit depth encoded in the FFmpeg `pix_fmt` value (for example, `yuv420p10le` -> 10) and the two MUST agree. `ImageBitDepth` is nonetheless useful as a more directly discoverable summary, and as the primary precision field for image-only sidecars whose producing tools do not naturally surface `pix_fmt`. Must be a number greater than or equal to 1.

The following fields are available for image files (_image):

Key name	Requirement Level	Data type	Description
ImageWidth	RECOMMENDED	integer	Width of the video frame or image, in pixels. Corresponds to the number of columns in the stored pixel grid as captured, without applying any orientation correction that may be reported by container metadata (for example, the EXIF `Orientation` tag). Must be a number greater than or equal to 1.
ImageHeight	RECOMMENDED	integer	Height of the video frame or image, in pixels. Corresponds to the number of rows in the stored pixel grid as captured, without applying any orientation correction that may be reported by container metadata (for example, the EXIF `Orientation` tag). Must be a number greater than or equal to 1.
ImagePixelFormat	OPTIONAL	string	The pixel format of the video frame or image, as reported by FFmpeg's `pix_fmt` field (for example, `"yuv420p"`, `"yuv420p10le"`, `"gray16le"`, `"rgb24"`). A single `pix_fmt` value encodes the color model, channel count, chroma subsampling, and bit depth, and can be extracted automatically with `ffprobe`.
ImageBitDepth	OPTIONAL	integer	Bit depth per channel of the stored pixel data of the video frame or image (for example, `8`, `10`, `12`, `16`). For multi-channel data this is the depth of each individual channel. When `ImagePixelFormat` is also provided, this field is redundant with the bit depth encoded in the FFmpeg `pix_fmt` value (for example, `yuv420p10le` -> 10) and the two MUST agree. `ImageBitDepth` is nonetheless useful as a more directly discoverable summary, and as the primary precision field for image-only sidecars whose producing tools do not naturally surface `pix_fmt`. Must be a number greater than or equal to 1.

Example audio-video sidecar JSON

For an audio-video file containing both video and audio streams:

{
  "TaskName": "RestingState",
  "Device": "Sony FDR-AX53",
  "RecordingDuration": 600.5,
  "AudioCodec": "aac",
  "AudioSampleRate": 48000,
  "AudioChannelCount": 2,
  "VideoCodec": "h264",
  "VideoFrameRate": 30.0,
  "VideoFrameCount": 18015,
  "ImageWidth": 1920,
  "ImageHeight": 1080
}

Example video sidecar JSON

For a video-only recording (no audio stream):

{
  "TaskName": "RestingState",
  "Device": "Sony FDR-AX53",
  "RecordingDuration": 600.5,
  "VideoCodec": "h264",
  "VideoFrameRate": 30.0,
  "VideoFrameCount": 18015,
  "ImageWidth": 1920,
  "ImageHeight": 1080
}

Example audio sidecar JSON

For an audio-only recording:

{
  "TaskName": "Vocalization",
  "Device": "Zoom H6 Handy Recorder",
  "RecordingDuration": 300.2,
  "AudioCodec": "flac",
  "AudioSampleRate": 44100,
  "AudioChannelCount": 2,
  "AudioBitDepth": 24
}

Example image sidecar JSON

For a still image:

{
  "TaskName": "Reaching",
  "Device": "GoPro Hero 10",
  "ImageWidth": 1920,
  "ImageHeight": 1080,
  "DevicePosition": "overhead"
}

Annotations and events

Behavioral annotations or event markers for audio and video recordings SHOULD be stored in accompanying _events.tsv files following the standard events file format. These events files use the same filename entities as the audio/video file they describe, but with the _events suffix.

For example:

└─ sub-01/
   └─ beh/
      ├─ sub-01_task-speech_audio.wav 
      ├─ sub-01_task-speech_audio.json 
      ├─ sub-01_task-speech_events.tsv 
      └─ sub-01_task-speech_events.json

Example `_beh.tsv`

trial	response	response_time	stim_file
congruent	red	1.435	images/word-red_color-red.jpg
incongruent	red	1.739	images/word-red_color-blue.jpg

In the accompanying JSON sidecar, the trial column might be documented as follows:

{
   "TaskName": "Stroop",
   "trial": {
      "LongName": "Trial name",
      "Description": "Indicator of the type of trial",
      "Levels": {
         "congruent": "Word and font color match.",
         "incongruent": "Word and font color do not match."
      }
   }
}