Accessing Data
Contents
Accessing Data¶
For this data competition, we will be using the Friday Night Lights dataset from:
In this dataset, participants (n=35) watched the pilot episode of the NBC television show Friday Night Lights while undergoing fMRI. We are sharing the raw data in the BIDS format. We are also sharing data that has been preprocessed using fMRIPrep version 20.2.1. See derivatives/code/fmriprep.sh for the script used to run preprocessing. We have also performed denoising on the preprocessed data by running a univariate GLM, which included the following regressors:
Intercept, Linear & Quadratic Trends (3)
Expanded Realignment Parameters (24). Derivatives, quadratics, quadratic derivatives
Spikes based on global outliers greater than 3 SD and also based on average frame differencing
CSF regressor (1)
See derivatives/code/Denoise_Preprocessed_Data.ipynb file for full details of the denoising model. We have included denoised data that is unsmoothed and also has been smoothed with a 6mm FWHM Gaussian kernel. We have included nifti and hdf5 versions of the data. Nifti files can be loaded using any software. HDF5 files are much faster to load if you are using our nltools toolbox for your data analyses. Note that subject sub-sid000496 had bad normalization using this preprocessing pipeline, so we have not included this participant in the denoised data.
We are also sharing additional data from this paper, which may be used in projects:
Study 1 (n=13) fMRI Data viewing episodes 1 & 2 - Available on OpenNeuro.
Study 3 (n=20) Face Expression Data - Available on OSF. We are only sharing extracted Action Unit Values. We do not have permission to share raw video data.
Study 4 (n=192) Emotion Ratings - Available on OSF. Rating data was collected on Amazon Mechanical Turk using a custom Flask web application.
Code from the original paper is available on Github
Downloading Data¶
All of the data is being hosted by OpenNeuro. The entire dataset is about 202gb and can be downloaded directly from the website and also using the AWS command line interface. See the OpenNeuro instructions for downloading. However, as most people have limited storage space, we recommend using Datalad to only download the files you want to work with. We have included a brief overview of working with Datalad from the command line and within Python below.
DataLad¶
We encourage participants to download the data using DataLad, which is an open source version control system for data built on top of git-annex. Think of it like git for data. It provides a handy command line interface for downloading data, tracking changes, and sharing it with others.
While DataLad offers a number of useful features for working with datasets, there are three in particular that we think make it worth the effort to install for this competition.
Cloning a DataLad Repository can be completed with a single line of code
datalad install <repository>and provides the full directory structure in the form of symbolic links. This allows you to explore all of the files in the dataset, without having to download the entire dataset at once.Specific files can be easily downloaded using
datalad get <filename>, and files can be removed from your computer at any time usingdatalad drop <filename>. As the dataset is relatively large, this will allow you to only work with the data that you need and you can drop the rest when you are done with it.All of the DataLad commands can be run within Python using the datalad python api.
We will only be covering a few basic DataLad functions to get and drop data. We encourage the interested reader to read the very comprehensive DataLad User Handbook for more details and troubleshooting.
Installing Datalad¶
DataLad can be easily installed using pip.
pip install datalad
Unfortunately, it currently requires manually installing the git-annex dependency, which is not automatically installed using pip.
If you are using OSX, we recommend installing git-annex using homebrew package manager.
brew install git-annex
If you are on Debian/Ubuntu we recommend enabling the NeuroDebian repository and installing with apt-get.
sudo apt-get install datalad
For more installation options, we recommend reading the DataLad installation instructions.
!pip install datalad
Requirement already satisfied: datalad in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (0.15.4)
Requirement already satisfied: tqdm in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from datalad) (4.62.3)
Requirement already satisfied: iso8601 in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from datalad) (0.1.16)
Requirement already satisfied: chardet>=3.0.4 in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from datalad) (4.0.0)
Requirement already satisfied: fasteners>=0.14 in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from datalad) (0.16.3)
Requirement already satisfied: boto in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from datalad) (2.49.0)
Requirement already satisfied: simplejson in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from datalad) (3.17.3)
Requirement already satisfied: distro in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from datalad) (1.6.0)
Requirement already satisfied: patool>=1.7 in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from datalad) (1.12)
Requirement already satisfied: requests>=1.2 in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from datalad) (2.26.0)
Requirement already satisfied: wrapt in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from datalad) (1.12.1)
Requirement already satisfied: annexremote in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from datalad) (1.5.0)
Requirement already satisfied: packaging in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from datalad) (21.3)
Requirement already satisfied: keyring>=8.0 in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from datalad) (23.4.0)
Requirement already satisfied: whoosh in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from datalad) (2.7.4)
Requirement already satisfied: keyrings.alt in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from datalad) (4.1.0)
Requirement already satisfied: msgpack in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from datalad) (1.0.2)
Requirement already satisfied: PyGithub in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from datalad) (1.55)
Requirement already satisfied: humanize in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from datalad) (3.11.0)
Requirement already satisfied: appdirs in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from datalad) (1.4.4)
Requirement already satisfied: python-gitlab in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from datalad) (2.10.0)
Requirement already satisfied: six in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from fasteners>=0.14->datalad) (1.16.0)
Requirement already satisfied: importlib-metadata>=3.6 in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from keyring>=8.0->datalad) (4.8.2)
Requirement already satisfied: zipp>=0.5 in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from importlib-metadata>=3.6->keyring>=8.0->datalad) (3.6.0)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from requests>=1.2->datalad) (1.26.7)
Requirement already satisfied: idna<4,>=2.5 in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from requests>=1.2->datalad) (3.3)
Requirement already satisfied: certifi>=2017.4.17 in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from requests>=1.2->datalad) (2021.10.8)
Requirement already satisfied: charset-normalizer~=2.0.0 in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from requests>=1.2->datalad) (2.0.4)
Requirement already satisfied: future in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from annexremote->datalad) (0.18.2)
Requirement already satisfied: setuptools in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from humanize->datalad) (58.0.4)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from packaging->datalad) (3.0.4)
Requirement already satisfied: pyjwt>=2.0 in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from PyGithub->datalad) (2.1.0)
Requirement already satisfied: pynacl>=1.4.0 in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from PyGithub->datalad) (1.4.0)
Requirement already satisfied: deprecated in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from PyGithub->datalad) (1.2.12)
Requirement already satisfied: cffi>=1.4.1 in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from pynacl>=1.4.0->PyGithub->datalad) (1.14.6)
Requirement already satisfied: pycparser in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from cffi>=1.4.1->pynacl>=1.4.0->PyGithub->datalad) (2.21)
Requirement already satisfied: requests-toolbelt>=0.9.1 in /Users/lukechang/opt/anaconda3/lib/python3.8/site-packages (from python-gitlab->datalad) (0.9.1)
Download Data with DataLad¶
The FNL dataset can be accessed at the following location https://openneuro.org/datasets/ds003521. To download the dataset run datalad install https://github.com/OpenNeuroDatasets/ds003521.git in a terminal in the location where you would like to install the dataset. Don’t forget to change the directory to a folder on your local computer. The full dataset is approximately 202gb.
You can run this from the notebook using the ! cell magic.
import os
download_path = '/Users/lukechang/Downloads/FridayNightLights'
if not os.path.exists(download_path):
os.makedirs(download_path)
%cd {download_path}
!datalad install https://github.com/OpenNeuroDatasets/ds003521.git
/Users/lukechang/Downloads/FridayNightLights
Datalad Basics¶
You might be surprised to find that after cloning the dataset that it barely takes up any space using du -sh. This is because cloning only downloads the metadata of the dataset to see what files are included.
You can check to see how big the entire dataset would be if you downloaded everything using datalad status.
%cd ~/Downloads/FridayNightLights/ds003521
!datalad status --annex
/Users/lukechang/Downloads/FridayNightLights/ds003521
1368 annex'd files (202.6 GB recorded total size)
nothing to save, working tree clean
Getting Data¶
One of the really nice features of datalad is that you can see all of the data without actually storing it on your computer. When you want a specific file you use datalad get <filename> to download that specific file. Importantly, you do not need to download all of the data at once, only when you need it.
Now that we have cloned the repository we can grab individual files. For example, suppose we wanted to grab the first subject’s confound regressors generated by fmriprep.
!datalad get participants.tsv
Now we can check and see how much of the total dataset we have downloaded using datalad status
!datalad status --annex all
1368 annex'd files (0.0 B/202.6 GB present/total size)
nothing to save, working tree clean
If you would like to download all of the files you can use datalad get .. Depending on the size of the dataset and the speed of your internet connection, this might take awhile. One really nice thing about datalad is that if your connection is interrupted you can simply run datalad get . again, and it will resume where it left off.
You can also install the dataset and download all of the files with a single command datalad install -g https://github.com/OpenNeuroDatasets/ds003521.git. You may want to do this if you have a lot of storage available and a fast internet connection. For most people, we recommend only downloading the files you need for your project.
Dropping Data¶
Most people do not have unlimited space on their hard drives and are constantly looking for ways to free up space when they are no longer actively working with files. Any file in a dataset can be removed using datalad drop. Importantly, this does not delete the file, but rather removes it from your computer. You will still be able to see file metadata after it has been dropped in case you want to download it again in the future.
As an example, let’s drop the participants.tsv file.
!datalad drop participants.tsv
Datalad has a Python API¶
One particularly nice aspect of datalad is that it has a Python API, which means that anything you would like to do with datalad in the commandline, can also be run in Python. See the details of the datalad Python API.
For example, suppose you would like to clone a data repository, such as the Localizer dataset. You can run dl.clone(source=url, path=location). Make sure you set download_path to the location where you would like the Localizer repository installed.
Also, note that datalad uses asyncio, which can lead to RuntimeError: Cannot run the event loop while another loop is running errors. You may run into this issue if you are running the code within a jupyter notebook. See the datalad documentation for more details on workarounds using nest_asyncio.
import os
import glob
import datalad.api as dl
import pandas as pd
download_path = '/Users/lukechang/Downloads/FridayNightLights'
dl.clone(source='https://github.com/OpenNeuroDatasets/ds003521.git', path=download_path)
[INFO] Cloning dataset to Dataset(/Users/lukechang/Downloads/FridayNightLights)
[INFO] Attempting to clone from https://github.com/OpenNeuroDatasets/ds003521.git to /Users/lukechang/Downloads/FridayNightLights
[INFO] Start enumerating objects
[INFO] Start counting objects
[INFO] Start compressing objects
[INFO] Start receiving objects
[INFO] Start resolving deltas
[INFO] Completed clone attempts for Dataset(/Users/lukechang/Downloads/FridayNightLights)
[INFO] scanning for annexed files (this may take some time)
[INFO] Remote origin not usable by git-annex; setting annex-ignore
[INFO] https://github.com/OpenNeuroDatasets/ds003521.git/config download failed: Not Found
[INFO] access to 1 dataset sibling s3-PRIVATE not auto-enabled, enable with:
| datalad siblings -d "/Users/lukechang/Downloads/FridayNightLights" enable -s s3-PRIVATE
Dataset('/Users/lukechang/Downloads/FridayNightLights')
We can now create a dataset instance using dl.Dataset(path_to_data).
ds = dl.Dataset(download_path)
How much of the dataset have we downloaded? We can check the status of the annex using ds.status(annex='all').
results = ds.status(annex='all')
1368 annex'd files (0.0 B/202.6 GB present/total size)
nothing to save, working tree clean
Looks like it’s empty, which makes sense since we only cloned the dataset.
Now we need to get some data. Let’s start with something small to play with first.
Let’s use glob to find all of the tab-delimited confound data generated by fmriprep.
file_list = glob.glob(os.path.join(download_path, 'derivatives', 'fmriprep', '*', 'func', '*tsv'))
file_list.sort()
file_list[:10]
['/Users/lukechang/Downloads/FridayNightLights/derivatives/fmriprep/sub-sid000216/func/sub-sid000216_task-movie_run-1_desc-confounds_timeseries.tsv',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/fmriprep/sub-sid000217/func/sub-sid000217_task-movie_run-1_desc-confounds_timeseries.tsv',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/fmriprep/sub-sid000375/func/sub-sid000375_task-movie_run-1_desc-confounds_timeseries.tsv',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/fmriprep/sub-sid000475/func/sub-sid000475_task-movie_run-1_desc-confounds_timeseries.tsv',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/fmriprep/sub-sid000476/func/sub-sid000476_task-movie_run-1_desc-confounds_timeseries.tsv',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/fmriprep/sub-sid000494/func/sub-sid000494_task-movie_run-1_desc-confounds_timeseries.tsv',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/fmriprep/sub-sid000496/func/sub-sid000496_task-movie_run-1_desc-confounds_timeseries.tsv',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/fmriprep/sub-sid000498/func/sub-sid000498_task-movie_run-1_desc-confounds_timeseries.tsv',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/fmriprep/sub-sid000529/func/sub-sid000529_task-movie_run-1_desc-confounds_timeseries.tsv',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/fmriprep/sub-sid000570/func/sub-sid000570_task-movie_run-1_desc-confounds_timeseries.tsv']
glob can search the filetree and see all of the relevant data even though none of it has been downloaded yet.
Let’s now download the first subjects confound regressor file and load it using pandas.
result = ds.get(file_list[0])
confounds = pd.read_csv(file_list[0], sep='\t')
confounds.head()
| global_signal | global_signal_derivative1 | global_signal_power2 | global_signal_derivative1_power2 | csf | csf_derivative1 | csf_power2 | csf_derivative1_power2 | white_matter | white_matter_derivative1 | ... | motion_outlier25 | motion_outlier26 | motion_outlier27 | motion_outlier28 | motion_outlier29 | motion_outlier30 | motion_outlier31 | motion_outlier32 | motion_outlier33 | motion_outlier34 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 449.788009 | NaN | 202309.252709 | NaN | 609.594384 | NaN | 371605.312452 | NaN | 451.645901 | NaN | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| 1 | 448.416865 | -1.371143 | 201077.685202 | 1.880034 | 600.366320 | -9.228063 | 360439.718706 | 85.157149 | 451.395569 | -0.250332 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| 2 | 447.555550 | -0.861315 | 200305.970725 | 0.741864 | 597.729338 | -2.636982 | 357280.362098 | 6.953674 | 450.599238 | -0.796331 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| 3 | 447.312503 | -0.243048 | 200088.475226 | 0.059072 | 596.315244 | -1.414094 | 355591.870663 | 1.999662 | 450.606712 | 0.007475 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| 4 | 446.627385 | -0.685118 | 199476.020980 | 0.469387 | 596.015874 | -0.299371 | 355234.921789 | 0.089623 | 450.673392 | 0.066680 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
5 rows × 585 columns
What if we wanted to drop that file? Just like the CLI, we can use ds.drop(file_name).
result = ds.drop(file_list[0])
To confirm that it is actually removed, let’s try to load it again with pandas.
confounds = pd.read_csv(file_list[0], sep='\t')
Looks like it was successfully removed.
We can also load the entire dataset in one command if want using ds.get(dataset='.', recursive=True). We are not going to do it right now as this will take awhile and require lots of free hard disk space.
Let’s actually download one of the files we will be using in the tutorial. First, let’s use glob to get a list of all of the functional data that has been preprocessed by fmriprep, denoised, and smoothed.
file_list = glob.glob(os.path.join(download_path, 'derivatives', 'denoised', 'smoothed', '*_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5'))
file_list.sort()
file_list
['/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000216_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000217_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000375_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000475_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000476_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000494_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000498_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000529_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000570_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000573_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000667_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000668_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000669_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000671_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000677_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000678_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000679_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000680_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000681_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000682_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000683_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000688_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000692_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000694_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000695_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000697_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000698_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000699_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000704_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000705_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000706_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000707_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000708_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000710_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.hdf5']
file_list = glob.glob(os.path.join(download_path, 'derivatives', 'denoised', 'smoothed', '*_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz'))
file_list.sort()
file_list
['/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000216_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000217_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000375_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000475_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000476_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000494_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000498_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000529_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000570_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000573_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000667_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000668_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000669_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000671_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000677_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000678_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000679_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000680_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000681_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000682_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000683_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000688_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000692_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000694_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000695_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000697_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000698_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000699_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000704_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000705_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000706_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000707_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000708_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz',
'/Users/lukechang/Downloads/FridayNightLights/derivatives/denoised/smoothed/sub-sid000710_task-movie_run-1_space-MNI152NLin2009cAsym_desc-preproc_trim_smooth6_denoised_bold.nii.gz']
Now let’s download the first subject’s file using ds.get(). This file is 1.2gb, so this might take a few minutes depending on your internet speed.
result = ds.get(file_list[0])
How much of the dataset have we downloaded? We can check the status of the annex using ds.status(annex='all').
result = ds.status(annex='all')
untracked: .DS_Store (file)
1368 annex'd files (0.0 B/202.6 GB present/total size)