Transcript Media:ShortCourse4

Automatic Processing Pipelines
with XNAT and REDCap
Vanderbilt University
2013-08-20
Benjamin Yvernault,
Bennett Landman, Brian Boyd,
1
Requirements
At this point, you should know how to :
–
–
–
–
–
–
–
–
Create project / sharing data
Browse data and view image previews
Search for projects, subjects, and data
Download imaging data, processed, and stored meta-data
Delete scans / sessions / projects
Use the Pre-Archive
Upload scans DICOMs
Quality Assurance
2
After this short course, you will be able to:
•
•
•
•
Create a project on REDCap
Set up the design of your REDCap project
Request API access for your project on REDCap
What is a spider:
- source code
- Which spiders already exist?
- Run spiders from your computer
• Run a spider on your local machine
• ACCRE: run spiders (status / failed / ….)
3
Setting REDCap project for
XNAT processing
4
REDCap: Setup the main instrument
Variable Name: project_xnat -> type: Text Box
Variable Name: subject_xnat -> type: Text Box
Variable Name: experiment_xnat -> type: Text Box
Variable Name: scan_xnat -> type: Text Box
Variable Name: scan_type_xnat -> type: Text Box
Variable Name: scan_sd_type -> type: Text Box
Variable Name: script_name -> type: Text Box
Variable Name: script_version -> type: Text Box
Variable Name: pdf_report -> type: File Upload
5
REDCap: Upload main instrument
Take time to create all this fields each time you have a new project.
Variable Name: project_xnat -> type: Text Box
Variable Name: subject_xnat -> type: Text Box
Variable Name: experiment_xnat -> type: Text Box
Variable Name: scan_xnat -> type: Text Box
Variable Name: scan_type_xnat -> type: Text Box
Variable Name: scan_sd_type -> type: Text Box
Variable Name: script_name -> type: Text Box
Variable Name: script_version -> type: Text Box
Variable Name: pdf_report -> type: File Upload
Upload the main instrument dictionary as a first step. File location:
http://xnat.vanderbilt.edu -> Everything about spiders -> How to upload to
REDCap -> link with the csv file.
6
REDCap: Download Shared
Libraries
7
REDCap: API Key
 To send data from XNAT to REDCap, you need to have
an API key for your project on REDCap
8
REDCap Project Online Designer
9
Spiders …
 Script executing a process on XNAT
 Download images / data from XNAT
 Execute a process on them
 Copy the data to a directory
 A specific spider to upload results on XNAT
 All the spiders on http://xnat.vanderbilt.edu
 Source code in the project Masimatlab on NITRC
 Three kind of Spiders :
 Spider that creates results for a Scan
 Spider that corresponds to an assessor
 Infrastructure spiders
10
Scan Spiders
Converter Spider :
Spider_dcm2nii_all.py
(using dcm2nii)
Spider_dcm2nii_all_v2.py (using r2agui)
Changing attributes:
Spider_set_scan_type.py
Spider_Series_Description.py
Extracting/creating files:
Spider_physlog.py
Spider_Make_Preview_from_nifti.py
11
Process Spiders
This spiders can work on a scan level (one assessor per
scan) or an experiment level (one assessor per session)
Spider that generates an assessor:
 Spider_fMRQA.py
 Spider_VBMQA.py
 Spider_dtiQA_v2.py
 Spider_labelFusion.py
 Spider_intra_sess_reg.py
 Spider_nonrigi_reg_to_ATLAS.py
 Spider_FreesurferReconAllWithQC.py
 Spider_ON_segmentation.py
12
Infrastructure Spider
Spider_report_for_projects.py to create a report on
projects (same than Xnatreport)
Spider_send_data_to_redcap.py to send data from
complet processes on XNAT to REDCap
Spider_Process_Upload to upload the processes
that finished. Any process Spiders will copy the
results needed to a specific directory. You should
run it when you need to upload processes.
13
Spider flow
Spider_process_upload.py / Xnatprocessupload
Python script
Download data
Spiders
Folder
Run process
Any language
script:
- Matlab/SPM
- Perl
- shell script
UPLOAD_SPIDER_DIR
14
How to run a spider?
Spiders can run on your local machine if you
have the requirements.
E.G: Most of the spiders use xvfb-run
python Spider_fMRIQA.py -h to see the options
Your .bashrc or .profile set up with your XNAT logs
Python packages install
Specific python package to run spiders on
ACCRE develop by our lab.
15
ACCRE and Spiders





Advanced Computing Center for Research & Education
ACCRE is a Cluster for Vanderbilt University
Python script running every night on the cluster
Submitting one job per assessor
Status on the assessor related to the job status:
 NEED_INPUTS-> the inputs files are not present on XNAT
 NEED_TO_RUN-> ready to be submitted on ACCRE
 JOB_RUNNING -> job submitted and running on ACCRE
 JOB_FAILED -> job stop with an error
 READY_TO_UPLOAD-> job ended and copy the results to the Upload directory
 UPLOADING-> Spider_Process_Upload running and uploading results
 READY_TO_COMPLETE-> a script will check and set the walltime and memory
used by the job on ACCRE.
 COMPLETE-> jobs done (processes finished and uploaded) with results on XNAT
16
ACCRE and CCI package
Spider_process_upload.py / Xnatprocessupload
Crontab job running every night
a python script
CCI
Package
Get assessors that
need to run
Folder
UPLOAD_SPIDER_DIR
Run jobs for each assessors
(Spider flow on ACCRE)
17
Status sync between
XNAT and REDCap
18
Interactive Assignment
• Add to your .bashrc the line (or .profile on Mac OS):
export UPLOAD_SPIDER_DIR=path
• Create a project on REDCap
• Upload the dictionary from http://xnat.vanderbilt.edu
– Everything about Spiders -> How to upload on REDCap
• Download the shared libraries for fMRIQA / VBMQA / dtiQA
• Run the Spider VBMQA on one T1 in the project you created in
the last courses (it’s going to take 10mins)
• Look at the results and upload them using Xnatprocessupload
• Delete the project
19