Setup quarentine EC2/S3 instance for user data uploads

[k1o0]

Users need to register manually curated spike data. The registration could be done in ibllib by registering to an 3S instance mounted to an EC2 instance running Globus. Alyx could then handle the transfers however something needs to assert that no Flatiron data are overwritten by this S3 data.

This issue is for simply determining the best course of action:

• Confirm S3 can allow CREATE without DELETE
• Determine whether possible to create an Globus ID from an S3 bucket (may need to ask SDSC)
• Determine whether to use Alyx Globus management command and/or to run S3 sync from SDSC

[mayofaulkner]

Step 1: Run from local server of AWS instance
Run a task using the s3 patcher

1. Check out s3_patcher branch of ibllib
2. Run a task with location='EC2', e.g

task = SpikeSorting(session_path, one=one, location='EC2', **kwargs)
task.run()
# This will upload the files via the s3 patcher to the [s3 patcher bucket](https://us-east-1.console.aws.amazon.com/s3/buckets/s3-patcher?prefix=patcher/&region=us-east-1&bucketType=general)
task.register_datasets()

Step 2: Run from SDSC
Launch the sync script from the patcher to flatiron

1. Checkout s3_patcher branch of iblalyx on SDSC
   Run the following

~/Documents/PYTHON/alyx/alyx/alyxvenv/bin/python ~/Documents/PYTHON/alyx/alyx/manage.py sync_patcher > /home/datauser/ibl_logs/sync_patcher.log 2>&1

To force an overwrite of a dataset

~/Documents/PYTHON/alyx/alyx/alyxvenv/bin/python ~/Documents/PYTHON/alyx/alyx/manage.py sync_patcher --force True > /home/datauser/ibl_logs/sync_patcher.log 2>&1

[mayofaulkner]

Items still TODO

• Add dev group to better control who can patch datasets using the sync script
• Look into boto3 authentication via alyx
• Need to implement deletion of datasets from the s3 patcher once they have transferred to flatiron
• Make sync script a cron job that is run daily
• Add AWS cloudwatch to monitor the size of s3 patcher bucket
• Need to change file name patter of spikesorting datasets to allow namspace prefix