cortex stores some recently used data on disk, but most data (especially if it hasn't been used in a while) is on tape. Staging is the process of asking cortex to move data from tape back to disk. If you don't stage your data, and you request multiple files, this is what will happen;

cortex loads a tape and moves your data from tape to disk

cortex unloads the tape and loads someone elses' tape

cortex sends you your file

cortex reloads your tape and gets the next file

When we setup your account, we also setup a project directory on cortex. This is usually of the form /pbstore/tempfs/<project-name> or /pbstore/groupfs/<project-name>.  Most of the time, the project name consists of two letters and two numbers (this is a historical thing).  You are free to put your data in any folder you desire bellow this point.

You can find your old pbstore projects on cortex here: /pbstore/tempfs/<projectname>.

There are many was to get data off cortex. Tools like scp provide a simple solution, while tools such as gridftp provide higher performance across the network once your data is on disk.

The easiest way to get data to cognac is to use the sscp command.

sscp -p <username>@cortex.ivec.org:/pbstore/<my_project>/<my_folder>/<my_files> 

The preferred method is to bundle your files into a single tar or zip prior to sending them to cortex. This is to reduce the amount of tiny files within the system, as this cause significant issues.

# tar cvf - /path-to/my_data | ssh my-uid@pbstore.ivec.org "cat > /pbstore/my_project/my_data.tar"

If possible use the -non-cipher for ssh to reduce the amount of CPU overhead generated from the encryption. Please refer to the "Using the Data Store" section for more information on this feature.

sscp -p <username>@cortex.ivec.org:/pbstore/<my_project>/<my_folder>/<my_files> 

cortex provides a range of useful tools that allow you to interact more directly with the tapes. Most of these have the same name as standard unix commands but are preceded by s (short form of SAMQFS, the name of the filesystem). For example, sls serves the same function as ls but provides additional options for the tape system. The command sls -D <filename> will tell you about whether the file is on disk or on tape, and if it’s on tape, which tape it’s on. For more info please refer to this web page.

The rm command is aliased to "rm -i". This is done to help ensure you do not accidentally delete your data, but can be difficult when you require the deletion of many files.

To work around this, use one of the following;

\rm -f

/usr/bin/rm -f

unalias rm

To find and remove many 0 byte files;

cd /pbstore/<my_project>

sfind . -size 0 -exec /usr/bin/rm -f {} \;

How to prevent OS X from creating new .DS_Store files when opening folders on remote volumes mounted using SMB/CIFS, AFP, NFS, sftp-drive and WebDAV. The creation of .DS_Store files (and more so, ._AppleDouble files which are not covered in this hint) is frequently the source of complaints against Mac users, who often leave a trail of these files scattered throughout the filesystem when "visiting" a shared network drive.

If you have existing data on cortex that needs to be put into a single tar or zip file you will need to do the following;

/opt/SUNWsamfs/bin/stage -r /pbstore/zz00/rob
/opt/SUNWsamfs/bin/stage -w -r /pbstore/zz00/rob
/usr/bin/gtar cfvp /pbstore/zz00/rob_files.tar /pbstore/zz00/rob

The above will create a file called "rob_files.tar" in the "/pbstore/zz00" directory.

There is no need to compress the file qith gzip as the hardware compression on the tape drive is better.

NOTE:

Your data is all offline on tape. You will need to bring it back onto the disk cache before trying to transfer it.

Migrating files from tape to the disk cache

NOTE: "-w" Wait for each file to be staged back on-line before completing.

This does not allow the system to sort the stage requests in the order that the files are archived on tape media. In order to get the best performance in this situation, do the following:

Basic Staging

The most likely reason is that you have had too many unsuccessful attempts at logging in and our security software has banned the IP address that the failed attempts were made from.  This software is one of several used to mitigate the constant hack attempts on iVEC services.

To get it fixed, email help@ivec.org the IP address of your computer and the name of the iVEC machine you cannot log into.  If you don't know your IP address, tell us your username and when you last tried to log into the machine.