# File-Staging approaches in Grid Engine

Although Grid Engine is often used in an environment where NFS is used to share files, sometimes it's useful to have a way of transferring specific files associated with Grid jobs. Situations where this is applicable could include:

- Grids where NFS is not used on some or all compute hosts, either for performance or stability reasons

- cases where files have to be obtained from some remote system. It's possible that not every host in the Grid has the ability to retrieve these files, due to connectivity, security, or bandwidth issues, but it would still be nice to have any compute host operate on them

This HOWTO will cover two general cases for setting up File-Staging in SGE.

1. Local File-Staging: this is when you wish to transfer files directly to the compute host on which a job will run

2. Site File-Staging: this means that you need to obtain a file from outside the Grid and place it in a central repository, from which any compute host in the Grid can access it (via NFS or some secondary staging mechanism)

## Local File-Staging

For non-NFS SGE clusters, file staging can be provided by prolog and epilog methods. The actual mechanism of file transfer will depend upon the local setup, but here we outline the generic setup. The "remote" compute hosts are ones which do not share any NFS filesystems with the master host or submit hosts.

This setup uses `rcp` as the file transfer mechanism. The underlying permissions for `rcp` need to be in place ahead of time. This includes:

- any potential submit host must be allowed to `rcp` to any potential compute host

- vice versa, that is, any potential compute host must be allowed to `rcp` back to any potential submit host

It is advisable that you first manually test `rcp` between hosts before implementing the file-staging procedure. Of course, the use of `rcp` is a security concern. However, it should be straightforward to convert this example to use `scp` instead of `rcp`.

### Configuration

The example given here is a simple job which transfers an input text file to the compute host, converts all characters to uppercase, and transfers the output file back to the directory from which the input file was obtained.

The prototype consists of: the prolog and epilog scripts, which run on the remote compute hosts, and a sample job script which shows how to invoke the file transfer facility. The configuration is as follows:

- for queues on the compute hosts, set the prolog and epilog to be the scripts included here:
  `prolog file_trans_prlg.sh
  epilog file_trans_eplg.sh`

- use the sample jobs `changecase.sh` as a model to create any arbitrary job script which makes use of stage-in/stage-out.

- submit the job as specified in the script. You can use this file `song_of_wreck.txt` as sample input.

- It is recommended to use `$TMPDIR` as the working directory for staged files during a job run. This environment variable is automatically set by SGE to be a directory unique to every job. The directory and its contents are automatically deleted after the epilog completes.

## Site File-Staging

The transfer of files from outside the Grid to a local central repository can often times be a more complicated task. Additionally, it might be possible that only certain hosts in the grid are appropriate for doing this transfer process. Maybe only these hosts have the ability to go outside the firewall, or perhaps these hosts have high bandwidth to the network or to the local storage. Finally, it could be possible that extremely large files need to be transferred, eg, a large dataset from a public archive.

In this situation, the ideal approach is to have the file transfer procedure be a separate job in Grid Engine. The processing jobs which depend upon this data are then submitted with a dependency on this transfer job. With this setup, the transfer jobs can be configured to run only on those hosts appropriate for this task, leaving your compute hosts free to do other jobs, instead of being held up while the data transfer occurs. By transferring to a central repository, you can then allow any compute host to process the file.

### Configuration

In this second example, a transfer job is used to download selected texts from the Gutenberg Project archive via anoymous ftp. A processing job then does a word frequency analysis and generates a report. The whole process is driven by a submit script, which submits the two tasks with proper dependencies.

The setup is as follows:

- since we are using anoymous ftp to transfer files, we need a `.netrc` file in the user home directory in order to make the ftp scriptable. Rename the download to “`.netrc”` and place this file in the user's home directory with permissions 700.

- the `transferjob.sh` script takes some arguments to specify which Gutenberg text to download and where to put it. The destination in this case should be an NFS directory available to all compute hosts. (NOTE: we are using the Gutenberg archive found at ftp.cdrom.com)

- The `countwords.pl` script does the word frequency analysis for the file given as an argument.

- The `submitscript.sh` script drives the process, and accepts as arguments the directory and filename from the Gutenberg archive

Modify the scripts to match your environment. It is recommended that you first test the transfer job by itself from the command line, to ensure that the anoymous ftp is working properly.

## Appendix: list of example files and scripts

- Local File-Staging

1. `file_trans_prlg.sh`

2. `file_trans_eplg.sh`

3. `changecase.sh`

4. `song_of_wreck.txt`

- Site File-Staging

1. `.netrc`

2. `transferjob.sh`

3. `countwords.pl`

4. `submitscript.sh`