File
Staging/Specific File Movement to Local Machine
If you
need to use data on a local /scratch filesystem you can use
the UNIX cp (copy) command in the shell scripts. Because the
scratch filesystem on the HPCC systems could be 10 - 100 times
as fast as AFS, this can yield a large improvement in file
I/O and decrease the time it takes to run jobs which do large
amounts of I/O, e.g. sorting algorithms.
The use
of the local /scratch space might also be used in order to
minimize the loss of work should a network outage or AFS server
outage occur. Many users may however just want to hassle with
this and just run this out of their AFS directories. This
is fine especially if large amounts of IO aren't being used.
Here's a sample batch script which explicitly moves files
from the AFS directory which I submitted this script from
to the local /scratch filesystem. batch.script.with.io contains
the following:
#!/bin/csh
cp input.file /scratch/input.file
job_to_run_with_lots_of_IO < /scratch/input.file > /scratch/output.file
cp /scratch/output.file output.file
(Back
to HPCC Batch System Page)
modified
12/12/02
|
