SSH, AFS and permissions
If you are running jobs that last for a few days on CS managed machines you have probably run into some permission or input/output problems.
The reason for this is that Kerberos tickets (used to grant you access to a server) and AFS tickets (used to grant you access to the AFS file system) are not eternal. So when you run your program it successfully obtains all the credentials it needs, but as it keeps chugging along, the tickets expire and you are left with input/output errors.
Tackling the issue
Option 1: reauth
Once you ssh into one of the CS managed servers run the reauth script and it will take care of renewing your tickets.
~$ ssh your_csid@madmax.stanford.edu Last login: Thu Jan 10 13:51:03 2013 from gonzo.stanford.edu ~$ /afs/cs/software/bin/reauth Password for your_csid: Background process pid is: 60858 ~$ runMyJob.sh ...
Option 2: krbscreen
Ssh to one of the CS managed servers and run krbscreen. This is just like a regular screen command but it will take care of renewing your tickets as well.
~$ ssh your_csid@madmax.stanford.edu Last login: Thu Jan 10 13:51:03 2013 from gonzo.stanford.edu ~$ krbscreen ~$ runMyJob.sh ...
Option 3: manually
Once you are logged into a CS managed server, you can use the kinit and aklog commands to renew your tickets.
~$ ssh your_csid@madmax.stanford.edu Last login: Thu Jan 10 13:51:03 2013 from gonzo.stanford.edu ~$ ... ~$ kinit -R ~$ aklog ~$ ...