2983
Comment: hadoop job/tasktracker web UI access tricks
|
2989
adjusted heading level
|
Deletions are marked like this. | Additions are marked like this. |
Line 3: | Line 3: |
== SSH Logins == | === SSH Logins === |
Line 37: | Line 37: |
== Hadoop Job/Task Tracker web UI == | === Hadoop Job/Task Tracker web UI === |
Line 52: | Line 52: |
== References == | === References === |
SSH Tricks for Accessing Infolab Servers outside the Gates Building
SSH Logins
Most of the InfolabServers are only accessible within the Gates building network or on-campus. There are remote login machines for accessing them from outside of Gates, or off-campus, but logging in twice is an annoyance, and makes life hard when copying files, etc. So, here's how you can configure your ssh to make it feel like directly accessing them regardless of your location.
If you're using a Mac or Linux machine, add the following lines to your ~/.ssh/config file:
# try to share connection across sessions to the same host ControlMaster auto ControlPath ~/.ssh/master-%r@%h:%p # use remote login server for InfolabServers Host il*.stanford.edu madmax*.stanford.edu rulk.stanford.edu hulk.stanford.edu rocky.stanford.edu rambo.stanford.edu bruce.stanford.edu zarya.stanford.edu eel.stanford.edu snap.stanford.edu shark.stanford.edu skate.stanford.edu whale.stanford.edu silk.stanford.edu xenon.stanford.edu ProxyCommand ssh -q ${CS_PROXY_HOST:-shark}.stanford.edu exec nc %h %p
This will make all your ssh commands go through the remote login machine "shark" by default, and you don't have to worry about where you're accessing them from. When you need, you can specify a different remote login server on the fly over the CS_PROXY_HOST environment variable as follows:
CS_PROXY_HOST=xenon ssh server ...
When you want to transfer large amount of data directly from/to the server, you might want to bypass this proxy and session sharing configuration. You can either comment out the config lines above for a while, or disable those options with command-line arguments as follows:
ssh -o ControlMaster=no -o ProxyCommand=none server ... rsync -e 'ssh -o ControlMaster=no -o ProxyCommand=none' ... server: scp -o ControlMaster=no -o ProxyCommand=none ... server:
Hadoop Job/Task Tracker web UI
Access to the InfolabClusterHadoop's Job Tracker and Task Trackers' web interface is restricted to Gates network. However, you can access them from practically anywhere using SSH port forwarding as follows. Just add a local port forwarding option as you login to the node.
ssh iln29.stanford.edu -L50030:localhost:50030
Now you can access it via your http://localhost:50030.
However, this won't solve the problem of accessing individual Task Tracker logs. To have full access to them, you should enable SSH's dynamic forwarding as you login to the head node as follows:
ssh iln29.stanford.edu -D1080
And, set your browser's SOCKS proxy to localhost:1080.
References
- ssh_config(5) man page