Linux : Using Comanche

Linux, VIM & Python

Most electronic structure and modelling codes are run from the Linux command line. There is nothing very complicated about this; instead of using the mouse to access programs and commands, we will type out commands in what is known as the shell. Some of you will already be fairly familiar with the Linux environment, the command line, using non graphical editors and perhaps even programming. If so, the links provided here will lead you to pages/documents that could be a useful refresher. If this is all new to you, then you will learn enough about Linux, the VIM editor and programming in Python to get you going.

Linux

The best way to come to grips with an operating system like Linux is to get your hands dirty and interact with it. You should make sure you have access to the Linux shell either through a native Linux workstation, or by logging onto one, or, if you prefer Windows, by using a shell like Cygwin or MobaXterm which is also free. If you are unsure how to get this going, ask one of our Administrators.

Tutorials

• Start here: James Spencer (a colleague of mine, now at Imperial) has developed a series of tutorials that can be found here. We will use the Introduction to Unix/Linux/Bash tutorial. This tutorial comes with a series of exercises. Please do them. Solutions are provided on James' page. Use these links to find local copies of James' tutorials and solutions.

• Earlier versions of James' tutorials can be found at these links:

1. Linux1

2. Linux2

3. Linux3

4. Linux4

   • The University of Surrey has a nice and comprehensive Linux tutorial.

   • Linux Journey : This looks like a **very good site**. (Thanks Charlie Woollard for pointing this out). You don't need to install anything!

VIM

The World is divided into those who use Emacs and those who use VIM. The others don't count. I will not force either of these on you, but if you use the Linux command line, you will need to learn to use one of these and I recommend VIM.

Tutorials

• The OpenVIM project has a cool interactive tutorial. Begin here.

• Vim comes with a built-in tutorial that can be accessed as follows:

  • At the command line (represented by a $'') type: ''$ vimtutor

  • This takes you directly into the tutor. You navigate the cursor using either the arrow keys or the letters h, k,j,l.

  • Vim has two modes: an EDIT mode and a COMMAND mode. The lesson explains it all, but bear in mind that the behaviour of the editor changes dramatically depending on which mode you are in.

  • Go through all lessons (it will take you a hour or so). I learnt how to use Vim through this tutor.

Using Comanche

Overview

Comanche is a 16 core Sandybridge-E machine with 64GB RAM and two RAID0 scratch drives:

• /scratch/HDD_2T/ : contains two HDDs in RAID0 (you will use this)
• /scratch/SSD_1TB/ and /scratch/SSH_220G/ : Two faster scratch drives, both containing two SSDs in RAID0. You do not have accounts on these!

The home drive /home/ is mounted from Poset via NFS (I think). This means access to /home/ is slow as it goes over the network. Additionally, the file system used on /home/ is GlusterFS which is wonderful (you can expand it indefinitely) but it is slow.

We will run all jobs on our /scratch/HDD_2T/ directory.

Access to Comanche

You cannot access comanche directly. It is behind a firewall, and must be accessed through the gateway: poset.ph.qmul.ac.uk

Follow these steps:

• Start an X-session.
  • Windows: Use Putty or Cygwin
  • OSX: Start the Terminal
  • Linux: Start the Terminal

• Log onto poset using: ssh -X $USER@poset.ph.qmul.ac.uk

• You will be asked for your password. Type it in. If this is your first time logging onto poset, change your password using the passwd command. See directions below.

• Now you should be ready to log onto comanche. To do this use: ssh -X $USER@comanche.ph.qmul.ac.uk.

• Once again you will be asked for a password. Here too you will need to reset your password the first time you log in.

First things

On your first login you will need to change your password using the passwd command:

  $ passwd <enter>

and follow the instructions.

Next, set up a few environment variables and commands. Linux allows you to customize your shell using commands/definitions placed in the .bashrc (it starts with a '.' - this type of file is usually hidden from view) and, sometimes, the .bash_profile file. Both of these are located in your $HOME'' directory. ''$HOME is itself an environment variable that contains the location of your home directory:

  $ echo $HOME
  /home/alston

That's my home directory. The echo command echos to the contents of the environment variable (all these begin with a $ and contain uppercase letters only! $ echo $home will not work. Try it!) to the screen.

Another useful variable is $USER. Figure out what it contains.

We need to define a few more commands/variables as follows: First go to your $HOME directory:

  $ cd              <--- 'cd' without an argument will take us home.
  $ vi .bashrc      <--- or use emacs. Try not to use pico!

Now edit the .bashrc file to include the following (do not erase anything that might already exist in the file):

#Set the type of prompt
PS1="[\u@\h \W]\\$ "
export PS1
# Make rm interactive (you can override this using rm -f
alias rm='rm -i'
# Better listing commands:
alias l='ls -FG --color=always | less -R'
alias ls='ls -G --color=always '
#
# Define some environment variables:
#export AJM=/home/alston
export AJMOPT=/opt/ajm/
export CAMCASP=${AJMOPT}/CamCASP/current
export PARALLEL_NWCHEM='nwchem'


if [[ $HOSTNAME = 'comanche' ]]; then
        export SCRATCH=/scratch/HDD_2T/esm/${USER}
        export NWCHEM_TOP=${AJMOPT}/nwchem/current
        # Now define the search path for programs:
        export PATH=${HOME}/bin:$CAMCASP/bin:$CAMCASP/utilities/:$PATH
        export PATH=${NWCHEM_TOP}/bin/:$PATH
        export PSI4_HOME=${AJMOPT}/Psi4/current
        export PSI_SCRATCH=${SCRATCH}
fi

if [[ $HOSTNAME = 'spasrv12' ]]; then
        export SCRATCH=/scratch/${USER}
        export NWCHEM_TOP=${AJMOPT}/nwchem/current
        # Now define the search path for programs:
        export PATH=${HOME}/bin:$CAMCASP/bin:$CAMCASP/utilities/:$PATH
        export PATH=${NWCHEM_TOP}/bin/:$PATH
        export PSI4_HOME=${AJMOPT}/Psi4/current
        export PSI_SCRATCH=${SCRATCH}
fi

# Having defined NWCHEM_TOP we can now define where the basis sets are:
export NWCHEM_BASIS_LIBRARY=${NWCHEM_TOP}/data/libraries/

export QUEUE='batch'
export CORES=12
export OMP_NUM_THREADS=1

If you have an ESM account (it will be of the form esm_xxxx_x) then you need to define the **SCRATCH** variable differently:

export SCRATCH=/scratch/HDD_2T/esm/${USER}

Once all these commands are in the .bashrc file you need to either log out and log back on (this file is executed only on login) or force execution using

  $ source .bashrc

Check if this has worked using the <TAB> auto-completion. This is cool. You type in the first few letters of the command you want and hit <TAB> which then attempts to complete the command for you. So if nwchem is now correctly on our $PATH then

  $ nw<TAB>

should result in

  $ nwchem

If not, then something is wrong. Either the $PATH is not correctly defined or the binary file nwchem is not where it is.

To see where commands are located (where the files actually reside) use

  $ which nwchem
  /opt/alston/nwchem/current/bin/nwchem

The which command tells you which binary you are using and where it is.

Running NWChem

First, make sure the NWChem environment variables are defined in your .bashrc file. Note that the commands you may have recently introduced in the .bashrc file may not be available. This file is executed only on login. To force the shell to execute the commands in this file use

  $ source .bashrc

Our NWChem code has been compiled using the MPICH2 version of the message passing interface (MPI). This library allows the program to run in parallel on many processors. To run the MPI code the usual usage is:

  mpirun.mpich -np <num_proc> nwchem input.nw >& output

Here, num proc is the number of processors, input.nw is the file containing the NWChem commands and output is the output file. We will normally use only one processor and label our input and output files to reflect the type of calculation being performed. For example, for a calculation of the Hartree-Fock energy of water we might use file names with H2O_HF as a prefix:

  mpirun.mpich -np 1 nwchem H2O_HF.nw >& H2O_HF.out

We might have to use a serial version of the code should something go wrong with the MPI version. In this case we use:

  $ nwchem  IN_FILE > OUT_FILE &

Here we send the commands in IN_FILE into the NWChem binary (called nwchem) and send the output to OUT_FILE. The & put the job in the *background* and lets you get on with other things while the job runs.

Monitoring

To see what's going on use the top command:

  $ top

This will give you a screen with the most significant processes, usually ranked by CPU usage. See if your job is included. It will be difficult to see this if there are a lot of NWChem jobs running as all will have the same name, i.e., nwchem. The thing to look for is the JOB_ID.

One way of obtaining your job ID is to use the process command ps. For example, I would use the following

  $ ps -ef | grep 'alston'

There are two commands here. The first is ps -ef which on its own results in way too much information. All I need are the processes associated with me, i.e. user alston. To view only these I have *pipped* the output of ps -ef into a search command grep alston which prints out only processes that have the word 'alston' in them, like so

  $ ps -ef | grep 'alston'
  501   584   211   0  2:53pm ??         0:13.10 /Users/alston/Downloads/SparkleShare.app/Contents/MacOS/SparkleShare -psn_0_147492
    0   252   225   0  2:52pm ttys000    0:00.02 login -pfq alston /bin/bash
    0   256   225   0  2:52pm ttys001    0:00.01 login -pfq alston /bin/bash
    0   260   225   0  2:52pm ttys002    0:00.02 login -pfq alston /bin/bash
    0  2820   225   0  8:59am ttys003    0:00.02 login -pfq alston /bin/bash
  501  2826  2821   0  9:00am ttys003    0:00.00 grep alston

The pipe command is |. You could use many pipes. For example, to search for NWChem jobs I have initiated I would use:

  $ ps -ef | grep 'alston' | grep 'nwchem'

Once you have found your JOB_ID using, you can now locate it in the output of top and make sure it is running (remember that short jobs could finish before you have a chance to type all these commands).

Stopping and killing a job

Sometimes you've made a mistake and need to stop a job, or kill it, or perhaps you've forgotten to put it in the background and would like to do so. I you have the JOB_ID, then you could do the following:

• Kill the job:

$ kill JOB_ID

  This will send a kill signal to the JOB_ID and terminate it.
  Use ''top'' to monitor the progress of the *kill* command.
  * To kill all 'nwchem' jobs you have initiated:
    $ killall nwchem
  Use with caution as this will kill all your 'nwchem' jobs.
  * Stop a job:
    $ kill -STOP JOB_ID
  This will stop the job. You can re-start it using
    $ kill -CONT JOB_ID

If you've forgotten to run a job in the background and wish to logoff and go for a coffee you could do the following:

  $ nwchem  IN_FILE > OUT_FILE

Opps, forgot the &. You won't be given a $ prompt till this job finishes. But you can stop it using

  <Control-z>

This will result in

  [1]+  Stopped       nwchem
  $

Now you've got the prompt back and can re-start the job in the background using

  $ bg <enter>
  $ [1]+ nwchem &

And check to see if it is running using top.

NWChem Jobs

Hydrogen Atom

First with a small STO-3G basis. Put this in file h-sto3g.nw

h-sto3g.nw

Memory 500 mb

charge 0
Geometry units bohr
  H     0.0  0.0  0.0 
End

Basis "ao basis" spherical
  H  library  STO-3G
End

Title "H STO-3G "

scf
  uhf
  Doublet
end

task scf energy

Water

Here's a simple energy calculation with the STO-3G basis set:

h2o-sto3g.nw

Memory 500 mb

charge 0
Geometry units bohr
  O     0.0000000000    0.0000000000    0.0000000000
  H1   -1.4536519600    0.0000000000   -1.1216873200
  H2    1.4536519600    0.0000000000   -1.1216873200
End

Basis "ao basis" spherical
  H  library  STO-3G
  O  library  STO-3G
End

Title "H2O STO-3G "

scf
  Singlet
end

task scf energy

Let's try an optimization using the same basis set. Here the positions of the atoms will be changed till the energy of the system is a minimum.

h2o-sto3g-opt.nw

Memory 500 mb

charge 0
Geometry units bohr
  O     0.0000000000    0.0000000000    0.0000000000
  H1   -1.4536519600    0.0000000000   -1.1216873200
  H2    1.4536519600    0.0000000000   -1.1216873200
End

Basis "ao basis" spherical
  H  library  STO-3G
  O  library  STO-3G
End

Title "H2O STO-3G Geometry Optimization"

scf
  Singlet
end

task scf optimize

And here's a properties calculation. Here you will calculate the dipole moment and quadrupole moment of the water molecule.

h2o-sto3g-moments.nw

Memory 500 mb

charge 0
Geometry units bohr
  O     0.0000000000    0.0000000000    0.0000000000
  H1   -1.4536519600    0.0000000000   -1.1216873200
  H2    1.4536519600    0.0000000000   -1.1216873200
End

Basis "ao basis" spherical
  H  library  STO-3G
  O  library  STO-3G
End

Title "H2O STO-3G Multipoles "

scf
  Singlet
end
property
  dipole
  quadrupole
end

task scf property