Linux : Using Comanche

Linux, VIM & Python

Most electronic structure and modelling codes are run from the Linux command line. There is nothing very complicated about this; instead of using the mouse to access programs and commands, we will type out commands in what is known as the shell. Some of you will already be fairly familiar with the Linux environment, the command line, using non graphical editors and perhaps even programming. If so, the links provided here will lead you to pages/documents that could be a useful refresher. If this is all new to you, then you will learn enough about Linux, the VIM editor and programming in Python to get you going.

Linux

The best way to come to grips with an operating system like Linux is to get your hands dirty and interact with it. You should make sure you have access to the Linux shell either through a native Linux workstation, or by logging onto one, or, if you prefer Windows, by using a shell like Cygwin or MobaXterm which is also free. If you are unsure how to get this going, ask one of our Administrators.

Tutorials

• Start here: James Spencer (a colleague of mine, now at Imperial) has developed a series of tutorials that can be found here. We will use the Introduction to Unix/Linux/Bash tutorial. This tutorial comes with a series of exercises. Please do them. Solutions are provided on James' page. Use these links to find local copies of James' tutorials and solutions.

• Earlier versions of James' tutorials can be found at these links:
• The University of Surrey has a nice and comprehensive Linux tutorial.

• Linux Journey : This looks like a **very good site**. (Thanks Charlie Woollard for pointing this out). You don't need to install anything!

Important

Best option as of now: Linux Journey

VIM

The World is divided into those who use Emacs and those who use VIM. The others don't count. I will not force either of these on you, but if you use the Linux command line, you will need to learn to use one of these and I recommend VIM.

Tutorials

• The OpenVIM project has a cool interactive tutorial. Begin here.

• Vim comes with a built-in tutorial that can be accessed as follows:
• At the command line (represented by a $'') type: ''$ vimtutor

• This takes you directly into the tutor. You navigate the cursor using either the arrow keys or the letters h, k,j,l.

• Vim has two modes: an EDIT mode and a COMMAND mode. The lesson explains it all, but bear in mind that the behaviour of the editor changes dramatically depending on which mode you are in.

• Go through all lessons (it will take you a hour or so). I learnt how to use Vim through this tutor.

Overview

Note

• Machine names:
• comanche.ph.qmul.ac.uk
• poset.ph.qmul.ac.uk

Comanche is a 16 core Sandybridge-E machine with 64GB RAM and two RAID0 scratch drives:

• /scratch/HDD_2T/ : contains two HDDs in RAID0 (you will use this)
• /scratch/SSD_1TB/ and /scratch/SSH_220G/ : Two faster scratch drives, both containing two SSDs in RAID0. You do not have accounts on these!

The home drive /home/ is mounted from Poset via NFS (I think). This means access to /home/ is slow as it goes over the network. Additionally, the file system used on /home/ is GlusterFS which is wonderful (you can expand it indefinitely) but it is slow.

Note

Consequently no jobs should be run on /home/ as excessive file access will make the machine and network unresponsive.

We will run all jobs on our /scratch/HDD_2T/ directory.

You cannot access comanche directly. It is behind a firewall, and must be accessed through the gateway: poset.ph.qmul.ac.uk

Important

Each of you will have a username on poset. You will have been given this by our systems administrators. In the instructions below, where you see $USER, use your user name. Don't have a user name? Let me know immediately! Follow these steps: • Start an X-session. • Windows: Use Putty or Cygwin • OSX: Start the Terminal • Linux: Start the Terminal • Log onto poset using: ssh -X$USER@poset.ph.qmul.ac.uk

• Now you should be ready to log onto comanche. To do this use: ssh -X $USER@comanche.ph.qmul.ac.uk. • Once again you will be asked for a password. Here too you will need to reset your password the first time you log in. Important When you log on to comanche you may not be in the BASH shell. If the terminal says something like • -sh: 11: .: Can't open /usr/share/modules/init/sh$

then you are in the sh shell. To get into the BASH shell type

• $bash on the prompt. Here the$ indicates the command prompt. You don't type this! Now you should see something like:

• $bash ug_2017_17@comanche:~$

First things

  $passwd <enter> and follow the instructions. Next, set up a few environment variables and commands. Linux allows you to customize your shell using commands/definitions placed in the .bashrc (it starts with a '.' - this type of file is usually hidden from view) and, sometimes, the .bash_profile file. Both of these are located in your$HOME'' directory. ''$HOME is itself an environment variable that contains the location of your home directory: $ echo $HOME /home/alston That's my home directory. The echo command echos to the contents of the environment variable (all these begin with a$ and contain uppercase letters only! $echo$home will not work. Try it!) to the screen.

Another useful variable is $USER. Figure out what it contains. We need to define a few more commands/variables as follows: First go to your$HOME directory:

  $cd <--- 'cd' without an argument will take us home.$ vi .bashrc      <--- or use emacs. Try not to use pico!

Now edit the .bashrc file to include the following (do not erase anything that might already exist in the file):

#Set the type of prompt
PS1="[\u@\h \W]\\$" export PS1 # Make rm interactive (you can override this using rm -f alias rm='rm -i' # Better listing commands: alias l='ls -FG --color=always | less -R' alias ls='ls -G --color=always ' # # Define some environment variables: #export AJM=/home/alston export AJMOPT=/opt/ajm/ export CAMCASP=${AJMOPT}/CamCASP/current
export PARALLEL_NWCHEM='nwchem'

if [[ $HOSTNAME = 'comanche' ]]; then export SCRATCH=/scratch/HDD_2T/esm/${USER}
export NWCHEM_TOP=${AJMOPT}/nwchem/current # Now define the search path for programs: export PATH=${HOME}/bin:$CAMCASP/bin:$CAMCASP/utilities/:$PATH export PATH=${NWCHEM_TOP}/bin/:$PATH export PSI4_HOME=${AJMOPT}/Psi4/current
export PSI_SCRATCH=${SCRATCH} fi if [[$HOSTNAME = 'spasrv12' ]]; then
export SCRATCH=/scratch/${USER} export NWCHEM_TOP=${AJMOPT}/nwchem/current
# Now define the search path for programs:
export PATH=${HOME}/bin:$CAMCASP/bin:$CAMCASP/utilities/:$PATH
export PATH=${NWCHEM_TOP}/bin/:$PATH
export PSI4_HOME=${AJMOPT}/Psi4/current export PSI_SCRATCH=${SCRATCH}
fi

# Having defined NWCHEM_TOP we can now define where the basis sets are:
export NWCHEM_BASIS_LIBRARY=${NWCHEM_TOP}/data/libraries/ export QUEUE='batch' export CORES=12 export OMP_NUM_THREADS=1 If you have an ESM account (it will be of the form esm_xxxx_x) then you need to define the **SCRATCH** variable differently: export SCRATCH=/scratch/HDD_2T/esm/${USER}

Once all these commands are in the .bashrc file you need to either log out and log back on (this file is executed only on login) or force execution using

  $source .bashrc Check if this has worked using the <TAB> auto-completion. This is cool. You type in the first few letters of the command you want and hit <TAB> which then attempts to complete the command for you. So if nwchem is now correctly on our$PATH then

  $nw<TAB> should result in $ nwchem

If not, then something is wrong. Either the $PATH is not correctly defined or the binary file nwchem is not where it is. To see where commands are located (where the files actually reside) use $ which nwchem
/opt/alston/nwchem/current/bin/nwchem

The which command tells you which binary you are using and where it is.

Important

Both spasrv12 and comanche have system installations of NWChem 6.3. This version is located in

• /usr/bin/nwchem

It is faster than then NWChem 6.6 installed in /opt/ajm/nwchem/, but as it is an older version, please don't use it unless something appears to be wrong with the current version.

In order to use the system installation of NWChem use the following commands in your .bashrc:

• export NWCHEM_TOP=/usr/bin/nwchem export NWCHEM_BASIS_LIBRARY=${AJMOPT}/nwchem/current/data/libraries/ export PATH=/usr/bin/:${PATH}

The last command ensures that /usr/bin is searched for the NWChem binary before looking elsewhere.

Running NWChem

First, make sure the NWChem environment variables are defined in your .bashrc file. Note that the commands you may have recently introduced in the .bashrc file may not be available. This file is executed only on login. To force the shell to execute the commands in this file use

  $source .bashrc Our NWChem code has been compiled using the MPICH2 version of the message passing interface (MPI). This library allows the program to run in parallel on many processors. To run the MPI code the usual usage is:  mpirun.mpich -np <num_proc> nwchem input.nw >& output Here, num proc is the number of processors, input.nw is the file containing the NWChem commands and output is the output file. We will normally use only one processor and label our input and output files to reflect the type of calculation being performed. For example, for a calculation of the Hartree-Fock energy of water we might use file names with H2O_HF as a prefix:  mpirun.mpich -np 1 nwchem H2O_HF.nw >& H2O_HF.out Note What is the >& and & for? On a Linux/Unix system, we will often run commands using • code < IN.file >& OUT.file & Here's what is all means: • code: This is the name of the executable. • < IN.file: This says pass the contents of the file IN.file to the executable. Sometimes it is not needed. We do not use this for NWChem. • >& OUT.file: The > says send the output to the file OUT.file. Sometimes things go wrong and we get an error message. To send this error message to the output file too we use >&. • &: Finally, we often end a command with a terminal &. This tells the system to run the process in the background. We might have to use a serial version of the code should something go wrong with the MPI version. In this case we use: $ nwchem  IN_FILE > OUT_FILE &

Here we send the commands in IN_FILE into the NWChem binary (called nwchem) and send the output to OUT_FILE. The & put the job in the *background* and lets you get on with other things while the job runs.

Monitoring

To see what's going on use the top command:

  $top This will give you a screen with the most significant processes, usually ranked by CPU usage. See if your job is included. It will be difficult to see this if there are a lot of NWChem jobs running as all will have the same name, i.e., nwchem. The thing to look for is the JOB_ID. One way of obtaining your job ID is to use the process command ps. For example, I would use the following $ ps -ef | grep 'alston'

There are two commands here. The first is ps -ef which on its own results in way too much information. All I need are the processes associated with me, i.e. user alston. To view only these I have *pipped* the output of ps -ef into a search command grep alston which prints out only processes that have the word 'alston' in them, like so

  $ps -ef | grep 'alston' 501 584 211 0 2:53pm ?? 0:13.10 /Users/alston/Downloads/SparkleShare.app/Contents/MacOS/SparkleShare -psn_0_147492 0 252 225 0 2:52pm ttys000 0:00.02 login -pfq alston /bin/bash 0 256 225 0 2:52pm ttys001 0:00.01 login -pfq alston /bin/bash 0 260 225 0 2:52pm ttys002 0:00.02 login -pfq alston /bin/bash 0 2820 225 0 8:59am ttys003 0:00.02 login -pfq alston /bin/bash 501 2826 2821 0 9:00am ttys003 0:00.00 grep alston The pipe command is |. You could use many pipes. For example, to search for NWChem jobs I have initiated I would use: $ ps -ef | grep 'alston' | grep 'nwchem'

Once you have found your JOB_ID using, you can now locate it in the output of top and make sure it is running (remember that short jobs could finish before you have a chance to type all these commands).

Stopping and killing a job

Important

This doe not apply to MPI jobs!

Sometimes you've made a mistake and need to stop a job, or kill it, or perhaps you've forgotten to put it in the background and would like to do so. I you have the JOB_ID, then you could do the following:

• Kill the job:

$kill JOB_ID  This will send a kill signal to the JOB_ID and terminate it. Use ''top'' to monitor the progress of the *kill* command. * To kill all 'nwchem' jobs you have initiated:$ killall nwchem
Use with caution as this will kill all your 'nwchem' jobs.
* Stop a job:
$kill -STOP JOB_ID This will stop the job. You can re-start it using$ kill -CONT JOB_ID

If you've forgotten to run a job in the background and wish to logoff and go for a coffee you could do the following:

  $nwchem IN_FILE > OUT_FILE Opps, forgot the &. You won't be given a$ prompt till this job finishes. But you can stop it using

  <Control-z>

This will result in

  [1]+  Stopped       nwchem
$ Now you've got the prompt back and can re-start the job in the background using $ bg <enter>
$[1]+ nwchem & And check to see if it is running using top. NWChem Jobs Hydrogen Atom First with a small STO-3G basis. Put this in file h-sto3g.nw Memory 500 mb charge 0 Geometry units bohr H 0.0 0.0 0.0 End Basis "ao basis" spherical H library STO-3G End Title "H STO-3G " scf uhf Doublet end task scf energy Important Question: Next use a larger basis: aug-cc-pVTZ Copy the above file to h-avtz.nw. Do this using the command •$ cp h-sto3g.nw h-avtz.nw

Do not type '$': the dollar symbol is the sign for the Linux prompt. Now edit the file using pico or vim: •$ pico h-avtz.nw

Find and replace 'STO-3G' with 'aug-cc-pVTZ'. Save the file.

Now run NWChem on this file:

• \$ mpirun.mpich -np 1 nwchem h-avtz.nw >& h-avtz.out

Have a look at the output. What is the final Hartree-Fock energy? How does it compare with that you obtained using the much smaller STO-3G basis.

Locate the basis set in the output file. How large is it?

Water

Important

When you have done the above question successfully, perform the following calculations.

• Save these input files.
• Run them.
• Look at the outputs.
• Get a feel for the kind of things NWChem writes in the output files.

Here's a simple energy calculation with the STO-3G basis set:

Memory 500 mb

charge 0
Geometry units bohr
O     0.0000000000    0.0000000000    0.0000000000
H1   -1.4536519600    0.0000000000   -1.1216873200
H2    1.4536519600    0.0000000000   -1.1216873200
End

Basis "ao basis" spherical
H  library  STO-3G
O  library  STO-3G
End

Title "H2O STO-3G "

scf
Singlet
end

task scf energy

Let's try an optimization using the same basis set. Here the positions of the atoms will be changed till the energy of the system is a minimum.

Memory 500 mb

charge 0
Geometry units bohr
O     0.0000000000    0.0000000000    0.0000000000
H1   -1.4536519600    0.0000000000   -1.1216873200
H2    1.4536519600    0.0000000000   -1.1216873200
End

Basis "ao basis" spherical
H  library  STO-3G
O  library  STO-3G
End

Title "H2O STO-3G Geometry Optimization"

scf
Singlet
end

task scf optimize

And here's a properties calculation. Here you will calculate the dipole moment and quadrupole moment of the water molecule.

Memory 500 mb

charge 0
Geometry units bohr
O     0.0000000000    0.0000000000    0.0000000000
H1   -1.4536519600    0.0000000000   -1.1216873200
H2    1.4536519600    0.0000000000   -1.1216873200
End

Basis "ao basis" spherical
H  library  STO-3G
O  library  STO-3G
End

Title "H2O STO-3G Multipoles "

scf
Singlet
end
property
dipole
task scf property