<> = Linux : Using Comanche = == Linux, VIM & Python == Most electronic structure and modelling codes are run from the Linux [[http://en.wikipedia.org/wiki/Command-line_interface|command line]]. There is nothing very complicated about this; instead of using the mouse to access programs and commands, we will type out commands in what is known as the shell. Some of you will already be fairly familiar with the Linux environment, the command line, using non graphical editors and perhaps even programming. If so, the links provided here will lead you to pages/documents that could be a useful refresher. If this is all new to you, then you will learn enough about Linux, the VIM editor and programming in Python to get you going. == Linux == The best way to come to grips with an operating system like Linux is to get your hands dirty and interact with it. You should make sure you have access to the [[http://en.wikipedia.org/wiki/Linux_shell|Linux shell]] either through a native Linux workstation, or by logging onto one, or, if you prefer Windows, by using a shell like [[http://en.wikipedia.org/wiki/Cygwin|Cygwin]] or [[http://mobaxterm.mobatek.net/|MobaXterm]] which is also free. If you are unsure how to get this going, ask one of our Administrators. == Tutorials == * Start here: James Spencer (a colleague of mine, now at Imperial) has developed a series of tutorials that can be [[http://www.cmth.ph.ic.ac.uk/people/j.spencer/cdt.php|found here]]. We will use the [[http://www.cmth.ph.ic.ac.uk/people/j.spencer/cdt.php|Introduction to Unix/Linux/Bash tutorial]]. This tutorial comes with a series of exercises. Please do them. Solutions are provided on James' page. Use these links to find local copies of James' [[attachment:james-unix-2013.pdf|tutorials]] and [[attachment:james-unix-2013_answers.pdf|solutions]]. * Earlier versions of James' tutorials can be found at these links: 1. [[attachment:unix_session1.pdf|Linux1]] 1. [[attachment:unix_session2.pdf|Linux2]] 1. [[attachment:unix_session3.pdf|Linux3]] 1. [[attachment:unix_session4.pdf|Linux4]] * The University of Surrey has a nice and comprehensive [[http://www.ee.surrey.ac.uk/Teaching/Unix/|Linux tutorial]]. * [[https://linuxjourney.com/|Linux Journey]] : This looks like a **very good site**. (Thanks Charlie Woollard for pointing this out). You don't need to install anything! {{{#!wiki important Important Best option as of now: [[https://linuxjourney.com/|Linux Journey]] }}} == VIM == The World is divided into those who use [[http://en.wikipedia.org/wiki/Emacs|Emacs]] and those who use [[http://en.wikipedia.org/wiki/Vim_(text_editor)|VIM]]. The others don't count. I will not force either of these on you, but if you use the Linux command line, you will need to learn to use one of these and I recommend VIM. == Tutorials == * The [[http://www.openvim.com/tutorial.html|OpenVIM project]] has a cool interactive tutorial. Begin here. * Vim comes with a built-in tutorial that can be accessed as follows: * At the command line (represented by a ''$'') type: ''$ vimtutor'' * This takes you directly into the tutor. You navigate the cursor using either the arrow keys or the letters ''h'', ''k'',''j'',''l''. * Vim has two modes: an ''EDIT'' mode and a ''COMMAND'' mode. The lesson explains it all, but bear in mind that the behaviour of the editor changes dramatically depending on which mode you are in. * Go through all lessons (it will take you a hour or so). I learnt how to use Vim through this tutor. == Using Comanche == == Overview == {{{#!wiki note Note Machine names: * comanche.ph.qmul.ac.uk * poset.ph.qmul.ac.uk }}} Comanche is a 16 core Sandybridge-E machine with 64GB RAM and two RAID0 scratch drives: * /scratch/HDD_2T/ : contains two HDDs in RAID0 (you will use this) * /scratch/SSD_1TB/ and /scratch/SSH_220G/ : Two faster scratch drives, both containing two SSDs in RAID0. You do not have accounts on these! The home drive /home/ is mounted from Poset via NFS (I think). This means access to /home/ is slow as it goes over the network. Additionally, the file system used on /home/ is GlusterFS which is wonderful (you can expand it indefinitely) but it is slow. {{{#!wiki note Note --------------------------------------------------------------------- Consequently no jobs should be run on /home/ as excessive file access will make the machine and network unresponsive. --------------------------------------------------------------------- }}} We will run all jobs on our ''/scratch/HDD_2T/'' directory. == Access to Comanche == You cannot access comanche directly. It is behind a firewall, and must be accessed through the gateway: poset.ph.qmul.ac.uk {{{#!wiki important Important Each of you will have a username on poset. You will have been given this by our systems administrators. In the instructions below, where you see ''$USER'', use your user name. Don't have a user name? Let me know immediately! }}} Follow these steps: * Start an X-session. * Windows: Use Putty or Cygwin * OSX: Start the Terminal * Linux: Start the Terminal * Log onto poset using: ''ssh -X $USER@poset.ph.qmul.ac.uk'' * You will be asked for your password. Type it in. If this is your first time logging onto poset, change your password using the ''passwd'' command. See directions below. * Now you should be ready to log onto comanche. To do this use: ''ssh -X $USER@comanche.ph.qmul.ac.uk''. * Once again you will be asked for a password. Here too you will need to reset your password the first time you log in. {{{#!wiki important Important When you log on to comanche you may not be in the BASH shell. If the terminal says something like -sh: 11: .: Can't open /usr/share/modules/init/sh $ then you are in the sh shell. To get into the BASH shell type $ bash on the prompt. Here the $ indicates the command prompt. You don't type this! Now you should see something like: $ bash ug_2017_17@comanche:~$ }}} == First things == On your first login you will need to change your password using the ''passwd'' command: {{{ $ passwd }}} and follow the instructions. Next, set up a few environment variables and commands. Linux allows you to customize your shell using commands/definitions placed in the .bashrc (it starts with a '.' - this type of file is usually hidden from view) and, sometimes, the .bash_profile file. Both of these are located in your ''$HOME'' directory. ''$HOME'' is itself an environment variable that contains the location of your home directory: {{{ $ echo $HOME /home/alston }}} That's my home directory. The ''echo'' command echos to the contents of the environment variable (all these begin with a ''$'' and contain uppercase letters only! ''$ echo $home'' will not work. Try it!) to the screen. Another useful variable is ''$USER''. Figure out what it contains. We need to define a few more commands/variables as follows: First go to your ''$HOME'' directory: {{{ $ cd <--- 'cd' without an argument will take us home. $ vi .bashrc <--- or use emacs. Try not to use pico! }}} Now edit the ''.bashrc'' file to include the following (do not erase anything that might already exist in the file): {{{ #Set the type of prompt PS1="[\u@\h \W]\\$ " export PS1 # Make rm interactive (you can override this using rm -f alias rm='rm -i' # Better listing commands: alias l='ls -FG --color=always | less -R' alias ls='ls -G --color=always ' # # Define some environment variables: #export AJM=/home/alston export AJMOPT=/opt/ajm/ export CAMCASP=${AJMOPT}/CamCASP/current export PARALLEL_NWCHEM='nwchem' if [[ $HOSTNAME = 'comanche' ]]; then export SCRATCH=/scratch/HDD_2T/esm/${USER} export NWCHEM_TOP=${AJMOPT}/nwchem/current # Now define the search path for programs: export PATH=${HOME}/bin:$CAMCASP/bin:$CAMCASP/utilities/:$PATH export PATH=${NWCHEM_TOP}/bin/:$PATH export PSI4_HOME=${AJMOPT}/Psi4/current export PSI_SCRATCH=${SCRATCH} fi if [[ $HOSTNAME = 'spasrv12' ]]; then export SCRATCH=/scratch/${USER} export NWCHEM_TOP=${AJMOPT}/nwchem/current # Now define the search path for programs: export PATH=${HOME}/bin:$CAMCASP/bin:$CAMCASP/utilities/:$PATH export PATH=${NWCHEM_TOP}/bin/:$PATH export PSI4_HOME=${AJMOPT}/Psi4/current export PSI_SCRATCH=${SCRATCH} fi # Having defined NWCHEM_TOP we can now define where the basis sets are: export NWCHEM_BASIS_LIBRARY=${NWCHEM_TOP}/data/libraries/ export QUEUE='batch' export CORES=12 export OMP_NUM_THREADS=1 }}} If you have an '''ESM''' account (it will be of the form ''esm_xxxx_x'') then you need to define the **SCRATCH** variable differently: {{{ export SCRATCH=/scratch/HDD_2T/esm/${USER} }}} Once all these commands are in the ''.bashrc'' file you need to either log out and log back on (this file is executed only on login) or force execution using {{{ $ source .bashrc }}} Check if this has worked using the '''' auto-completion. This is cool. You type in the first few letters of the command you want and hit '''' which then attempts to complete the command for you. So if nwchem is now correctly on our ''$PATH'' then {{{ $ nw }}} should result in {{{ $ nwchem }}} If not, then something is wrong. Either the ''$PATH'' is not correctly defined or the binary file nwchem is not where it is. To see where commands are located (where the files actually reside) use {{{ $ which nwchem /opt/alston/nwchem/current/bin/nwchem }}} The ''which'' command tells you which binary you are using and where it is. {{{#!wiki important Important Both spasrv12 and comanche have system installations of NWChem 6.3. This version is located in /usr/bin/nwchem It is faster than then NWChem 6.6 installed in /opt/ajm/nwchem/, but as it is an older version, please don't use it unless something appears to be wrong with the current version. In order to use the system installation of NWChem use the following commands in your .bashrc: export NWCHEM_TOP=/usr/bin/nwchem export NWCHEM_BASIS_LIBRARY=${AJMOPT}/nwchem/current/data/libraries/ export PATH=/usr/bin/:${PATH} The last command ensures that /usr/bin is searched for the NWChem binary before looking elsewhere. }}} == Running NWChem == First, make sure the NWChem environment variables are defined in your .bashrc file. Note that the commands you may have recently introduced in the .bashrc file may not be available. This file is executed only on login. To force the shell to execute the commands in this file use {{{ $ source .bashrc }}} Our NWChem code has been compiled using the MPICH2 version of the message passing interface (MPI). This library allows the program to run in parallel on many processors. To run the MPI code the usual usage is: {{{ mpirun.mpich -np nwchem input.nw >& output }}} Here, ''num proc'' is the number of processors, ''input.nw'' is the file containing the NWChem commands and ''output'' is the output file. We will normally use only one processor and label our input and output files to reflect the type of calculation being performed. For example, for a calculation of the Hartree-Fock energy of water we might use file names with ''H2O_HF'' as a prefix: {{{ mpirun.mpich -np 1 nwchem H2O_HF.nw >& H2O_HF.out }}} {{{#!wiki note Note What is the ''>&'' and ''&'' for? On a Linux/Unix system, we will often run commands using code < IN.file >& OUT.file & Here's what is all means: * ''code'': This is the name of the executable. * ''< IN.file'': This says pass the contents of the file IN.file to the executable. Sometimes it is not needed. We do not use this for NWChem. * ''>& OUT.file'': The ''>'' says send the output to the file OUT.file. Sometimes things go wrong and we get an error message. To send this error message to the output file too we use ''>&''. * ''&'': Finally, we often end a command with a terminal ''&''. This tells the system to run the process in the background. }}} We might have to use a serial version of the code should something go wrong with the MPI version. In this case we use: {{{ $ nwchem IN_FILE > OUT_FILE & }}} Here we send the commands in IN_FILE into the NWChem binary (called nwchem) and send the output to OUT_FILE. The ''&'' put the job in the *background* and lets you get on with other things while the job runs. == Monitoring == To see what's going on use the ''top'' command: {{{ $ top }}} This will give you a screen with the most significant processes, usually ranked by CPU usage. See if your job is included. It will be difficult to see this if there are a lot of NWChem jobs running as all will have the same name, i.e., ''nwchem''. The thing to look for is the JOB_ID. One way of obtaining your job ID is to use the process command ''ps''. For example, I would use the following {{{ $ ps -ef | grep 'alston' }}} There are two commands here. The first is ''ps -ef'' which on its own results in way too much information. All I need are the processes associated with me, i.e. user ''alston''. To view only these I have *pipped* the output of ''ps -ef'' into a search command ''grep alston'' which prints out only processes that have the word 'alston' in them, like so {{{ $ ps -ef | grep 'alston' 501 584 211 0 2:53pm ?? 0:13.10 /Users/alston/Downloads/SparkleShare.app/Contents/MacOS/SparkleShare -psn_0_147492 0 252 225 0 2:52pm ttys000 0:00.02 login -pfq alston /bin/bash 0 256 225 0 2:52pm ttys001 0:00.01 login -pfq alston /bin/bash 0 260 225 0 2:52pm ttys002 0:00.02 login -pfq alston /bin/bash 0 2820 225 0 8:59am ttys003 0:00.02 login -pfq alston /bin/bash 501 2826 2821 0 9:00am ttys003 0:00.00 grep alston }}} The pipe command is ''|''. You could use many pipes. For example, to search for NWChem jobs I have initiated I would use: {{{ $ ps -ef | grep 'alston' | grep 'nwchem' }}} Once you have found your JOB_ID using, you can now locate it in the output of ''top'' and make sure it is running (remember that short jobs could finish before you have a chance to type all these commands). == Stopping and killing a job == {{{#!wiki important Important This doe not apply to MPI jobs! }}} Sometimes you've made a mistake and need to stop a job, or kill it, or perhaps you've forgotten to put it in the background and would like to do so. I you have the JOB_ID, then you could do the following: * Kill the job: $ kill JOB_ID {{{ This will send a kill signal to the JOB_ID and terminate it. Use ''top'' to monitor the progress of the *kill* command. * To kill all 'nwchem' jobs you have initiated: $ killall nwchem Use with caution as this will kill all your 'nwchem' jobs. * Stop a job: $ kill -STOP JOB_ID This will stop the job. You can re-start it using $ kill -CONT JOB_ID }}} If you've forgotten to run a job in the background and wish to logoff and go for a coffee you could do the following: {{{ $ nwchem IN_FILE > OUT_FILE }}} Opps, forgot the ''&''. You won't be given a ''$'' prompt till this job finishes. But you can stop it using {{{ }}} This will result in {{{ [1]+ Stopped nwchem $ }}} Now you've got the prompt back and can re-start the job in the background using {{{ $ bg $ [1]+ nwchem & }}} And check to see if it is running using ''top''. == NWChem Jobs == === Hydrogen Atom === First with a small STO-3G basis. Put this in file ''h-sto3g.nw'' [[attachment:h-sto3g.nw]] {{{ Memory 500 mb charge 0 Geometry units bohr H 0.0 0.0 0.0 End Basis "ao basis" spherical H library STO-3G End Title "H STO-3G " scf uhf Doublet end task scf energy }}} {{{#!wiki important Important Question: Next use a larger basis: ''aug-cc-pVTZ'' Copy the above file to h-avtz.nw. Do this using the command $ cp h-sto3g.nw h-avtz.nw Do not type '$': the dollar symbol is the sign for the Linux prompt. Now edit the file using pico or vim: $ pico h-avtz.nw Find and replace 'STO-3G' with 'aug-cc-pVTZ'. Save the file. Now run NWChem on this file: $ mpirun.mpich -np 1 nwchem h-avtz.nw >& h-avtz.out Have a look at the output. What is the final Hartree-Fock energy? How does it compare with that you obtained using the much smaller STO-3G basis. Locate the basis set in the output file. How large is it? }}} == Water == {{{#!wiki important Important When you have done the above question successfully, perform the following calculations. * Save these input files. * Run them. * Look at the outputs. * Get a feel for the kind of things NWChem writes in the output files. }}} Here's a simple energy calculation with the STO-3G basis set: [[attachment:h2o-sto3g.nw]] {{{ Memory 500 mb charge 0 Geometry units bohr O 0.0000000000 0.0000000000 0.0000000000 H1 -1.4536519600 0.0000000000 -1.1216873200 H2 1.4536519600 0.0000000000 -1.1216873200 End Basis "ao basis" spherical H library STO-3G O library STO-3G End Title "H2O STO-3G " scf Singlet end task scf energy }}} Let's try an optimization using the same basis set. Here the positions of the atoms will be changed till the energy of the system is a minimum. [[attachment:h2o-sto3g-opt.nw]] {{{ Memory 500 mb charge 0 Geometry units bohr O 0.0000000000 0.0000000000 0.0000000000 H1 -1.4536519600 0.0000000000 -1.1216873200 H2 1.4536519600 0.0000000000 -1.1216873200 End Basis "ao basis" spherical H library STO-3G O library STO-3G End Title "H2O STO-3G Geometry Optimization" scf Singlet end task scf optimize }}} And here's a properties calculation. Here you will calculate the dipole moment and quadrupole moment of the water molecule. [[attachment:h2o-sto3g-moments.nw]] {{{ Memory 500 mb charge 0 Geometry units bohr O 0.0000000000 0.0000000000 0.0000000000 H1 -1.4536519600 0.0000000000 -1.1216873200 H2 1.4536519600 0.0000000000 -1.1216873200 End Basis "ao basis" spherical H library STO-3G O library STO-3G End Title "H2O STO-3G Multipoles " scf Singlet end property dipole quadrupole end task scf property }}}