Changes between Version 9 and Version 10 of JASMIN/LOTUS

17/11/16 12:36:14 (3 years ago)




    v9 v10  
    1 = LOTUS Batch Processing Facility = 
    3 [[PageOutline]] 
    5 == Introduction == 
    7 LOTUS is the Batch Processing Facility available on JASMIN. Users can employ LOTUS for large processing requests that benefit from multiple tasks being run in parallel (not necessarily using parallel memory capabilities). 
    9 The main documentation on using LOTUS is now available at   
    11 If you want to use LOTUS please request access to the LOTUS queues through the [ CEDA Help Desk]. 
    13 See [wiki:JASMIN/LOTUS/ExampleUsage LOTUS example usage page]. 
    15 [[Image(LOTUS.jpg,width=40%,align=right)]] 
    17 == Specifications == 
    19  - Login ‘head’ node ( a VM.  4 Batch queues: 
    20    1. lotus (8 Nodes with 2x6 Cores Intel 3.5GHz, 48G RAM, 10G Networking. 10Gb Std latency TCP MPI ) = 96 Cores 
    21    1. lotus-g (3..6 Nodes with 2x6 Cores Intel 3.0GHz 96G RAM, 10G Networking. 10Gb Gnodal low latency TCP MPI ) = 36..72 cores 
    22    1. lotus-smp (1 node with 4x12 cores AMD 2.6GHZ 256GB RAM, 10Gb Networking) 
    23    1. lotus-serial (co-exists with lotus-smp and lotus queue hardware) 
    24  - Software 
    25    - RHEL6.2 OS 
    26    - Platform LSF batch scheduler 
    27    - Platform MPI (+/- OpenMPI) 
    28      - Full Support for MPI I/O on Panasas parallel filesystems 
    29    - Intel and PGI compilers 
    30    - Central repository for community installed software 
    31    - Environment modules 
    32   - Panasas Parallel Storage connected at 10Gb throughout: 
    33     - CEDA Archive 
    34     - 4TB scratch (expandable to 100’s TB) 
    35     - Home directories 
    37 == Installing application code == 
    39 All the software on the cluster is stored under /apps . These are mounted from our central applications repository from is added the RHEL6 intel and PGI compilers and libs as a starting point (we have GSL, HDF5, fftw etc). Its available to all hosts in the cluster. We have apps others like R and octave that we can add from the repository later. 
    41 User contributed software is installed under /apps/contrib/<name> after root has created a <name> directory. Its then available to all nodes in the cluster. As an example  /apps/contrib/cmip5qc. The intent is that an expert user installs the code under /apps/contrib and then ‘publishes’ this through the environment modules (so its listed under “module avail” list). If you place a module file in the directory /apps/modulefiles/contrib/cmip5qc it will appear to all users. There are ways we can limit access to the software to certain users if there are propriety code issues. 
    43 Compiling is an interactive session rather than a batch session, but you use the LSF batch system to allocate (and log you into) one of the job slots on the cluster nodes (using the “bsub –q lotus-g –Is /bin/bash” command). This way you are compiling using the physical processors the final code will run on, and move the compilation load off the small virtual machine ‘head’ node. All the directories are network mounted so /apps/contrib/ and /home/users/ are the same on all cluster nodes. 
    45 == Environment Modules == 
    47  In the simplest case its just a convenient way of adding just the bits of s/w you need to $PATH without having 100’s of unused packages in the $PATH. Its bash/csh agnostic so it doesn’t matter which shell you use. You don’t have to use it but it makes it easier for users. Users “module load cmip5qc” then submit the job (using bsub). LSF inherits the $PATH so the code is in the PATH when the job starts. Then your scripts only need refer to just the executable name without its full pathname. A simple module file is shown below. You can add code specific environment variables (see the other module files under /apps/modulesfiles/...). Its also a great way to have several versions of the code installed at the same time and be able to switch from one to other quickly. 
    49 {{{ 
    50 #%Module -*- tcl -*- 
    51 ## 
    52 ## modulefile 
    53 ## 
    54 proc ModulesHelp { } { 
    55   puts stderr "\tAdds CMIP5QC tools to your environment variables," 
    56 } 
    58 module-whatis " Adds CMIP5QC tools to your environment variables" 
    60 set              root              /apps/contrib./cmip5qc 
    61 prepend-path     PATH              $root/bin 
    62 }}} 
     1Documentation about '''LOTUS''' has moved to: