Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
caviness:start [2019/03/15 15:59]
frey
caviness:start [2019/06/27 15:53] (current)
frey
Line 5: Line 5:
 The larger amount of reusable infrastructure is afforded by the Open Compute Project (OCP) design of the cluster. ​ Penguin Computing'​s //Tundra// design uses centralized DC power conversion in the racks to make node sleds as small and efficient as possible. ​ Remove the old sleds and insert the new. The larger amount of reusable infrastructure is afforded by the Open Compute Project (OCP) design of the cluster. ​ Penguin Computing'​s //Tundra// design uses centralized DC power conversion in the racks to make node sleds as small and efficient as possible. ​ Remove the old sleds and insert the new.
  
-Many Engineering research groups own capacity in Caviness.+Many Engineering research groups own capacity in Caviness. ​ UD IT maintains [[http://​docs.hpc.udel.edu/​abstract/​caviness/​caviness|its own Caviness documentation]];​ on this site you will find summaries specific to:
  
-===== Resources ===== +  * [[caviness:​login|logging in]] to the cluster 
- +  ​* [[caviness:​resources#​compute|compute resources]] present in the cluster 
-The design of Caviness is similar ​to previous community clusters. +  * [[caviness:​resources#​storage|storage resources]] available ​to you, when to use them, and how, including
- +    * your personal ​[[caviness:​resources#​home_directories|home directory]] 
-==== Networking ==== +    ​* ​[[caviness:resources#​workgroup_directories|workgroup storage]] available ​to members ​of a workgroup 
- +    * high-speed shared [[caviness:resources#​lustre_scratch|Lustre scratch]] ​storage 
-There are two private ethernet networks in the cluster.  A dedicated 1 Gbps network carries management traffic (remote power control of nodes, console access, etc.). ​ A dedicated 10 Gbps network carries all data traffic (NFS, job scheduling, SSH access) to the nodes. +  ​* [[caviness:valet|using ​VALET]] to manage your environment
- +
-A 100 Gbps Intel Omni-path network also connects all nodes. ​ The OPA network carries Lustre filesystem traffic as well as most MPI internode communications. +
- +
-==== Storage ==== +
- +
-Each rack of compute ​equipment added to Caviness is designed to add storage capacity to the cluster+
- +
-  * Lustre Object Storage Targets (OSTs) and Servers (OSSs) +
-  * NFS server +
- +
-The addition of OSTs/OSSs increases the aggregate capacity and bandwidth of the ''/​lustre/​scratch''​ filesystem. ​ Individual NFS servers provide distinct capacity and bandwidth but do not aggregate with existing capacity or bandwidth — in short, they'​re just "more space."​ +
- +
-=== Home directories === +
- +
-Each user is granted a home directory with a 20 GiB limit (quota). ​ Typically users will build software in their home directory. ​ The relatively low quota often means that users cannot (and should not) submit computational jobs from their home directories. ​ Home directories are mounted at the path ''/​home/<​uid_number>'',​ where ''<​uid_number>''​ is a user's Unix UID number (an integer value, use the ''​id''​ command ​to determine it). +
- +
-<WRAP center round info 60%> +
-The Bash shell allows ​you to reference your home directory as ''​~/''​ in most commands. ​ For example''​ls -al ~/''​ displays a long listing of the all hidden ​and normally-visible files and directories inside your home directory. +
-</​WRAP>​ +
- +
-The home directory is the location of a few important files and directories+
- +
-^File^Description^ +
-|''​.bashrc''​|Commands executed by any new Bash shell spawned for the user| +
-|''​.bash_profile''​|Commands executed specifically by a new login shell for the user| +
-|''​.bash_history''​|Saved sequence of commands the user has interactively entered at the shell prompt|  +
-|''​.bash_udit''​|Configuration file controlling UD-specific behaviors of the Bash shell| +
-|''​.valet/''​|Directory containing a user'​s ​personal ​VALET package definitions;​ does not exist by default, should be created by the user if wanted (automatically added to ''​VALET_PATH''​ by UD Bash login scripts)| +
-|''​.zfs/​snapshot/''​|Directory containing historical //​snapshots//​ of the home directory| +
- +
-The use of [[abstract:zfs-snapshots|ZFS snapshots]] as backup copies of a home directory is discussed elsewhere. ​ In general, the editing of ''​.bashrc''​ and ''​.bash_profile''​ is discouraged,​ especially for the alteration of the ''​PATH''​ and ''​LD_LIBRARY_PATH''​ environment variables. +
- +
-=== Workgroup directories === +
- +
-Each //​workgroup//​ that purchases capacity in the cluster receives a //workgroup directory// with a quota in proportion ​to its level of investment in the cluster: ​ the more compute capacity purchased, the more space granted. ​ Workgroup directories are mounted at the path ''/​work/<​workgroup-id>''​ on all nodes in the cluster. +
- +
-<WRAP center round info 60%> +
-Once you've started a shell in a workgroup ​using the ''​workgroup -g <​workgroup-id>''​ command the ''​WORKDIR''​ environment variable contains the path to the workgroup directory. ​ This allows you to reference it in commands like ''​ls -l ${WORKDIR}/​users''​. +
- +
-Adding the ''​-c''​ flag to the ''​workgroup''​ command automatically starts the workgroup shell in ''​$WORKDIR''​. +
-</​WRAP>​ +
- +
-The typical layout of a workgroup directory often includes: +
- +
-^Subdirectory^Description^ +
-|''/​work/<​workgroup-id>/​sw/''​|A directory to hold software used by multiple members of the workgroup| +
-|''/​work/<​workgroup-id>/​sw/​valet/​|Directory for VALET package definitions (automatically added to ''​VALET_PATH''​ by UD Bash login scripts)| +
-|''/​work/<​workgroup-id>/​users/''​|Directory to contain per-user ​storage ​rather than having them exist directly under ''/​work/<​workgroup-id>''​| +
-|''/​work/<​workgroup-id>/​projects/''​|Directory to contain per-project storage rather than having them exist directly under ''/​work/<​workgroup-id>''​| +
- +
-None of these directories are mandatory, but they do tend to make management of a workgroup'​s resources easier. ​ In particular, the fact that ''/​work/<​workgroup-id>/​sw/​valet''​ will be automatically added to the VALET search path means workgroup users do not need to alter ''​VALET_PATH''​ manually, in their ''​.bashrc''/''​.bash_profile'',​ or in their job scripts.+
  • caviness/start.1552665594.txt.gz
  • Last modified: 2019/03/15 15:59
  • by frey