Scientific Computing Facilities Technical Summary
June, 2009 Configuration
Table of contents
Blue Gene
Hardware Configuration
Each node contains two processors so the number of processors is twice the number of nodes listed.
| Machine Name |
Role in Cluster |
# Nodes |
Cache |
Memory (per node) |
Scratch Disk |
Network |
| Levi and Lee |
Login Node |
2 x 1.5 GHz Power5 |
see below
|
4 GB |
72 GB |
1Gbps
Ethernet |
| |
Compute Nodes |
1024 x 700 MHz PPC440 |
see below
|
512 MB |
none |
Torus, Tree, Global Interrupt |
| |
IO Nodes |
128 x 700 MHz PPC440 |
see below
|
512 MB |
none |
1 Gbps
Ethernet, Tree |
Note on front end caches: Each processor has a 64 KB instruction, 32 KB data L1 cache. Each node has a shared 1.9MB L2 and 36 MB L3 cache.
Note on compute and IO node caches: Each node has a 32 KB L1 cache, 2 KB L2 cache, and a 4 MB L3 cache.
Batch System and Usage
The IBM Blue Gene's batch system is IBM's LoadLeveler. The current limitation is that all jobs must use a partition of exactly 32, 128, 512 or 1024 (the entire machine) nodes and no job may run for more than 5 hours of wall-clock time. 1024-node jobs are only allowed to run in off-hours.
Each CPU hour (counted by wall clock time) used on the Blue Gene will use 0.25 SUs of a project's allocation.
pSeries
Hardware Configuration
| Host Name |
Model |
# Processors |
Cache |
Memory (Aggregate) |
Scratch Disk |
Network |
| Twister |
IBM pSeries 655 |
8 x 1.1 GHz Power4 |
see below
|
16 GB |
36 GB |
1 Gbps
Ethernet |
| Scrabble |
IBM pSeries 655 |
8 x 1.1 GHz Power4 |
see below
|
16 GB |
36 GB |
1 Gbps
Ethernet |
| Marbles |
IBM pSeries 655 |
8 x 1.1 GHz Power4 |
see below
|
16 GB |
36 GB |
1 Gbps
Ethernet |
| Crayon |
IBM pSeries 655 |
8 x 1.1 GHz Power4 |
see below
|
16 GB |
36 GB |
1 Gbps
Ethernet |
| Litebrite |
IBM pSeries 655 |
8 x 1.1 GHz Power4 |
see below
|
16 GB |
36 GB |
1 Gbps
Ethernet |
| Hotwheels |
IBM pSeries 655 |
8 x 1.1 GHz Power4 |
see below
|
16 GB |
36 GB |
1Gbps
Ethernet |
| Jacks1 |
IBM pSeries 655 |
8 x 1.7 GHz Power4 |
see below
|
8 GB |
72 GB |
1 Gbps
Ethernet |
| Playdoh1 |
IBM pSeries 655 |
8 x 1.7 GHz Power4 |
see below
|
8 GB |
72 GB |
1 Gbps
Ethernet |
| Slinky1 |
IBM pSeries 655 |
8 x 1.7 GHz Power4 |
see below
|
8 GB |
72 GB |
1 Gbps
Ethernet |
Note on 6xx caches: 32 KB L1 cache on each processor, 1.41 MB L2 cache shared by each pair of processors, 128 MB L3 cache shared by each set of eight processors.
Batch System and Usage
The IBM pSeries machines have a detailed queue structure and certain machines dedicated to certain job types as indicated below. The batch system used to manage the queues is the Load Sharing Facility (LSF).
| Host Name |
Function |
Service Level |
Batch Queues |
| Twister |
Interactive |
Production |
none |
| Scrabble |
SP batch
MP batch |
Production |
p4-short
p4-verylong
p4-mp4 |
| Marbles |
SP batch |
Production |
p4-long |
| Crayon |
SP batch |
Production |
p4-long |
| Litebrite |
MP batch |
Production |
p4-mp8 |
| Hotwheels |
MP batch |
Production |
p4-mp8 |
| Jacks |
MP batch/Interactive1 |
Production |
p4-cism-mp8 |
| Playdoh |
MP batch/Interactive1 |
Production |
p4-cism-mp8 |
| Slinky |
MP batch/Interactive1 |
Production |
p4-cism-mp8 |
[Notes: SP=Single Processor; MP=Multiple Processor]
| Queue Name |
# processors |
CPU Limit
(in hours) |
Wall Clock Limit
(in hours) |
Run Window |
Slots |
| p4-short |
1 |
2 |
2.5 |
always |
2 |
| p4-long |
1 |
32 |
40 |
always |
16 |
| p4-verylong |
1 |
64 |
80 |
always |
2 |
| p4-mp4 |
4 |
16 |
5 |
always |
1 |
| p4-mp8 |
8 |
32 |
5 |
always |
2 |
| p4-cism-mp8 |
8 |
32 |
5 |
always |
3 |
Note: In general, all of the LSF queues will have dedicated access to all of the processors they may access ("# processors" column above) as none of the machines are allowed to be oversubscribed.
Footnotes
1: Users in the CISM group are given higher priority on the machines jacks, playdoh, and slinky and are the only people who can log in to those machines. Also, non-CISM users using these machines are charged 1.31 SUs per CPU hour they use simply due to these machines having faster processors.
SUs used per processor hour on the pSeries machines.
| Processor Type and Speed |
SUs used per CPU hour |
| 1.1 Ghz p655 |
0.85 |
| 1.7 Ghz p655 |
1.31 |
Katana Cluster
Hardware Configuration
| Machine Name |
Role in Cluster |
# Blades |
Nodes / processors |
Cache |
Memory (per blade) |
Scratch Disk |
Network |
| Katana |
Login Node |
1 |
2 dual-core 2.6 GHz AMD Opteron 2218HE |
64 KB L1 and
1 MB L2
|
8 GB |
50 GB |
1 Gbps Ethernet/
10 Gbps 4X Infiniband |
| |
8 Processor, 3.0 Ghz Compute Nodes |
6 |
2 quad-core 3.0 GHz Intel Xeon E5450 |
32 KB L1 and
12 MB L2
|
16 GB |
50 GB |
1 Gbps Ethernet/
10 Gbps 4X Infiniband |
| |
4 Processor, 2.6 Ghz Compute Nodes |
21 |
2 dual-core 2.6 GHz AMD Opteron 2218HE |
64 KB L1 and
1 MB L2
|
8 GB |
50 GB |
1 Gbps Ethernet/
10 Gbps 4X Infiniband |
| |
4 Processor, 2.4 Ghz Compute Nodes |
14 |
2 dual-core 2.4 GHz AMD Opteron 2216HE |
64 KB L1 and
1 MB L2
|
8 GB |
50 GB |
1 Gbps Ethernet |
Batch System and Usage
Users are restricted to using a maximum of 32 processors and running for 24 hours of wall-clock time (but note that the default limit is 2 hrs, to run for longer you must request that). The batch system on the Katana Cluster is the Sun Grid Engine.
Each CPU hour used on the 2.6 GHz AMD Opteron 2218HE machines is charged at 1.0 SUs per hour, while each CPU hour on the slightly slower 2.4 GHz AMD Opteron 2216HE is charged at a rate of 0.9 SUs per hour. On the faster 3.0 GHz Intel Xeon E5450 machines, the charge rate is 1.5 SUs per CPU hour.
Linux Cluster
Hardware Configuration
Each node contains two processors so the number of processors is twice the number of nodes listed.
| Machine Name |
Role in Cluster |
# Nodes |
Cache |
Memory (per node) |
Scratch Disk |
Network |
| Cootie |
Login Node |
1 x 1.261 GHz Pentium III |
16 KB L1 and
512 KB L2
|
1 GB |
36 GB |
100 Mbps Ethernet/
2.2 Gbps Myrinet |
| |
Compute Nodes |
30 x 1.261 GHz Pentium III |
16 KB L1 and
512 KB L2
|
1 GB |
36 GB |
100 Mbps Ethernet/
2.2 Gbps Myrinet |
Batch System and Usage
Users are restricted to using a maximum of 24 nodes (48 processors); there is currently no run-time limit on jobs. The number of nodes that can be used simultaneously is an aggregate, so it is fine to run 24 separate single-processor jobs (one per node) at the same time. The batch system on the Linux Cluster is PBS.
Each CPU hour used on the Linux Cluster will use 0.3 SUs of a project's allocation.
Computer Graphics Lab Workstations
The Computer Graphics Lab houses a number of high performance Linux and Windows workstations. The lab is accessible only to those with appropriate card keys. Follow the above link for more details on the machines available and getting access to the lab.
|
