Newer
Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
Network
=======
All compute and login nodes of Anselm are interconnected by
[Infiniband](http://en.wikipedia.org/wiki/InfiniBand)
QDR network and by Gigabit
[Ethernet](http://en.wikipedia.org/wiki/Ethernet)
network. Both networks may be used to transfer user data.
Infiniband Network
------------------
All compute and login nodes of Anselm are interconnected by a
high-bandwidth, low-latency
[Infiniband](http://en.wikipedia.org/wiki/InfiniBand)
QDR network (IB 4x QDR, 40 Gbps). The network topology is a fully
non-blocking fat-tree.
The compute nodes may be accessed via the Infiniband network using ib0
network interface, in address range 10.2.1.1-209. The MPI may be used to
establish native Infiniband connection among the nodes.
The network provides **2170MB/s** transfer rates via the TCP connection
(single stream) and up to **3600MB/s** via native Infiniband protocol.
The Fat tree topology ensures that peak transfer rates are achieved
between any two nodes, independent of network traffic exchanged among
other nodes concurrently.
Ethernet Network
----------------
The compute nodes may be accessed via the regular Gigabit Ethernet
network interface eth0, in address range 10.1.1.1-209, or by using
aliases cn1-cn209.
The network provides **114MB/s** transfer rates via the TCP connection.
Example
-------
`
$ qsub -q qexp -l select=4:ncpus=16 -N Name0 ./myjob
$ qstat -n -u username
Req'd Req'd Elap
Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time
--------------- -------- -- |---|---| ------ --- --- ------ ----- - -----
15209.srv11 username qexp Name0 5530 4 64 -- 01:00 R 00:00
cn17/0*16+cn108/0*16+cn109/0*16+cn110/0*16
$ ssh 10.2.1.110
$ ssh 10.1.1.108
`
In this example, we access the node cn110 by Infiniband network via the
ib0 interface, then from cn110 to cn108 by Ethernet network.