Basic cluster software

Scientific Linux 2.6.18-238.9.1.el5 #1 SMP (Symmetric Multi Processors) Tue Apr 12 18:53:46 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

OpenSSH as part of the OS installation

Hadoop 0.20.2
Subversion  -r 81256ad0f2e4ab2bd34b04f53d25a6c23686dd14

Cloudera CDH3u0

java version "1.6.0_31"
Java(TM) SE Runtime Environment (build 1.6.0_31-b04)
Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)

Cluster hardware

NETWORK infrastructure

The head nodes and the slaves are connected to the Tier1's core network at 1Gbps, they are connected to several switch stacks with 80Gbps backplanes.

The switch stacks are in turn connected by 20Gbps backbone links to a core non-blocking switch.

NameNode and JobTracker

 
  Total disk capacity: 70+TBs
 
  CPU: 2 x Xeon [email protected] Dualcore CPU ( 4 cores )
  RAM: 8GB
  NIC: 2 x GbE (1 used)
  DISK: 1 x 250GB DISK

Slaves 1 - 4

  Supermicro/Boston custom storage solution
  CPU: 2 x Dual-Core AMD Opteron(tm) Processor 2220
  RAM: 8GB
  NIC: 2 x GbE (1 used)
  DISK: 16 x 750GB disks as JBOD

Slaves 5 - 6

Supermicro/Boston custom storage solution

CPU: 2 x  Intel(R) Xeon(R) CPU E5405 @ 2.00GHz ( 8 cores )
RAM: 8GB
NIC: 2 x GbE (1 used)
DISK: 24 x 1TB disks as JBOD