Ppol
From Digitalis
Contents |
Overview
Ppol is a 3 nodes cluster featuring an hybrid HDD+SSD hard drive configuration and 10Gbps Ethernet.
The configuration of the hard drives is as follows:
- the first hard drive is a 500 GB sata HDD. This drive hosts the system (kadeploy).
- the second hard drives are 50GB sata SSD (MLC)
Those disks are connected to the system via a Raid Controller with does not provide JBOD. Hence, disks are configured as Raid0 logical disks with only one disk, so that they appear as /dev/sda, /dev/sdb in Linux. See megacli usage to change settings.
The network configuration is as follows:
- eth0 is a 1Gbps NIC connected to Grid'5000 production network vlan (connected directly to the core router)
- eth2 is a 10Gbps NIC connected to a dedicated VLAN (same a idkat) (via CIMENT's Dell 8024 Switch and then to the core router). No IP configuration is provided, so you need to do the setup manually, by yourself in a deploy job.
- eth1 and eth3 have no link.
Configuration details
Disk configuration
Some notes for the configuration of the "JBOD"-like hard drives setup
ppol-3:~# cat <<EOF >> /etc/apt/sources.list deb http://hwraid.le-vert.net/debian wheezy main EOF ppol-3:~# apt-get update ppol-3:~# apt-get install --force-yes -y megacli # Remove any foreign or previous configuration ppol-2:~# megacli -CfgForeign -Dsply -aall ppol-2:~# megacli -CfgForeign -Clear -aall ppol-2:~# megacli -CfgClr -aall # The machines Raid Controller does not support JBOD. We must use the EachDskRaid0 mode, and disable cache. ppol-2:~# megacli -CfgEachDskRaid0 WT NORA DIRECT -aall Adapter 0: Created VD 0 Configured physical device at Encl-32:Slot-0. Adapter 0: Created VD 1 Configured physical device at Encl-32:Slot-1. Adapter 0: Created VD 2 Configured physical device at Encl-32:Slot-2. 3 physical devices are Configured on adapter 0. Exit Code: 0x00
Misc
- Adapter details
ppol-3:~# megacli -AdpAllInfo -aall
Adapter #0
==============================================================================
Versions
================
Product Name : PERC H700 Integrated
Serial No : 06A002M
FW Package Build: 12.3.0-0032
Mfg. Data
================
Mfg. Date : 06/12/10
Rework Date : 06/12/10
Revision No : A02
Battery FRU : N/A
Image Versions in Flash:
================
BIOS Version : 3.09.00
FW Version : 2.30.03-0872
Preboot CLI Version: 04.02-004:#%00008
Ctrl-R Version : 2.02-0009
NVDATA Version : 2.03.0053
Boot Block Version : 2.02.00.00-0000
BOOT Version : 01.250.04.219
Pending Images in Flash
================
None
PCI Info
================
Controller Id : 0000
Vendor Id : 1000
Device Id : 0079
SubVendorId : 1028
SubDeviceId : 1f17
Host Interface : PCIE
Link Speed : 0
Number of Frontend Port: 0
Device Interface : PCIE
Number of Backend Port: 8
Port : Address
0 4433221105000000
1 4433221106000000
2 4433221107000000
3 0000000000000000
4 0000000000000000
5 0000000000000000
6 0000000000000000
7 0000000000000000
HW Configuration
================
SAS Address : 5842b2b014b22d00
BBU : Present
Alarm : Absent
NVRAM : Present
Serial Debugger : Present
Memory : Present
Flash : Present
Memory Size : 512MB
TPM : Absent
On board Expander: Absent
Upgrade Key : Absent
Temperature sensor for ROC : Absent
Temperature sensor for controller : Absent
Settings
================
Current Time : 13:41:25 7/15, 2014
Predictive Fail Poll Interval : 300sec
Interrupt Throttle Active Count : 16
Interrupt Throttle Completion : 50us
Rebuild Rate : 30%
PR Rate : 30%
BGI Rate : 30%
Check Consistency Rate : 30%
Reconstruction Rate : 30%
Cache Flush Interval : 4s
Max Drives to Spinup at One Time : 4
Delay Among Spinup Groups : 12s
Physical Drive Coercion Mode : 128MB
Cluster Mode : Disabled
Alarm : Disabled
Auto Rebuild : Enabled
Battery Warning : Enabled
Ecc Bucket Size : 15
Ecc Bucket Leak Rate : 1440 Minutes
Restore HotSpare on Insertion : Disabled
Expose Enclosure Devices : Disabled
Maintain PD Fail History : Disabled
Host Request Reordering : Enabled
Auto Detect BackPlane Enabled : SGPIO/i2c SEP
Load Balance Mode : Auto
Use FDE Only : Yes
Security Key Assigned : No
Security Key Failed : No
Security Key Not Backedup : No
Default LD PowerSave Policy : Controller Defined
Maximum number of direct attached drives to spin up in 1 min : 0
Auto Enhanced Import : No
Any Offline VD Cache Preserved : No
Allow Boot with Preserved Cache : No
Disable Online Controller Reset : No
PFK in NVRAM : No
Use disk activity for locate : No
POST delay : 90 seconds
BIOS Error Handling : Stop On Errors
Current Boot Mode :Normal
Capabilities
================
RAID Level Supported : RAID0, RAID1, RAID5, RAID6, RAID00, RAID10, RAID50, RAID60, PRL 11, PRL 11 with spanning, PRL11-RLQ0 DDF layout with no span, PRL11-RLQ0 DDF layout with span
Supported Drives : SAS, SATA
Allowed Mixing:
Mix in Enclosure Allowed
Status
================
ECC Bucket Count : 0
Limitations
================
Max Arms Per VD : 32
Max Spans Per VD : 8
Max Arrays : 128
Max Number of VDs : 64
Max Parallel Commands : 1008
Max SGE Count : 60
Max Data Transfer Size : 8192 sectors
Max Strips PerIO : 42
Max LD per array : 16
Min Strip Size : 8 KB
Max Strip Size : 1.0 MB
Max Configurable CacheCade Size: 0 GB
Current Size of CacheCade : 0 GB
Current Size of FW Cache : 0 MB
Device Present
================
Virtual Drives : 3
Degraded : 0
Offline : 0
Physical Devices : 4
Disks : 3
Critical Disks : 0
Failed Disks : 0
Supported Adapter Operations
================
Rebuild Rate : Yes
CC Rate : Yes
BGI Rate : Yes
Reconstruct Rate : Yes
Patrol Read Rate : Yes
Alarm Control : Yes
Cluster Support : No
BBU : Yes
Spanning : Yes
Dedicated Hot Spare : Yes
Revertible Hot Spares : Yes
Foreign Config Import : Yes
Self Diagnostic : Yes
Allow Mixed Redundancy on Array : No
Global Hot Spares : Yes
Deny SCSI Passthrough : No
Deny SMP Passthrough : No
Deny STP Passthrough : No
Support Security : Yes
Snapshot Enabled : No
Support the OCE without adding drives : Yes
Support PFK : No
Support PI : No
Support Boot Time PFK Change : No
Disable Online PFK Change : No
Support Shield State : No
Block SSD Write Disk Cache Change: No
Supported VD Operations
================
Read Policy : Yes
Write Policy : Yes
IO Policy : Yes
Access Policy : Yes
Disk Cache Policy : Yes
Reconstruction : Yes
Deny Locate : No
Deny CC : No
Allow Ctrl Encryption: No
Enable LDBBM : Yes
Support Breakmirror : No
Power Savings : No
Supported PD Operations
================
Force Online : Yes
Force Offline : Yes
Force Rebuild : Yes
Deny Force Failed : No
Deny Force Good/Bad : No
Deny Missing Replace : No
Deny Clear : No
Deny Locate : No
Support Temperature : No
NCQ : No
Disable Copyback : No
Enable JBOD : No
Enable Copyback on SMART : No
Enable Copyback to SSD on SMART Error : No
Enable SSD Patrol Read : No
PR Correct Unconfigured Areas : Yes
Enable Spin Down of UnConfigured Drives : Yes
Disable Spin Down of hot spares : No
Spin Down time : 30
T10 Power State : No
Error Counters
================
Memory Correctable Errors : 0
Memory Uncorrectable Errors : 0
Cluster Information
================
Cluster Permitted : No
Cluster Active : No
Default Settings
================
Phy Polarity : 0
Phy PolaritySplit : 0
Background Rate : 30
Strip Size : 64kB
Flush Time : 4 seconds
Write Policy : WB
Read Policy : Adaptive
Cache When BBU Bad : Disabled
Cached IO : No
SMART Mode : Mode 6
Alarm Disable : Yes
Coercion Mode : 128MB
ZCR Config : Unknown
Dirty LED Shows Drive Activity : No
BIOS Continue on Error : 0
Spin Down Mode : None
Allowed Device Type : SAS/SATA Mix
Allow Mix in Enclosure : Yes
Allow HDD SAS/SATA Mix in VD : No
Allow SSD SAS/SATA Mix in VD : No
Allow HDD/SSD Mix in VD : No
Allow SATA in Cluster : No
Max Chained Enclosures : 1
Disable Ctrl-R : No
Enable Web BIOS : No
Direct PD Mapping : Yes
BIOS Enumerate VDs : Yes
Restore Hot Spare on Insertion : No
Expose Enclosure Devices : No
Maintain PD Fail History : No
Disable Puncturing : No
Zero Based Enclosure Enumeration : Yes
PreBoot CLI Enabled : No
LED Show Drive Activity : Yes
Cluster Disable : Yes
SAS Disable : No
Auto Detect BackPlane Enable : SGPIO/i2c SEP
Use FDE Only : Yes
Enable Led Header : No
Delay during POST : 0
EnableCrashDump : No
Disable Online Controller Reset : No
EnableLDBBM : Yes
Un-Certified Hard Disk Drives : Allow
Treat Single span R1E as R10 : Yes
Max LD per array : 16
Power Saving option : All power saving options are disabled
Default spin down time in minutes: 30
Enable JBOD : No
Time taken to detect CME : 60s
Exit Code: 0x00
- logical disks details
ppol-3:~# megacli -ldinfo -lall -aall
Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name :
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Size : 465.25 GB
Sector Size : 512
Parity Size : 0
State : Optimal
Strip Size : 64 KB
Number Of Drives : 1
Span Depth : 1
Default Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Disk's Default
Encryption Type : None
Bad Blocks Exist: No
Is VD Cached: No
Virtual Drive: 1 (Target Id: 1)
Name :
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Size : 46.0 GB
Sector Size : 512
Parity Size : 0
State : Optimal
Strip Size : 64 KB
Number Of Drives : 1
Span Depth : 1
Default Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Disk's Default
Encryption Type : None
Bad Blocks Exist: No
Is VD Cached: No
Virtual Drive: 2 (Target Id: 2)
Name :
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Size : 46.0 GB
Sector Size : 512
Parity Size : 0
State : Optimal
Strip Size : 64 KB
Number Of Drives : 1
Span Depth : 1
Default Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Disk's Default
Encryption Type : None
Bad Blocks Exist: No
Is VD Cached: No
Exit Code: 0x00
Acknolegment
The ppol machines are reused machines of the former Pipol Cluster (continuous integration) of Inria Grenoble Rhone-Alpes (dismantled).
