wiki:twRAID

Version 1 (modified by niles, 4 months ago) (diff)

--

Setting up a RAID with 3ware's tw_cli tool

You typically start off just having plugged in your new disks, so your RAID might look something like :

[root@netdrms02 ~]# tw_cli /c0 show

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    RAID-1    OK             -       -       -       223.507   RiW    ON     

VPort Status         Unit Size      Type  Phy Encl-Slot    Model
------------------------------------------------------------------------------
p0    OK             -    3.63 TB   SATA  0   -            WDC WD4002FYYZ-01B7 
p1    OK             -    3.63 TB   SATA  1   -            WDC WD4002FYYZ-01B7 
p2    OK             -    3.63 TB   SATA  2   -            WDC WD4002FYYZ-01B7 
p3    OK             -    3.63 TB   SATA  3   -            WDC WD4002FYYZ-01B7 
p4    OK             -    3.63 TB   SATA  4   -            WDC WD4002FYYZ-01B7 
p5    OK             -    3.63 TB   SATA  5   -            WDC WD4002FYYZ-01B7 
p6    OK             u0   223.57 GB SATA  6   -            INTEL SSDSC2BB240G7 
p7    OK             u0   223.57 GB SATA  7   -            INTEL SSDSC2BB240G7 

We can see that there are six new Western Digital disks and two already existing INTEL disks.

By the way, if your controller does not show up as /c0 then do "tw_cli show" to see available controllers, which in this case shows :

[root@netdrms02 ~]# tw_cli show

Ctl   Model        (V)Ports  Drives   Units   NotOpt  RRate   VRate  BBU
------------------------------------------------------------------------
c0    9750-8i      8         8        4       0       1       1      -        

Anyway, we are on controller /c0. We want to add a new RAID. I decided to add RAID type 5 using the first 4 disks, and using the last two new disks as spares. First, add the new RAID using the first 4 disks :

[root@netdrms02 ~]# tw_cli /c0 add type=raid5 disk=0-3 stripe=256 name=INTERNAL autoverify
Creating new unit on controller /c0 ... Done. The new unit is /c0/u1.
Naming unit /c0/u1 to [INTERNAL] ... Done.
Setting AutoVerify=ON for the new unit ... Done.
Setting default Storsave policy to [balance] for the new unit ... Done.
Setting default Command Queuing policy for unit /c0/u1 to [on] ... Done.
Setting write cache = ON for the new unit ... Done.
Warning: You do not have a battery backup unit for /c0/u1 and the enabled
write cache (default) may cause data loss in the event of power failure.

And after that we have :

[root@netdrms02 ~]# tw_cli /c0 show

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    RAID-1    OK             -       -       -       223.507   RiW    ON     
u1    RAID-5    OK             -       -       256K    11175.8   RiW    ON     

VPort Status         Unit Size      Type  Phy Encl-Slot    Model
------------------------------------------------------------------------------
p0    OK             u1   3.63 TB   SATA  0   -            WDC WD4002FYYZ-01B7 
p1    OK             u1   3.63 TB   SATA  1   -            WDC WD4002FYYZ-01B7 
p2    OK             u1   3.63 TB   SATA  2   -            WDC WD4002FYYZ-01B7 
p3    OK             u1   3.63 TB   SATA  3   -            WDC WD4002FYYZ-01B7 
p4    OK             -    3.63 TB   SATA  4   -            WDC WD4002FYYZ-01B7 
p5    OK             -    3.63 TB   SATA  5   -            WDC WD4002FYYZ-01B7 
p6    OK             u0   223.57 GB SATA  6   -            INTEL SSDSC2BB240G7 
p7    OK             u0   223.57 GB SATA  7   -            INTEL SSDSC2BB240G7 

So we then want to add the last two disks as spares :

[root@netdrms02 ~]# tw_cli /c0 add type=spare disk=4
Creating new unit on controller /c0 ... Done. The new unit is /c0/u2.
WARNING: This Spare unit may replace failed drive of same interface type only.

[root@netdrms02 ~]# tw_cli /c0 add type=spare disk=5
Creating new unit on controller /c0 ... Done. The new unit is /c0/u3.
WARNING: This Spare unit may replace failed drive of same interface type only.

Which gives us :

[root@netdrms02 ~]# tw_cli /c0 show

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    RAID-1    OK             -       -       -       223.507   RiW    ON     
u1    RAID-5    OK             -       -       256K    11175.8   RiW    ON     
u2    SPARE     OK             -       -       -       3726.01   -      OFF    
u3    SPARE     OK             -       -       -       3726.01   -      OFF    

VPort Status         Unit Size      Type  Phy Encl-Slot    Model
------------------------------------------------------------------------------
p0    OK             u1   3.63 TB   SATA  0   -            WDC WD4002FYYZ-01B7 
p1    OK             u1   3.63 TB   SATA  1   -            WDC WD4002FYYZ-01B7 
p2    OK             u1   3.63 TB   SATA  2   -            WDC WD4002FYYZ-01B7 
p3    OK             u1   3.63 TB   SATA  3   -            WDC WD4002FYYZ-01B7 
p4    OK             u2   3.63 TB   SATA  4   -            WDC WD4002FYYZ-01B7 
p5    OK             u3   3.63 TB   SATA  5   -            WDC WD4002FYYZ-01B7 
p6    OK             u0   223.57 GB SATA  6   -            INTEL SSDSC2BB240G7 
p7    OK             u0   223.57 GB SATA  7   -            INTEL SSDSC2BB240G7 

At this point it's worth checking the settings on the new RAID :

[root@netdrms02 ~]# tw_cli /c0/u1 show all
/c0/u1 status = OK
/c0/u1 is not rebuilding, its current state is OK
/c0/u1 is not verifying, its current state is OK
/c0/u1 is not initialized.
/c0/u1 Write Cache = on
/c0/u1 Read Cache = Intelligent
/c0/u1 volume(s) = 1
/c0/u1 name = INTERNAL             
/c0/u1 serial number = K3GVTMABEA21C600C615 
/c0/u1 Ignore ECC policy = off       
/c0/u1 Auto Verify Policy = on        
/c0/u1 Storsave Policy = balance     
/c0/u1 Command Queuing Policy = on        
/c0/u1 Rapid RAID Recovery setting = all

Unit     UnitType  Status         %RCmpl  %V/I/M  VPort Stripe  Size(GB)
------------------------------------------------------------------------
u1       RAID-5    OK             -       -       -     256K    11175.8   
u1-0     DISK      OK             -       -       p0    -       3725.28   
u1-1     DISK      OK             -       -       p1    -       3725.28   
u1-2     DISK      OK             -       -       p2    -       3725.28   
u1-3     DISK      OK             -       -       p3    -       3725.28   
u1/v0    Volume    -              -       -       -     -       11175.8   

Looks good.

When we created the RAID, it should have attached itself on the bus. The steps from here on in are similar to those for other RAID controllers. Use "dmesg" to figure out where it is attached :

[root@netdrms02 ~]# dmesg | tail -6
[4656757.804513] sd 0:0:1:0: [sdc] 23437436928 512-byte logical blocks: (11.9 TB/10.9 TiB)
[4656757.804557] sd 0:0:1:0: Attached scsi generic sg6 type 0
[4656757.849486] sd 0:0:1:0: [sdc] Write Protect is off
[4656757.849494] sd 0:0:1:0: [sdc] Mode Sense: 23 00 10 00
[4656757.880591] sd 0:0:1:0: [sdc] Write cache: disabled, read cache: enabled, supports DPO and FUA
[4656758.017518] sd 0:0:1:0: [sdc] Attached SCSI disk

So it is attached at /dev/sdc. Build a partition on it :

[root@netdrms02 ~]# parted /dev/sdc
GNU Parted 3.1
Using /dev/sdc
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel gpt                                                      
(parted) mkpart primary xfs 0% 100%                                       
(parted) quit                                                             
Information: You may need to update /etc/fstab.

Then make a filesystem on that partition :

[root@netdrms02 ~]# mkfs -t xfs /dev/sdc1
meta-data=/dev/sdc1              isize=512    agcount=11, agsize=268435455 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=2929679104, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Mount it to test, and then unmount it :

[root@netdrms02 ~]# mkdir /internalRAID; mount -t xfs /dev/sdc1 /internalRAID
[root@netdrms02 ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   50G  8.7G   42G  18% /
devtmpfs                  63G     0   63G   0% /dev
tmpfs                     63G  8.0K   63G   1% /dev/shm
tmpfs                     63G  634M   63G   1% /run
tmpfs                     63G     0   63G   0% /sys/fs/cgroup
/dev/sdb1               1014M  292M  723M  29% /boot
/dev/mapper/centos-home  169G  388M  169G   1% /home
/dev/sda1                117T   12T  106T  10% /SUM01
tmpfs                     13G   12K   13G   1% /run/user/42
tmpfs                     13G     0   13G   0% /run/user/1000
/dev/sdc1                 11T   33M   11T   1% /internalRAID

[root@netdrms02 ~]# umount /internalRAID

Get the UUID for that partition with blkid :

[root@netdrms02 ~]# blkid /dev/sdc1
/dev/sdc1: UUID="dadbf9b9-1110-4df0-933a-887ffdb3860c" TYPE="xfs" PARTLABEL="primary" PARTUUID="18fe5bbf-5798-4b62-8407-628bc5fb84ad" 

And edit /etc/fstab to add the mount. The new entry is the last one that mounts at /internalRAID :

[root@netdrms02 ~]# vi /etc/fstab
[root@netdrms02 ~]# cat /etc/fstab 

#
# /etc/fstab
# Created by anaconda on Wed Jan 31 16:02:17 2018
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root /                       xfs     defaults        0 0
UUID=d0c1aad0-8c6a-4881-87d7-deb96c160c2a /boot                   xfs     defaults        0 0
/dev/mapper/centos-home /home                   xfs     defaults        0 0
/dev/mapper/centos-swap swap                    swap    defaults        0 0

# SUMS partition
UUID=2b598aba-0b60-4966-a443-90c9ca730974 /SUM01           xfs     defaults        1 1

# Internal RAID
UUID=dadbf9b9-1110-4df0-933a-887ffdb3860c /internalRAID    xfs     defaults        1 1

Then mount everything :

[root@netdrms02 ~]# mount -a
[root@netdrms02 ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   50G  8.7G   42G  18% /
devtmpfs                  63G     0   63G   0% /dev
tmpfs                     63G  8.0K   63G   1% /dev/shm
tmpfs                     63G  634M   63G   1% /run
tmpfs                     63G     0   63G   0% /sys/fs/cgroup
/dev/sdb1               1014M  292M  723M  29% /boot
/dev/mapper/centos-home  169G  388M  169G   1% /home
/dev/sda1                117T   12T  106T  10% /SUM01
tmpfs                     13G   12K   13G   1% /run/user/42
tmpfs                     13G     0   13G   0% /run/user/1000
/dev/sdc1                 11T   33M   11T   1% /internalRAID

Yay! Big new disk.

Niles Oien May 2018