| Version 2 (modified by niles, 6 years ago) (diff) |
|---|
Setting up a RAID with 3ware's tw_cli tool
You typically start off just having plugged in your new disks, so your RAID might look something like :
[root@netdrms02 ~]# tw_cli /c0 show Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy ------------------------------------------------------------------------------ u0 RAID-1 OK - - - 223.507 RiW ON VPort Status Unit Size Type Phy Encl-Slot Model ------------------------------------------------------------------------------ p0 OK - 3.63 TB SATA 0 - WDC WD4002FYYZ-01B7 p1 OK - 3.63 TB SATA 1 - WDC WD4002FYYZ-01B7 p2 OK - 3.63 TB SATA 2 - WDC WD4002FYYZ-01B7 p3 OK - 3.63 TB SATA 3 - WDC WD4002FYYZ-01B7 p4 OK - 3.63 TB SATA 4 - WDC WD4002FYYZ-01B7 p5 OK - 3.63 TB SATA 5 - WDC WD4002FYYZ-01B7 p6 OK u0 223.57 GB SATA 6 - INTEL SSDSC2BB240G7 p7 OK u0 223.57 GB SATA 7 - INTEL SSDSC2BB240G7
We can see that there are six new Western Digital disks and two already existing INTEL disks.
By the way, if your controller does not show up as /c0 then do "tw_cli show" to see available controllers, which in this case shows :
[root@netdrms02 ~]# tw_cli show Ctl Model (V)Ports Drives Units NotOpt RRate VRate BBU ------------------------------------------------------------------------ c0 9750-8i 8 8 4 0 1 1 -
Anyway, we are on controller /c0. We want to add a new RAID. I decided to add RAID type 5 using the first 4 disks, and using the last two new disks as spares. First, add the new RAID using the first 4 disks :
[root@netdrms02 ~]# tw_cli /c0 add type=raid5 disk=0-3 stripe=256 name=INTERNAL autoverify Creating new unit on controller /c0 ... Done. The new unit is /c0/u1. Naming unit /c0/u1 to [INTERNAL] ... Done. Setting AutoVerify=ON for the new unit ... Done. Setting default Storsave policy to [balance] for the new unit ... Done. Setting default Command Queuing policy for unit /c0/u1 to [on] ... Done. Setting write cache = ON for the new unit ... Done. Warning: You do not have a battery backup unit for /c0/u1 and the enabled write cache (default) may cause data loss in the event of power failure.
And after that we have :
[root@netdrms02 ~]# tw_cli /c0 show Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy ------------------------------------------------------------------------------ u0 RAID-1 OK - - - 223.507 RiW ON u1 RAID-5 OK - - 256K 11175.8 RiW ON VPort Status Unit Size Type Phy Encl-Slot Model ------------------------------------------------------------------------------ p0 OK u1 3.63 TB SATA 0 - WDC WD4002FYYZ-01B7 p1 OK u1 3.63 TB SATA 1 - WDC WD4002FYYZ-01B7 p2 OK u1 3.63 TB SATA 2 - WDC WD4002FYYZ-01B7 p3 OK u1 3.63 TB SATA 3 - WDC WD4002FYYZ-01B7 p4 OK - 3.63 TB SATA 4 - WDC WD4002FYYZ-01B7 p5 OK - 3.63 TB SATA 5 - WDC WD4002FYYZ-01B7 p6 OK u0 223.57 GB SATA 6 - INTEL SSDSC2BB240G7 p7 OK u0 223.57 GB SATA 7 - INTEL SSDSC2BB240G7
So we then want to add the last two disks as spares :
[root@netdrms02 ~]# tw_cli /c0 add type=spare disk=4 Creating new unit on controller /c0 ... Done. The new unit is /c0/u2. WARNING: This Spare unit may replace failed drive of same interface type only. [root@netdrms02 ~]# tw_cli /c0 add type=spare disk=5 Creating new unit on controller /c0 ... Done. The new unit is /c0/u3. WARNING: This Spare unit may replace failed drive of same interface type only.
Which gives us :
[root@netdrms02 ~]# tw_cli /c0 show Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy ------------------------------------------------------------------------------ u0 RAID-1 OK - - - 223.507 RiW ON u1 RAID-5 OK - - 256K 11175.8 RiW ON u2 SPARE OK - - - 3726.01 - OFF u3 SPARE OK - - - 3726.01 - OFF VPort Status Unit Size Type Phy Encl-Slot Model ------------------------------------------------------------------------------ p0 OK u1 3.63 TB SATA 0 - WDC WD4002FYYZ-01B7 p1 OK u1 3.63 TB SATA 1 - WDC WD4002FYYZ-01B7 p2 OK u1 3.63 TB SATA 2 - WDC WD4002FYYZ-01B7 p3 OK u1 3.63 TB SATA 3 - WDC WD4002FYYZ-01B7 p4 OK u2 3.63 TB SATA 4 - WDC WD4002FYYZ-01B7 p5 OK u3 3.63 TB SATA 5 - WDC WD4002FYYZ-01B7 p6 OK u0 223.57 GB SATA 6 - INTEL SSDSC2BB240G7 p7 OK u0 223.57 GB SATA 7 - INTEL SSDSC2BB240G7
At this point it's worth checking the settings on the new RAID :
[root@netdrms02 ~]# tw_cli /c0/u1 show all /c0/u1 status = OK /c0/u1 is not rebuilding, its current state is OK /c0/u1 is not verifying, its current state is OK /c0/u1 is not initialized. /c0/u1 Write Cache = on /c0/u1 Read Cache = Intelligent /c0/u1 volume(s) = 1 /c0/u1 name = INTERNAL /c0/u1 serial number = K3GVTMABEA21C600C615 /c0/u1 Ignore ECC policy = off /c0/u1 Auto Verify Policy = on /c0/u1 Storsave Policy = balance /c0/u1 Command Queuing Policy = on /c0/u1 Rapid RAID Recovery setting = all Unit UnitType Status %RCmpl %V/I/M VPort Stripe Size(GB) ------------------------------------------------------------------------ u1 RAID-5 OK - - - 256K 11175.8 u1-0 DISK OK - - p0 - 3725.28 u1-1 DISK OK - - p1 - 3725.28 u1-2 DISK OK - - p2 - 3725.28 u1-3 DISK OK - - p3 - 3725.28 u1/v0 Volume - - - - - 11175.8
Looks good.
When we created the RAID, it should have attached itself on the bus. The steps from here on in are similar to those for other RAID controllers. Use "dmesg" to figure out where it is attached :
[root@netdrms02 ~]# dmesg | tail -6 [4656757.804513] sd 0:0:1:0: [sdc] 23437436928 512-byte logical blocks: (11.9 TB/10.9 TiB) [4656757.804557] sd 0:0:1:0: Attached scsi generic sg6 type 0 [4656757.849486] sd 0:0:1:0: [sdc] Write Protect is off [4656757.849494] sd 0:0:1:0: [sdc] Mode Sense: 23 00 10 00 [4656757.880591] sd 0:0:1:0: [sdc] Write cache: disabled, read cache: enabled, supports DPO and FUA [4656758.017518] sd 0:0:1:0: [sdc] Attached SCSI disk
So it is attached at /dev/sdc. Build a partition on it :
[root@netdrms02 ~]# parted /dev/sdc GNU Parted 3.1 Using /dev/sdc Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) mklabel gpt (parted) mkpart primary xfs 0% 100% (parted) quit Information: You may need to update /etc/fstab.
Then make a filesystem on that partition :
[root@netdrms02 ~]# mkfs -t xfs /dev/sdc1
meta-data=/dev/sdc1 isize=512 agcount=11, agsize=268435455 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=2929679104, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=521728, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Mount it to test, and then unmount it :
[root@netdrms02 ~]# mkdir /internalRAID; mount -t xfs /dev/sdc1 /internalRAID [root@netdrms02 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos-root 50G 8.7G 42G 18% / devtmpfs 63G 0 63G 0% /dev tmpfs 63G 8.0K 63G 1% /dev/shm tmpfs 63G 634M 63G 1% /run tmpfs 63G 0 63G 0% /sys/fs/cgroup /dev/sdb1 1014M 292M 723M 29% /boot /dev/mapper/centos-home 169G 388M 169G 1% /home /dev/sda1 117T 12T 106T 10% /SUM01 tmpfs 13G 12K 13G 1% /run/user/42 tmpfs 13G 0 13G 0% /run/user/1000 /dev/sdc1 11T 33M 11T 1% /internalRAID [root@netdrms02 ~]# umount /internalRAID
Get the UUID for that partition with blkid :
[root@netdrms02 ~]# blkid /dev/sdc1 /dev/sdc1: UUID="dadbf9b9-1110-4df0-933a-887ffdb3860c" TYPE="xfs" PARTLABEL="primary" PARTUUID="18fe5bbf-5798-4b62-8407-628bc5fb84ad"
And edit /etc/fstab to add the mount. The new entry is the last one that mounts at /internalRAID :
[root@netdrms02 ~]# vi /etc/fstab [root@netdrms02 ~]# cat /etc/fstab # # /etc/fstab # Created by anaconda on Wed Jan 31 16:02:17 2018 # # Accessible filesystems, by reference, are maintained under '/dev/disk' # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info # /dev/mapper/centos-root / xfs defaults 0 0 UUID=d0c1aad0-8c6a-4881-87d7-deb96c160c2a /boot xfs defaults 0 0 /dev/mapper/centos-home /home xfs defaults 0 0 /dev/mapper/centos-swap swap swap defaults 0 0 # SUMS partition UUID=2b598aba-0b60-4966-a443-90c9ca730974 /SUM01 xfs defaults 1 1 # Internal RAID UUID=dadbf9b9-1110-4df0-933a-887ffdb3860c /internalRAID xfs defaults 1 1
Then mount everything :
[root@netdrms02 ~]# mount -a [root@netdrms02 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos-root 50G 8.7G 42G 18% / devtmpfs 63G 0 63G 0% /dev tmpfs 63G 8.0K 63G 1% /dev/shm tmpfs 63G 634M 63G 1% /run tmpfs 63G 0 63G 0% /sys/fs/cgroup /dev/sdb1 1014M 292M 723M 29% /boot /dev/mapper/centos-home 169G 388M 169G 1% /home /dev/sda1 117T 12T 106T 10% /SUM01 tmpfs 13G 12K 13G 1% /run/user/42 tmpfs 13G 0 13G 0% /run/user/1000 /dev/sdc1 11T 33M 11T 1% /internalRAID
Yay! Big new disk.
Note : the 3ware software does write a log file, it will be named something like /var/log/tdm_aen_SV54738544.txt (the exact name likely depends on the details of the 3ware card). You can thus do things like look at the log file and leave out the INFORMATION only messages :
vso03:~ # cat /var/log/tdm_aen_SV54738544.txt | grep -v INFORMATION 20180830151936 WARNING Incomplete unit detected: unit=1 20180830000848 WARNING Drive removed: phy=2 20180830000807 ERROR Degraded unit: unit=1, vport=2 20180823031251 WARNING Drive removed: phy=2 20180823031152 ERROR Degraded unit: unit=1, vport=2 20180804062046 WARNING Drive removed: phy=2 20180804062005 ERROR Degraded unit: unit=1, vport=2 20180709002010 WARNING Drive removed: phy=2 20180709001930 ERROR Degraded unit: unit=1, vport=2
Niles Oien May 2018
