VNX5500 Storage Pool offline. Any chance to raise?

Question

Dell

Situation: VNX5500 does not start normally after a regular power off. Both CSs start, DataMovers start, and the processor modules seem to start normally, but three of the four VolumePools do not see it. VP hang offline: An error occurred while accessing the Storage Pool. Please resolve any hardware problems and retry the operation. (0x712D8514)".
Obviously, there has been no support for a long time, tomorrow I will try to contact DELL, but it seems to me that the matter will be deaf.
What a shame - the shutdown was clearly regular, without jambs. Everything turned on fine too. Of course, it’s better not to touch such a thing at all , but there was a hopeless situation - the power was cut off for five hours, and so many ups do not pull under load ...
Maybe there is someone who knows?

Answer 1

The question is removed, I raised it myself. Slept for three days =)

Answer 2

Good afternoon!
Torturing VNX 5200, we have Pool 0 (offline) and Private Lun's (offline) - 2 pcs.
I connected via the service port and created a dialup connection,
then got into the console via ssh (Clariion, RemotelyAnywhere SSH Server OpenSSH 6.6),
I see that we have Microsoft Windows 6.1.7601 (Win7) at the root there is an EMC directory (C: \ EMC)
I did not find fcli, searched for fcli and flarecons, but found in the C:\EMC directory a bunch of console utilities
C:\EMC\Base\05330095.186\flare_driver\bin - ApmCli.exe, fbecli.exe, IdmCli.exe, MigCli.exe, PerResTool. exe
I think fbecli.exe is our fcli and flarecons
Through MluCLI and NaviSecCli - I think I went through all the possible options, unfortunately somewhere the deletion is cyclical (when entering the command, it constantly asks if you want to delete), somewhere it says that they say deleted but when checking we see that everything is in place (our offiline is in place ).

naviseccli lun -destroy -l 0
Are you sure you want to perform this operation?(y/n): y

naviseccli storagepool -destroy -id 0
Are you sure you want to perform this operation?(y/n): y
Caller not privileged.

mlucli destroyingpool -srvc_cmd -poolid 0

         Received Error Code = 0x712d8503
         Error: Could not destroy Slice Pool 0x0


mlucli removeflu -srvc_cmd -poolid 0 -lun 0

         Received Error Code = 0x712d8503
         Error: Could not remove flu from slice pool 0 (0x0) Status 0x712d8503.


mlucli deletepool -srvc_cmd -poolid 0 -force

         Received Error Code = 0x712d8801
         Error: Could not delete Slice Pool 0x0


mlucli getpoolprop -poolid 0

         Received Error Code = 0x712d8503
         Error Description = K10MLUAdminLogicalGet(): Poll Failed for Complete Object !
         No response received!! Status=0x712d8503
         Error: Could not get properties for Slice Pool 0x0.

mlucli shrinkpool -srvc_cmd -poolid 0 -lun 0

 getpoolstats -revid <Revision ID> -poolid <Pool ID>
mlucli getpoolstats -revid 0 -poolid 0

         Error: Could not get stats for slice pool 0 0x57.

FLU_WWN will help clear the cache with the command: mlucli clearcachedirty -srvc_cmd -revid 1 -wwn .. Found
it through fbecli.exe -> ls and li -lun 0-5, but these are service luns of my pool0. - they've been cleansed.
Below is a log of an attempt to clear my lun's - looking for the correct FLU_WNN - I think there are none.

C:\EMC>mlucli clearcachedirty -srvc_cmd -revid 0 -wwn 60:06:01:60:49:a0:46:00:a1:09:ab:53:f4:96:e8:11

Error: Could not handle clearing Cache-Dirty status of FLU 60:06:01:60:49:a0:46:00:a1:09:ab:53:f4:96:e8:11. 0x2

C:\EMC>mlucli destroylu -srvc_cmd -poolid 0 -lun 0 -force

K10 -- User-space Message Passing Service (UMps)
Compiled for Free (Retail) Build

         LU in Slice Pool 0x0 is unbound successfully.

C:\EMC>mlucli getowner -all
LU Number   VU OID      WWN                                               Current Owner
=========   =========   ===============================================   =============
        0   A00000001   60:06:01:60:49:a0:46:00:53:f3:37:3a:f4:96:e8:11   SPB
        1   A00000002   60:06:01:60:49:a0:46:00:a1:09:ab:53:f4:96:e8:11   SPB

Command: mlucli objectops -b_careful -info -oid 0x300000001 - sees that I have two luns in private state, and also see

Existant Dependent Objs: 8
                0x400000001
                0x400000002
                0x400000003
                0x400000004
                0x400000005
                0x400000006
                0x200000001
                0x200000002.

I think the first 6 lines are service information about my LUNs, and the remaining 2 lines (0x200000001 and 0x200000002) are my LUNs.
PS: The correct oid is worth looking through the mlucli polllu command - the Slice Pool ID line (I have it 0x300000001)

mlucli objectops -b_careful -info -oid 0x300000001

        MLU OBJECT INFO:
        Private State: 2
        Public State:  Error
        Target State:  Ready
        OwnerShip Model:  Object Mgr Controlled
                Leader Object: 0x0
        Status: 0xe12d8514
        Existant Dependent Objs: 8
                0x400000001
                0x400000002
                0x400000003
                0x400000004
                0x400000005
                0x400000006
                0x200000001
                0x200000002
        State Dependent Objs: 0

        Operation Completed Successfully.

mlucli objectops -b_careful -info -oid 0x400000001

        MLU OBJECT INFO:
        Private State: 2
        Public State:  Error
        Target State:  Ready
        OwnerShip Model:  Follow Leader
                Leader Object: 0x300000001
        Status: 0xe12d8516
        Existant Dependent Objs: 0
        State Dependent Objs: 0

        Operation Completed Successfully.

mlucli objectops -b_careful -info -oid 0x200000002

        MLU OBJECT INFO:
        Private State: 2
        Public State:  Error
        Target State:  Expired
        OwnerShip Model:  Object Mgr Controlled
                Leader Object: 0x0
        Status: 0xe12d8514
        Existant Dependent Objs: 0
        State Dependent Objs: 0

        Operation Completed Successfully.

C:\EMC>mlucli getdriverprops

====================================================
             MLU Driver Properties
====================================================

 Driver Version                         : 360
 Debug Level                            : DEFAULT
 Driver State                           : Ready
 Max. Pools                             : 15
 Max. LUs                               : 1000
 Max. LU Size                           : 549755813888 Blocks
 Min. LU Size                           : 1 Blocks
 Max Drives per Pool                    : 121
 Max Drives per Operation               : 80
 Max Drives in All Pools                : 121
 Max. Alias LUs                         : 8
 Current Pool Count                     : 1
 Current FLU Count                      : 6
 Current LU Count                       : 2
 Current TLU Count                      : 0
 Current DLU Count                      : 2
 Current SliceAdder Count               : 0
 Current ExistenceDependency Count      : 8
 Current Number of Drives in All Pools  : 3
 Compression                            : NOT Enabled
 Auto-Tiering                           : NOT Enabled
 Virtual Provisioning                   : NOT Enabled
 Background Task                        : NOT Running
 Background Task Rate                   : 0 (Medium)

From the command above we see that we have 6 pcs. FLU - how to display the ID of these FLUs?
Now the question is how to find FLU_WWN ?, LU_WWN is a little different.
Tell me where to dig to find out FLU_WWN?
You just need to delete offline(pool and private lun)!!
PS: Pool status: "Status: An internal error occurred resulting in a Pool lun going offline. (0x712d8514)"
You can also see this status by commands:

naviseccli storagepool -list
naviseccli lun -list
mlucli polllu

UID:  60:06:01:60:49:A0:46:00:A1:09:AB:53:F4:96:E8:11
LU WWN: 60:06:01:60:49:a0:46:00:53:f3:37:3a:f4:96:e8:11

One command writes that this is UID, the other command says that this is LU WWN,
we need a command that will show FLU WWN... respond people.... mlucli mpoll command - does not work
Got to the so-called fcli and flarecons, namely:
C:\EMC \Base\05330095.186\flare_driver\bin\fbecli.exe
Elevated permissions: (tried acc -m 1 and access -m 1 both commands elevate)

FBE_CLI>acc -m 1
FBE_CLI>ls
Type    Obj ID       ID    Object Info    Lifecycle State  Downstream                        Drives
                                                           Objects
-----------------------------------------------------------------------------------------------------------------------
пропустил лог выше...думаю это служебная информация о lun системы
Unknown  0x103 :259   SPECLZ
LUN      0x108 :264   0     381536 MB       FAIL             263                               (FAIL) (FAIL) (FAIL)
LUN      0x109 :265   1     381536 MB       FAIL             263                               (FAIL) (FAIL) (FAIL)
LUN      0x10a :266   2     381536 MB       FAIL             263                               (FAIL) (FAIL) (FAIL)
LUN      0x10b :267   3     381536 MB       FAIL             263                               (FAIL) (FAIL) (FAIL)
LUN      0x10c :268   4     381536 MB       FAIL             263                               (FAIL) (FAIL) (FAIL)
LUN      0x10d :269   5     381433 MB       FAIL             263                               (FAIL) (FAIL) (FAIL)
FBE_CLI>

Above log - took out three disks on which pool and lun, below log - returned the disks to their place.
The line with Unknown 0x103: 259 SPECLZ is confusing, I also think 6 lines below are FLU - service ones for my LUNs

Unknown  0x103 :259   SPECLZ
LUN      0x108 :264   0     381536 MB       READY            263                               0_0_6  0_0_5  0_0_4
LUN      0x109 :265   1     381536 MB       READY            263                               0_0_6  0_0_5  0_0_4
LUN      0x10a :266   2     381536 MB       READY            263                               0_0_6  0_0_5  0_0_4
LUN      0x10b :267   3     381536 MB       READY            263                               0_0_6  0_0_5  0_0_4
LUN      0x10c :268   4     381536 MB       READY            263                               0_0_6  0_0_5  0_0_4
LUN      0x10d :269   5     381433 MB       READY            263                               0_0_6  0_0_5  0_0_4
FBE_CLI>

Below wwn - this is our FLU_WWN - I cleared the cache using them, through mlucli.exe - but still I can’t delete the ill-fated lun ... help .... ahh ...

FBE_CLI>li -lun 0
Lun information:
----------------
Logical Unit:    0
  Lun Name:        Virtual Disk_63993
  Raid type:       0x5 [RAID-5]
  Lifecycle State: 2 [READY]
  Lun Object-id:   0x108
  Offset:          0x0
  Capacity:        0x2ddb9200
  world_wide_name: 60:06:01:60:0a:a0:46:00:57:ae:b0:b3:f3:96:e8:11: