My VIO lab broke after a power failure. I powered on and brought back the vsphere infrastructure, Openstack Manager and the VIO VM’s, however I was unable to login to Openstack.

So I decided to do a clean shut down/start up of the VIO deployment from the VIO plugin in the vsphere web client.
The VIO deployment shut down fine. However, powering on the deployment failed with an error pointing to the DB-01 VM.

From the /var/log/oms/oms.log file in the Management Server the following errors were seen:

[2015-06-05T02:01:34.688+0000] ERROR SimpleAsyncTaskExecutor-4| com.vmware.openstack.aop.logging.ExceptionHandlerAspect: Service error  
com.vmware.openstack.exception.ClusteringServiceException: Failed cluster operation for cluster VIO.  
        at com.vmware.openstack.exception.ClusteringServiceException.CLUSTER\_OPERATION\_FAILED(ClusteringServiceException.java:109)  
        at com.vmware.openstack.service.job.ClusterNodeStatusVerifyStep.executeStep(ClusterNodeStatusVerifyStep.java:52)  
        at com.vmware.openstack.service.job.TrackableTasklet.execute(TrackableTasklet.java:37)  
[2015-06-05T02:01:34.690+0000] ERROR SimpleAsyncTaskExecutor-4| org.springframework.batch.core.step.AbstractStep: Encountered an error executing step startClusterVerifyNodeStatus in job startClusterJob  
com.vmware.openstack.exception.ClusteringServiceException: Failed cluster operation for cluster VIO.

I logged into the db-01 vm and noticed that mysql was stopped. I tried starting mysql with the command “service mysql start” and it failed without any errors.

After quite a bit of digging around, I found out that the mysql cluster needed to be recovered.  In order to recover the cluster:

  1. Run mysqld_safe --wsrep-recover on each of the db VMs.
  2. The command will print a line that is similar to, Recovered position 2c03837b-faa5-11e4-88cb-3e24d141b97b:1066549
  3. Compare the number after the colon, “:”, and choose the VM with the highest number to run the next set of instructions. If the number is equal then choose the first db VM.
  4. Stop the mysql service with the command, sudo service mysql stop
  5. Start the mysql service using the command, sudo service mysql start --wsrep-new-cluster. You can look at /var/log/syslog in the db VM for the mysql startup.
  6. On the other db VM’s start the service with the command sudo service mysql start

In order to get rid of the error in the vSphere Web client,

  1. Login to the Management Server as root.
  2. Run the following command to start psql,  /opt/vmware/vpostgres/current/bin/psql -U omsdb
  3. At the prompt type the following query to update the status, update cluster set status='RUNNING';
  4. Quit psql by typing \q
  5. In the vSphere WebClient, stop the openstack deployment, after that completes start the openstack deployment.