Problem

Bifrost's Ironic API (port 6385) architecture:

Client → nginx (TLS/HTTPS, port 6385) → unix socket → ironic process

The ironic process died, so /run/ironic/ironic.socket did not exist. Nginx returned 502 Bad Gateway.

connect() to unix:/run/ironic/ironic.socket failed (2: No such file or directory)

Resolution

Restarting the bifrost_deploy container re-spawned the ironic process and recreated the socket:

sudo docker restart bifrost_deploy

Verified with curl -vk https://10.72.1.11:6385/ → 200 OK.

⚠️ Root cause: oslo_config.cfg.NoSuchOptError: no such option unix_socket — suspected ironic.conf / ironic version compatibility issue.

Problem

Reprovision failed because the downloaded image checksum didn't match what Ironic expected:

Deploy step deploy.write_image failed.
Image failed to verify against checksum.
image checksum:        ea3eb0a7273eda10864d531a79390ef9
verification checksum: c131c2d6ae256de7d4a42a91046c5aa2

Even when deploy passed, SSH timed out waiting for port 22 (10 min timeout).

Resolution

1. Restarted ironic-inspector → undeploy → reprovision
2. During redeploy, Disk UUID changed → Ceph OSD lost track of the disk
3. Removed old OSD and re-added the disk with new UUID

Final: ping/SSH ✅, OpenStack ✅

Problem

Node booted into TinyCore mode instead of the actual OS — meaning provisioning never completed. IPMI showed power on, but no real OS was installed.

Reprovision attempt failed because Ironic API was down (bifrost_deploy container issue). After API recovery, SSH showed host key mismatch:

WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!

Resolution

ssh-keygen -f "/home/cloud/.ssh/known_hosts" -R "10.71.206.114"
sudo docker restart bifrost_deploy
kayobe overcloud provision --limit compute-206-114

Final: ping/SSH ✅, OpenStack ✅

Problem

Kayobe provisioning failed with SDK version incompatibility:

Incompatible openstacksdk library found:
Version MUST be >= 1.0 ... but 0.62.0 is smaller than minimum version 1.0.

Resolution

Upgraded openstacksdk in the Kayobe virtual environment:

sudo /opt/kayobe/venvs/openstacksdk/bin/pip install -U 'openstacksdk>=1.0,<2.0'
sudo /opt/kayobe/venvs/openstacksdk/bin/pip check

Also resolved recurring ironic.service timeout (Permission denied on socket → service restart fixed it).

Final: ping/SSH ✅, OpenStack ✅