Learn how to work around the Ubuntu Noble FIPS mode bug that prevents OpenStack Epoxy deployments by creating a custom Docker base image.

1. Overview - Understanding the FIPS mode bug
2. Prerequisites - What you need before starting
3. Creating the Custom Base Image - Building the Dockerfile
4. Ubuntu Pro Configuration - Setting up Pro services
5. Building and Testing - Creating and verifying the image
6. Registry Deployment - Pushing to your registry
7. Kolla Build Integration - Using with OpenStack Epoxy

Overview

This guide demonstrates how to work around a known bug in Ubuntu Noble that prevents FIPS 140-2 compliant OpenStack deployments using Epoxy 2025.1 with Kolla/Kolla-Ansible. The issue occurs when the ca-certificates package attempts to update certificates in FIPS mode, causing immediate out-of-memory errors that prevent Kolla's image builder from completing successfully.

The Problem: When building OpenStack images on Ubuntu Noble in FIPS mode, you'll encounter errors like:

#5 406.5 Updating certificates in /etc/ssl/certs...
#5 407.0 out of memory
#5 407.0 out of memory
#5 407.0 out of memory
#5 407.0 out of memory

This is caused by Bug #2066990 in the Ubuntu ca-certificates package, which affects FIPS-mode containers running on FIPS-enabled hosts.

The Solution: Create a custom Docker base image that:

• Uses Ubuntu Pro to get FIPS-compliant libraries
• Disables OpenSSL FIPS mode checks via environment variable
• Provides a stable foundation for OpenStack Epoxy deployments with Kolla/Kolla-Ansible

This workaround is essential for organizations requiring FIPS 140-2 compliance in their OpenStack environments using Kolla as the deployment method.

1. PREREQUISITES

Before implementing this workaround, ensure you have:

• Ubuntu Pro Subscription: Active Ubuntu Pro subscription with FIPS updates enabled
• FIPS-Enabled Host: Ubuntu host with FIPS mode enabled
• Docker with BuildKit: Docker installation with BuildKit support
• Registry Access: Access to a container registry for storing custom images
• OpenStack Epoxy with Kolla: Epoxy 2025.1 release for OpenStack deployment using Kolla/Kolla-Ansible

Required Information:

• Ubuntu Pro token for guest attachment
• Container registry URL and credentials
• Understanding of Docker build processes
• Access to the FIPS-enabled build environment

2. CREATING THE CUSTOM BASE IMAGE

The solution involves creating a custom Docker base image that combines Ubuntu Pro with disabled FIPS mode checks.

Step 1: Create the Dockerfile

Create a Dockerfile with the following content:

FROM ubuntu:noble

ENV DEBIAN_FRONTEND=noninteractive
ENV TZ=America/Chicago
ENV OPENSSL_FORCE_FIPS_MODE=0

RUN --mount=type=secret,id=pro-attach-config \
   apt-get update -y \
   && apt-get install -y ubuntu-pro-client ca-certificates \
   && apt-get upgrade -y  \
   && pro attach --attach-config /run/secrets/pro-attach-config \
   && apt-get upgrade -y  \
   && apt-get install -y openssl \
   && pro detach --assume-yes \
   && apt-get purge --auto-remove -y ubuntu-pro-client \
   && rm -rf /var/lib/apt/lists/*

Key Components:

• Base Image: Uses ubuntu:noble as the foundation
• Environment Variables: Sets timezone and disables FIPS mode checks
• Ubuntu Pro Integration: Attaches to Pro services during build
• Cleanup: Removes Pro client after installation to reduce image size

3. UBUNTU PRO CONFIGURATION

To use Ubuntu Pro services in the container, you need to generate a guest token and create a configuration file.

Step 1: Generate Guest Token

On your FIPS-enabled Ubuntu Pro host, generate a guest token:

pro api u.pro.attach.guest.get_guest_token.v1 | jq -r '.data.attributes.guest_token'

Step 2: Create Pro Configuration

Create a pro-attach-config.yaml file with the following content:

token: <your-guest-token>
enabled_services:
- esm-infra
- esm-apps
- fips-updates

Configuration Details:

• Token: The guest token generated in the previous step
• ESM Infra: Extended Security Maintenance for infrastructure packages
• ESM Apps: Extended Security Maintenance for application packages
• FIPS Updates: FIPS 140-2 compliant package updates

This configuration is based on the Ubuntu Pro documentation for creating FIPS Docker images.

4. BUILDING AND TESTING

Build the custom base image and verify it works correctly.

Step 1: Build the Image

Build the image using Docker BuildKit with the secret mount:

DOCKER_BUILDKIT=1 docker build . --secret id=pro-attach-config,src=pro-attach-config.yaml --network host -t ubuntu-pro:noble

Build Parameters:

• DOCKER_BUILDKIT=1: Enables BuildKit for advanced features
• --secret: Mounts the Pro configuration as a secret
• --network host: Uses host networking (may not be necessary in all environments)
• -t ubuntu-pro:noble: Tags the image for local use

Step 2: Test the Image

Verify the image works by testing OpenSSL functionality:

docker run --rm -it --network host ubuntu-pro:noble
echo | openssl s_client ubuntu.com:443

Expected Results:

• Container should start without out-of-memory errors
• OpenSSL should successfully connect to external services
• No FIPS mode related errors should occur

5. REGISTRY DEPLOYMENT

Once the image is built and tested, push it to your container registry.

Step 1: Tag for Registry

Tag the image for your registry:

docker tag ubuntu-pro:noble registry.company.local:5000/kolla/ubuntu-pro:noble

Tagging Notes:

• Replace registry.company.local:5000 with your registry URL
• The kolla namespace may not be necessary but helps organize images
• Use a consistent naming convention for your organization

Step 2: Push to Registry

Push the image to your registry:

docker push registry.company.local:5000/kolla/ubuntu-pro:noble

Registry Considerations:

• Ensure you have proper authentication configured
• Verify the image is accessible from your build environment
• Consider implementing image scanning for security compliance

6. KOLLA BUILD INTEGRATION

Use the custom base image with Kolla to build OpenStack images.

Step 1: Build OpenStack Images

Run the kolla-build command with your custom base image:

DOCKER_BUILDKIT=1 kolla-build -b ubuntu --tag 2025.1-ubuntu-noble-pro --base-image registry.company.local:5000/kolla/ubuntu-pro --base-tag noble --registry registry.company.local:5000 --push -n kolla --nopull

Command Parameters:

• -b ubuntu: Specifies Ubuntu as the base distribution
• --tag: Sets the tag for the built images
• --base-image: Points to your custom base image
• --base-tag: Specifies the tag of the base image
• --registry: Target registry for pushing images
• --push: Automatically pushes built images
• -n kolla: Namespace for the images
• --nopull: Skips pulling existing images

Step 2: Verify Build Success

Monitor the build process to ensure:

• No out-of-memory errors occur during certificate updates
• All OpenStack services build successfully
• Images are properly tagged and pushed to the registry

SUMMARY

This guide demonstrated how to work around the Ubuntu Noble FIPS mode bug that prevents OpenStack Epoxy deployments using Kolla/Kolla-Ansible, including:

1. Problem Identification: Understanding the ca-certificates bug in FIPS mode
2. Custom Base Image: Creating a Dockerfile with Ubuntu Pro and disabled FIPS checks
3. Ubuntu Pro Setup: Configuring Pro services and generating guest tokens
4. Build Process: Building and testing the custom base image
5. Registry Deployment: Pushing the image to your container registry
6. Kolla Integration: Using the custom base image for OpenStack builds

The result is a working FIPS 140-2 compliant OpenStack deployment using Kolla that bypasses the Ubuntu Noble certificate update bug.

Professional OpenStack Consulting

At Gozunga Cloud, we specialize in complex OpenStack deployments and FIPS compliance requirements. Our team has extensive experience with:

• FIPS 140-2 Compliance: Implementing and maintaining FIPS-compliant OpenStack environments
• Custom Solutions: Developing workarounds for complex deployment challenges
• Production Support: Ensuring reliable, secure cloud infrastructure
• Migration Services: Helping organizations transition to compliant cloud environments

When you need expert guidance for your OpenStack deployment challenges, our consulting team provides the technical expertise and practical solutions to keep your infrastructure running smoothly.

For more information about OpenStack Epoxy and FIPS compliance, visit the official OpenStack documentation.

Get $200 in free credits and start hosting your applications on Gozunga Cloud today!