Learn how to resolve timezone configuration conflicts in OpenStack Watcher containers built with Kolla/Kolla-Ansible that prevent proper deployment and cause rapid container cycling.

1. Overview - Understanding the timezone conflict issue
2. The Problem - Identifying the root cause
3. Quick Fix Solution - Immediate workaround steps
4. Verification - Confirming the fix works
5. Long-term Solutions - Better approaches for production

Overview

After reading about OpenStack Watcher's powerful resource optimization capabilities, I decided to deploy it in our OpenStack environment. However, I encountered a frustrating timezone configuration issue that caused all Watcher containers to start and stop in rapid succession, preventing the service from functioning properly.

The Problem: OpenStack Watcher containers were experiencing timezone configuration conflicts between /etc/timezone and /etc/localtime, causing Python's zoneinfo module to throw errors and containers to crash repeatedly.

The Solution: A quick workaround involving manually fixing the timezone symlinks within the running containers, though there are more elegant long-term solutions involving custom Kolla images.

The Problem

When deploying OpenStack Watcher, you may encounter this error in the container logs:

ERROR python-watcher zoneinfo._common.ZoneInfoNotFoundError: 'Multiple conflicting time zone configurations found:
/etc/timezone: America/Chicago
/etc/localtime is a symlink to: Etc/UTC
Fix the configuration, or set the time zone in a TZ environment variable.'

Root Cause: The issue occurs when there's a mismatch between:

• /etc/timezone file content (showing America/Chicago)
• /etc/localtime symlink destination (pointing to Etc/UTC)

This conflict confuses Python's zoneinfo module, which expects consistent timezone configuration across the system. This is a known issue with Ubuntu 24.04 Noble containers in OpenStack deployments, documented in Bug #2091161. The problem occurs because the tzdata package installation during container build creates a symlink that conflicts with the host's timezone configuration when mounted into the container.

Symptoms:

• Watcher containers start and stop in rapid succession
• Service fails to initialize properly
• Logs show repeated timezone-related errors
• OpenStack Watcher dashboard shows services as unavailable

Quick Fix Solution

The quickest way to resolve this issue is to manually fix the timezone configuration within the running containers. This approach works well for testing and evaluation purposes.

Step 1: Identify Running Containers

First, identify which Watcher containers are running:

docker ps | grep watcher

You should see containers like:

• watcher_engine
• watcher_api
• watcher_applier

Step 2: Fix Timezone Configuration

For each container, you need to start the container first, then run the timezone fix command. This approach is more reliable than trying to catch a running container.

docker start watcher_engine
docker exec -it -u0 watcher_engine bash -c 'rm /etc/localtime; ln -s /usr/share/zoneinfo/America/Chicago /etc/localtime'

Command Breakdown:

• docker start watcher_engine: Start the container to ensure it's running
• docker exec -it -u0: Execute command as root user in interactive mode
• rm /etc/localtime: Remove the conflicting symlink
• ln -s /usr/share/zoneinfo/America/Chicago /etc/localtime: Create correct symlink

Why This Works: By starting the container first, you ensure it's in a running state before attempting to execute the timezone fix command. This eliminates the need to retry multiple times and makes the process more reliable.

Step 3: Repeat for All Containers

Apply the same fix to all Watcher containers:

# Fix watcher_api
docker start watcher_api
docker exec -it -u0 watcher_api bash -c 'rm /etc/localtime; ln -s /usr/share/zoneinfo/America/Chicago /etc/localtime'

# Fix watcher_applier  
docker start watcher_applier
docker exec -it -u0 watcher_applier bash -c 'rm /etc/localtime; ln -s /usr/share/zoneinfo/America/Chicago /etc/localtime'

Verification

After applying the fixes, verify that Watcher is functioning properly.

Step 1: Check Container Status

Verify containers are running stably:

docker ps | grep watcher

Containers should show as "Up" for more than a few seconds without cycling.

Step 2: Check Logs

Review Kolla log files for timezone errors:

grep -i timezone /var/log/kolla/watcher/watcher_engine.log
grep -i timezone /var/log/kolla/watcher/watcher_api.log
grep -i timezone /var/log/kolla/watcher/watcher_applier.log

No timezone-related errors should appear.

Step 3: Test Watcher Functionality

Access the OpenStack Watcher dashboard or use CLI commands to verify the service is operational:

# Check Watcher service status
openstack optimize audit list

# Create a test audit template
openstack optimize audit template create --goal workload_balancing --strategy workload_stabilization test_template

Long-term Solutions

While the manual fix works for testing, production environments should implement more robust solutions. Note: This issue has been fixed in the 2025.1 (Epoxy) release of OpenStack Kolla.

Option 1: Custom Kolla Images

Create custom Kolla images with proper timezone configuration:

FROM kolla/ubuntu-source-watcher-base:2025.1

# Set timezone during image build
ENV TZ=America/Chicago
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

Option 2: Environment Variable Override

Set the TZ environment variable in your Kolla configuration:

# In your kolla configuration
watcher_engine:
  environment:
    - TZ=America/Chicago

watcher_api:
  environment:
    - TZ=America/Chicago

watcher_applier:
  environment:
    - TZ=America/Chicago

Option 3: Host Timezone Synchronization

Ensure your container hosts have consistent timezone configuration:

# Set timezone on all hosts
sudo timedatectl set-timezone America/Chicago

# Verify configuration
timedatectl status

Summary

This guide demonstrated how to resolve timezone configuration conflicts in OpenStack Watcher containers, including:

1. Problem Identification: Understanding the timezone conflict between /etc/timezone and /etc/localtime
2. Quick Fix: Manual timezone symlink correction within running containers
3. Multi-host Deployment: Applying fixes across all container hosts
4. Verification: Confirming Watcher services are operational
5. Long-term Solutions: Better approaches for production environments

The manual fix provides an immediate solution for testing OpenStack Watcher, while the long-term solutions ensure stable production deployments.

Professional OpenStack Consulting

At Gozunga Cloud, we specialize in complex OpenStack deployments and troubleshooting challenging configuration issues. Our team has extensive experience with:

• OpenStack Watcher: Implementing and optimizing resource management strategies
• Container Orchestration: Resolving Docker and Kubernetes deployment issues
• Timezone Configuration: Ensuring consistent time handling across distributed systems
• Production Support: Maintaining reliable, high-performance cloud infrastructure

When you need expert guidance for your OpenStack deployment challenges, our consulting team provides the technical expertise and practical solutions to keep your infrastructure running smoothly.

For more information about OpenStack Watcher and its optimization capabilities, visit the official OpenStack Watcher documentation.

Get $100 in free credits and start hosting your applications on Gozunga Cloud today!