Learn how to identify and clean up stale instance allocations in OpenStack Nova that can cause resource accounting issues and prevent proper resource allocation.

1. Overview - Understanding stale allocations
2. Prerequisites - What you need before starting
3. Identifying Stale Allocations - Running the audit command
4. Cleaning Up Allocations - Removing stale allocations
5. Verification - Confirming the cleanup worked

Overview

This guide demonstrates how to identify and clean up stale instance allocations in OpenStack Nova that can cause resource accounting issues. The process includes:

• Detection: Using the nova-manage placement audit command to identify stale allocations
• Cleanup: Removing allocations that are no longer associated with active instances
• Verification: Confirming that the cleanup was successful

Stale allocations occur when instance records are deleted but their resource allocations remain in the placement service, leading to:

• Incorrect resource accounting
• Inability to create new instances due to "insufficient resources" errors
• Resource provider capacity appearing lower than actual available resources

This guide is essential for OpenStack administrators managing Nova deployments, especially those using kolla-ansible.

1. PREREQUISITES

Before starting this cleanup process, ensure you have:

• OpenStack Nova: Running Nova deployment with placement service
• Controller Node Access: Administrative access to the Nova controller node
• Container Access: Ability to access the nova-conductor container (for kolla-ansible deployments)
• Admin Credentials: Nova admin user credentials or root access

Required Information:

• Controller node hostname or IP address
• Nova database access (if needed for troubleshooting)
• Understanding of your OpenStack deployment method (kolla-ansible, devstack, etc.)

2. IDENTIFYING STALE ALLOCATIONS

The first step is to identify if you have stale instance allocations in your OpenStack Nova deployment.

Step 1: Access the Nova Conductor Container

For kolla-ansible deployments, access the nova-conductor container:

# On your controller node
docker exec -it nova_conductor bash

Step 2: Run the Placement Audit Command

Run the audit command to identify stale allocations:

(nova-conductor)[nova@controller01 /]$ nova-manage placement audit --verbose

Expected Output:

If stale allocations are found, you'll see output similar to:

Allocations were set against consumer UUID ba2a35a1-e704-4de3-b583-66c6669a11ec but no existing instances or active migrations are related.
Allocations for consumer UUID ba2a35a1-e704-4de3-b583-66c6669a11ec on Resource Provider 6a7343eb-64af-4d62-b3fd-745b224874e0 can be deleted: {'DISK_GB': 100, 'MEMORY_MB': 8192, 'VCPU': 4}
Processed 1 allocation.

If there are no stale allocations, you'll receive no output.

Understanding the Output:

• Consumer UUID: The identifier of the stale allocation
• Resource Provider UUID: The compute node where resources were allocated
• Resource Details: Shows the specific resources (CPU, memory, disk) that can be freed
• Processed Count: Total number of allocations checked

3. CLEANING UP ALLOCATIONS

Once you've identified stale allocations, you can remove them using the --delete flag.

Step 1: Remove Stale Allocations

Run the audit command with the --delete flag to remove the identified stale allocations:

(nova-conductor)[nova@controller01 /]$ nova-manage placement audit --verbose --delete

Expected Output:

Allocations were set against consumer UUID ba2a35a1-e704-4de3-b583-66c6669a11ec but no existing instances or active migrations are related.
Deleted allocations for consumer UUID ba2a35a1-e704-4de3-b583-66c6669a11ec on Resource Provider 6a7343eb-64af-4d62-b3fd-745b224874e0: {'DISK_GB': 100, 'MEMORY_MB': 8192, 'VCPU': 4}
Processed 1 allocation.

What Happens:

• The command identifies the same stale allocations as before
• It deletes the allocations from the placement service
• Resources are freed and made available for new instance creation
• The process is logged for audit purposes

4. VERIFICATION

After cleaning up the stale allocations, verify that the process was successful.

Run the audit command again without the --delete flag to confirm no stale allocations remain:

(nova-conductor)[nova@controller01 /]$ nova-manage placement audit --verbose

Expected Output:

If there are no stale allocations, you'll receive no output.

This confirms that all stale allocations have been successfully removed.

SUMMARY

This guide walked you through the process of identifying and cleaning up stale instance allocations in OpenStack Nova, including:

1. Identification: Using nova-manage placement audit --verbose to detect stale allocations
2. Cleanup: Using the --delete flag to remove identified stale allocations
3. Verification: Confirming that the cleanup was successful and resources are available

The result is a clean Nova deployment with accurate resource accounting and the ability to create new instances without "insufficient resources" errors.

For more information about OpenStack Nova placement service, visit the official Nova documentation.

Get $200 in free credits and start hosting your applications on Gozunga Cloud today!