A typical use case when ops-ing servers is to rotate servers: Create a new server and remove the old server. Since our servers are managed by Terraform and Ansible this should be an easy exercise. If you know what you are doing...

Introduction

For an existing web-server we needed to update some token with an external api before a specific date. Since we did not want to do an in-place update, since this can break the current live environment, we wanted to add a web-server connect with the external api with a new token and then phase-out the old server.

Rotating servers is a typical use case and fully supported by Terraform if you use the correct stanza. With the help of the Terraform user forum we were able to figure out the correct way for our use case.

Setup

The simplified version of the web-server terraform module:

In terraform/modules/web-server/main.tf

resource "aws_instance" "web_server" {
 ami = "abc"
 key_name = "ben"
 tags = {
  Name = var.server_name
  ServerRole = "web"
  Env = var.environment
 }
}

Then we can create a single instance per environment as follows:

In terraform/acceptance/main.tf

module "web-server" {
  source      = "../modules/web-server"
  environment = "acceptance"
  server_name = "web01.acceptance.server.nl"
}

When you want to create multiple instances of the same module you can use count_and since 0.12.6 you can also use foreach.

Using count

Since we have some specific values per instance we need to refactor our setup for using count as follows:

In terraform/modules/web-server/main.tf

resource "aws_instance" "web_server" {
  ami = "abc"
  key_name = "ben"
  count = length(var.server_names)
  private_ip = var.ip_addresses[count.index]
  tags = {
    Name = var.server_names[count.index]
    ServerRole = "web"
    Env = var.environment
  }
}

Then we can create a single instance as follows:

In terraform/acceptance/main.tf

module "web-server" {
  source      = "../modules/web-server"
  environment = "acceptance"
  server_name = ["web01.acceptance.server.nl"]
  ip_addresses = ["1.2.3.4"]
}

Now this already felt a bit strange to create to lists that are connected by their index.

To create multiple instances we can then simply add a server_name to server_names

In terraform/acceptance/main.tf

module "web-server" {
  source       = "../modules/web-server"
  environment  = "acceptance"
  server_names = ["web01.acceptance.server.nl", "web02.acceptance.server.nl"]
  ip_addresses = ["1.2.3.4", "1.2.3.5"]
}

When you run terraform plan it will propose to create a new server called module.web-server.aws_instance.web_server[1]

So far so good. Now after provisioning the new server we can delete the old server by just removing the server from the server_names list. At least that is what we thought...

module "web-server" {
  source       = "../modules/web-server"
  environment  = "acceptance"
  server_names = ["web02.acceptance.server.nl"]
}

After running terraform plan terraform suggest to destroy one server and re-create (meaning destroy and create another server). As explained in the forum post by Martin Atkins:

The behavior you are describing is a typical result for using count against a list because it causes Terraform to identify the individual instances by the indexes into the list, and so removing an item will assign new indices to all items appearing after it.

This can be "solved" by fiddling around with the terraform state but that is quite tedious and error-prone. So this kind of ruled out using count for rotating servers described in our use case.

Using foreach

The foreach option is much more suitable for our use case. This is because of the fact that Terraform will no longer identify each instance by the index in the list but by a named id.

So we have to refactor the web-server module to use foreach:

In terraform/modules/web-server/main.tf

resource "aws_instance" "web_server" {
  foreach = var.host_to_ip
  ami = "abc"
  key_name = "ben"
  private_ip = each.value
  tags = {
    Name = each.key
    ServerRole = "web"
    Env = var.environment
  }
}

To create a single instance using foreach we need to pass in a map from host to ip addresses.

In terraform/acceptance/main.tf

module "web-server" {
  source       = "../modules/web-server"
  environment  = "acceptance"
  host_to_ip = {
    "web01.acceptance.server.nl" = "1.2.3.4"
  }
}

When running terraform plan it still wants to destroy the existing instance since its id does not match: module.web-server.aws_instance.web_server versus module.web-server.aws_instance.web_server["web01.acceptance.server.nl"]. In order to overcome this problem we need to import the existing aws instance with the new name and remote the old one only once:

# Import instance under the new name
terraform import module.web-server.aws_instance.web_server[\"web01.acceptance.server.nl\"] YOUR_ID_IN_AWS
# Remove the instance from the current terraform state
terraform state rm module.web-server.aws_instance.web_server

Now when running terraform plan terraform reports no changes! So we are good to go to add a new server and remove the old one.

In terraform/acceptance/main.tf

module "web-server" {
  source       = "../modules/web-server"
  environment  = "acceptance"
  host_to_ip = {
    "web01.acceptance.server.nl" = "1.2.3.4"
    "web02.acceptance.server.nl" = "1.2.3.5"
  }
}

Running terraform apply will create web02.acceptance.server.nl. After provisioning, and connecting it to our external api, we can remove the old server by just removing it from the terraform/acceptance/main.tf file and running terraform apply again.

In terraform/acceptance/main.tf

module "web-server" {
  source       = "../modules/web-server"
  environment  = "acceptance"
  host_to_ip = {
    "web02.acceptance.server.nl" = "1.2.3.5"
  }
}

Summary

In Terraform 0.12.6 foreach was introduced. This perfectly matches the rotate servers use case and saves a lot of fiddling around with the Terraform state. When ever you define modules that can have multiple instances I suggest you you foreach right from the start so you never have to import and change the state manually.