DEV Community

Darren Horwitz
Darren Horwitz

Posted on

pattern(composition) : terraform nested for loop

Recently I have discovered a pattern that has emerged to become extremely handy when wanting to build out resources that are composed together, typically a one-to-many relationship.

One-to-Many

one-to-many

Now in terraform/cloud land the above image one can express the above image with something as simple as a security-group. The security group can be expressed as the "parent" and a security-group rule can would be expressed as a child of security-group. Thus, the security group is a composition of many security-group rules.

Foo Bar Code




locals {
  map = {
    "foo" = ["child1", "child4"]
    "bar" = ["child2", "child5"]
    "baz" = ["child3", "child6"]
  }

  out = merge([for parentKey, parentValue in local.map : {
    for child in parentValue : "${parentKey}/${child}" => {
      value = child
    }
    }
  ]...)
}


output "out" {
  value = local.out
}



Enter fullscreen mode Exit fullscreen mode

The above code loops through each key-value pair in the map and then it loops through each item in the array (the value of the respective pair).

The for expression in the array will create a map based on the key and item . This is responsible for grouping the maps by the keys of the initial map.

The next for expression in the map is responsible for the creation of the key - values based on the respective value (array of strings ). The key will get namespaced by the respective key (e.g. "foo") and the respective value of the element in the array (e.g. "child1").

Resulting in the composed key : "foo/child1"

The result of the two for expressions before the merge looks like such:




  array = [
    {
      "foo/child1" = {
        value = "child1"
      },
      "foo/child4" = {
        value = "child4"
      }
    },
    {
      "bar/child2" = {
        value = "child2"
      },
      "bar/child5" = {
        value = "child5"
      }
    },
    {
      "baz/child3" = {
        value = "child3"
      },
      "baz/child6" = {
        value = "child6"
      }
    },
  ]




Enter fullscreen mode Exit fullscreen mode

The ellipsis at the end of the array is similar to the spread operator in Javascript , similarly in golang it's a vararg .

What it does is spread the three maps as args in the merge function like such merge({...},{...},{...}).

From there the merge function handles the rest by merge the 3 maps into a single map. Just be careful with this as you need unique keys , otherwise the duplicated key will get overridden with the last respective value.



console output
  + out = {
      + "bar/child2" = {
          + value = "child2"
        }
      + "bar/child5" = {
          + value = "child5"
        }
      + "baz/child3" = {
          + value = "child3"
        }
      + "baz/child6" = {
          + value = "child6"
        }
      + "foo/child1" = {
          + value = "child1"
        }
      + "foo/child4" = {
          + value = "child4"
        }
    }



Enter fullscreen mode Exit fullscreen mode

Example

So obviously the above pattern is pretty nifty, let's find an example of how this pattern can become useful.

Lets say , you need to create a bunch of vpc-endpoints and you want each endpoint to have its own security-group because you need to control what can access these endpoints.



terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.0"
    }
  }
}



locals {
  ## variables
  default_vpc_id = "vpc-id"
  subnet_ids     = ["subnet-1-id", "subnet-2-id"]
  endpoint_services = {
    "s3" = {
      service_name        = "com.amazonaws.eu-west-1.s3",
      private_dns_enabled = true,
      vpce_name_tag       = "s3-interface",

      security_group_ingress_rules = {
        "team-z-vpc-443" = {
          port        = 443,
          cidr_blocks = ["10.0.0.0/24"],
          protocol    = "tcp",
          description = "allow https vpc traffic in ",
        },
        "onprem-443" = {
          port        = 443,
          cidr_blocks = ["192.168.0.0/16"],
          protocol    = "tcp",
          description = "allow https onprem traffic in ",
        },
      },
    },
    "ddb" = {
      service_name        = "com.amazonaws.eu-west-1.dynamodb",
      private_dns_enabled = true,
      vpce_name_tag       = "ddb-interface",

      security_group_ingress_rules = {
        "team-a-vpc-443" = {
          port        = 443,
          cidr_blocks = ["10.0.1.0/24"],
          protocol    = "tcp",
          description = "allow https vpc traffic in ",
        },
      },
    }
  }


  ## nested for loop for sg-rules
  ## not that sg_key_ref is used to reference the sg which is created based on the key-values of local.endpoint_services
  endpoint_sg_ingress_rules = merge([for k, v in local.endpoint_services : {
    for ingressKey, ingressValue in v.security_group_ingress_rules : "${k}/${ingressKey}" => merge({ sg_key_ref = k }, ingressValue)
  }]...)
}

resource "aws_vpc_endpoint" "endpoint_services" {
  for_each = local.endpoint_services

  vpc_id            = local.default_vpc_id
  subnet_ids        = local.subnet_ids
  service_name      = each.value.service_name
  vpc_endpoint_type = "Interface"
  security_group_ids = [
    aws_security_group.this[each.key].id,
  ]
  private_dns_enabled = each.value.private_dns_enabled
  tags = {
    Name = each.value.vpce_name_tag
  }

}



resource "aws_security_group" "this" {
  for_each    = local.endpoint_services
  vpc_id      = local.default_vpc_id
  name        = "${each.value.vpce_name_tag}-sg"
  description = "sg for endpoint service: ${each.value.service_name}"

  tags = {
    Name = "${each.value.vpce_name_tag}-sg"
  }
}

resource "aws_security_group_rule" "ingress" {
  for_each          = local.endpoint_sg_ingress_rules
  type              = "ingress"
  from_port         = each.value.port
  to_port           = each.value.port
  protocol          = each.value.protocol
  cidr_blocks       = each.value.cidr_blocks
  ## referencing the parent ( security group) from which the rule is composed into
  security_group_id = aws_security_group.this[each.value.sg_key_ref].id
}



Enter fullscreen mode Exit fullscreen mode

The cool thing is: when you need to create another endpoint , all you do is add another object into the map that meets the interface thats defined by the variable(yes I didn't create a variable for this example).

Alternatives/Anti-pattern

I think the alternative to this pattern is to rather create a module for the logic that needs to composed and a for_each on the module by passing the respective properties/input variables into the module. The module would be responsible for handling the logic.



module "endpoints" {
  for_each = local.endpoint_services
  source   = "../"

  default_vpc_id = "vpc-id"
  subnet_ids     = ["subnet-1-id", "subnet-2-id"]

  service_name        = each.value.service_name
  private_dns_enabled = each.value.private_dns_enabled
  vpce_name_tag       = each.value.vpce_name_tag

  security_group_ingress_rules = each.value.security_group_ingress_rules
}




Enter fullscreen mode Exit fullscreen mode

Thanks for taking the time read this article , hopefully it will become useful for you. Lastly, comment below how you would implement this sort of pattern !

Top comments (2)

Collapse
 
phzietsman profile image
Paul Zietsman

We used to use this pattern extensively before modules supported for_each. I'm interested to hear why you think module looping is an anti-pattern.

We have found that using "local" modules as containers for this purpose has significantly improved code maintainability, at the cost of having to write a bit more "boilerplate" code in the beginning.

Nice writeup btw!

Collapse
 
darrenhorwitz1 profile image
Darren Horwitz

I appreciate it !πŸ˜†

So I'd imagine that modules will be anti pattern to this because tf now supports for_each , so each iteration of the module will be "namespaced" by the key. So that would take away the need to have to have that sort prefix "s3/" for example because your module would compose/encapsulate all of the respective resources that we did the nested for loop logic with inside the module.

When you refer to local modules, do refer to modularising code that needs to be done on an interactive basis ? If so I could imagine the benefits! I've also noticed that even sometimes when there are a ton of resources that are related to a particular deployment , it's better to modularise it because reading through a million resources in a workdir is a pain when that workspace growsπŸ˜‚ perhaps that single instance of a module may become useful in the future if it's abstracted enough from implementation details