Experts Share Best Practices for Building Terraform Modules
Modules should be focused and not try to accomplish everything — and that includes not using modules as a security and compliance tool, agreed a panel at last month’s HashiCorp HashiConf in Boston.
Two Terraform veterans shared best practices for building modules during a panel discussion. Drew Mullen, a principal architect for River Point Technology, has spent the past eight years working with HashiCorp tooling and was the Terraform primary architect for AWS-IA module standards and development. He has contributed to four terraform provider code bases and has been awarded core contributor three times.
Mullen was joined by Bruno Schaatsbergen, the group manager for developer relations at HashiCorp. His focus is on platform design, distributed systems and software engineering.
Finally, Mullen and Schaatsbergen were moderated by podcaster and author Ned Bellavance, founder of Ned in the Cloud, LLC. Bellavance is a HashiCorp Ambassador.
Before Writing a Module
Bellavance jump-started the discussion with a question about what Terraform developers needed to know about a HashiCorp Terraform module before they wrote a single line of code.
It’s important to talk to the different stakeholders, specifically those who will be or who are using the module, said Schaatsbergen. It’s also important to think about the broader perspective as a platform engineering team, he added.
He considers building modules like running a business — if customers don’t buy what you’re writing because you didn’t talk to them, then you’re doing the wrong thing, he said. It’s important to drill down on the use case for the module, he added.
“A common mistake that’s made is that people start building careful modules without talking to their individuals,” he said.
Mullen begins by asking whether or not the module can be public or should be private, because each has a different scope.
“Scope is a big question that you’re trying to answer,” he said. “You want to understand what the scopes in and what is out.”
Scope is also important because a more complicated module might see less adoption, he added.
“Obviously, if it’s a private module, you can talk directly to people that are actually going to consume it,” Bellavance summarized. “If it’s going to be a public module, then you are serving a wider audience that you don’t necessarily have the input from.”
When Not to Write a Module
Bellavance asked about when to write a module and when not to.
“I’m strongly against the idea of pushing and forcing everyone in your company to use modules,” Schaatsbergen said. “A lot of folks use it to force security and compliance standards.”
But that’s not the role of modules, he continued. Instead, security and compliance standards should be implemented with organizational policies.
“It’s great to provide modules if they solve a particular problem,” Schaatsbergen said. “I really think that a module should do one thing and it should do it good.”
Sometimes, he even builds similar or the same type of modules but for different teams. Abstraction levels should depend on the team, he added.
“Not everyone needs the same level of abstraction,” he said. “The data scientist team needs a very different level of abstraction to a workload team that completely runs everything on Amazon Web Services‘ EKS [Amazon Elastic Kubernetes Service] and I would give them a different EKS module than the ones I would give to the data scientists.”
Scope creep is a real challenge with modules, the panel agreed. But the goal shouldn’t be to write modules that do “everything,” but rather to help users deploy complicated structures in a simple way, Mullen added.
Input Variables and Null Is Not Nothing
Input variables in HashiCorp Terraform are placeholders for values that can be customized when a module is used or a configuration runs. Inputs are a fundamental aspect of a module.
“It’s great to provide modules if they solve a particular problem. I really think that a module should do one thing and it should do it good.”
—Bruno Schaatsbergen, group manager for developer relations at HashiCorp
Input variables are one of the most crucial questions to get at the beginning of a module, Mullen agreed. How you organize the variables is part of the contract that you form with your users, he added.
“Often it can be as simple as just a straight up Boolean or a like a single variable, but also you can structure your modules using types and objects so that you can actually make inputs that organize the module in interesting ways,” he said. “That might make it easier for [your users] to actually start using.”
It’s an underrated practice for default values to use null, Schaatsbergen said.
“Many people think that null is nothing. Null is actually not nothing,” he said. “Null replicates the absence of the value. And if you pass them, if you set the default value or null for an input argument for your module, Terraform actually omits it and it’s not passed down to the resource, which ends up that you fall back on the default value that’s specified in the schema of the resourcing provider.”
Bellavance summarized: “If I could read that back to you, what I’m understanding is, if I put a default value of, say, an empty string, that’s different than setting it to null whereas the empty string would actually get passed as a value for that argument, whereas null is no value set.”
If you choose not to set a default at all, it actually triggers you to input something, Schaatsbergen elaborated.
“If it’s an optional argument, then I set the default value to null, and if I think it’s a required input argument, I don’t specify anything, and it prompts them to supply value,” Schaatsbergen said.
Counts and For_each
Bellavance asked Schaatsbergen about the use of count and for_each inside of a module.
Schaatsbergen said, “Looping constructs are a necessary evil,” and added he tries to only use a count when it’s really needed.
“The only time when I really need to use a count is when I run into the modeling problem, which is when I want to deploy a traditional resource,” he said. “I typically don’t deploy 100 or 200 things or 10 of something, so I don’t typically run into the situation where I deploy the same resource 200 times. So I can get away with declaring those resources separately.”
Mullen differed when it came to count.
“Count is basically used as a Boolean for me. You either have one or zero,” he said. “Other than that, I hardly ever use Count.”
He does like for_each for resources.
“Part of it may even have to go back to what I was saying about complicated variable structures,” he said. “For example, I have a VPC (virtual private cloud) module where the subnets are defined in a map, and you can use any key you want. The nice part about when you design a variable that way is now I have a map that ends in length, or however many keys, and then I can use that to define the for_each loop.”
He can then use the name label as a key.
“You can actually make your outputs look very clean when you start doing things that way because your resources actually have name labels on the two,” he added.
Mullen added that whether you take his approach or Schaatsbergen’s approach may come down to the private versus public module discussion. You’re more likely to know the scope on modules that are privately used, he said, but when something’s public, then you generally write a more flexible module.
Schaatsbergen added that he tries to optimize for DRY (don’t repeat yourself) code.
“I used to be someone that declared for_eachs is everywhere, and I was optimizing for the least amount of lines in my Terraform code, seeing how sophisticated and fancy I could make it until I realized that it always bites me back in terms of maintainability and usability and readability,” he said.
Designing Outputs of Modules
Bellavance asked about their preferred structure for designing the outputs of modules.
“When I was starting to design modules, I decided early that I never wanted to receive an issue from a user that was like, ‘Can you please output this attribute from a resource?’ I just never wanted to see that,” Mullen responded.
His strategy is to output resources in such a way so that if you’re doing a looping construct or something similar, you can actually use TerraForm to output the group, and then you can use the name labels so that users can actually key on and get the specific values that they want.
“It does add a little bit of a burden on the users, though, because it can be a little bit more complicated and so what I try to do is include good examples on how you can actually use the output of the module,” Mullen added.
Schaatsbergen said it’s important to be consistent if that’s the approach you take because, otherwise, it could confuse users.
“Personally, I don’t want to force my company or teams to use modules everywhere,” he said. “I always try to put output all the attributes and I sometimes run into issues where someone says, ‘Hey, I really need this value. You have to output it. Can you create an output for it?’”
Bellavance added that one way to alleviate the issue is to make it part of the discovery discussion.
“If you can identify the outputs that they’re expecting to get from the module, then you can have more succinct outputs that are easier for the end user to consume and then you could expose the entire resource as a different output if you think that’s going to be helpful to them. So you have both options,” Bellavance summed up.
Testing Modules and Documentation
Mullen is a big fan of the TerraForm Test framework. Previously, they used Terra Test, but it’s written in a language not a lot of people know so it’s hard to expect your users to run those.
“The one thing that I’d add into testing is that when you structure your modules, you typically will see the module code. You’ll see examples. You might see a docs directory or something like that, but examples are one of the first things that people click into,” he said. “If you look at GitHub views, my example directories are where a lot of people spend a lot of their time, and so I always try to make sure that we have tests that are testing the example repos.”
You define the major use cases that your module is trying to achieve and write tests that use those, he added. That validates the use case, as well as the documentation you’re offering, he added.
When it comes to documentation, Schaatsergen recommended the open source tool on GitHub called Terraform-docs which can be used to generate module documentation if you’re building providers.
Mullen said that one thing he tries to do with documentation is include diagrams that show different types of uses in the example screens.
“That is very helpful because sometimes you look at the module naming and you think, you know what’s in there, but having that picture, it lines up better,” he said.
