This post is a reply to “just use Terraform” recommendation I’ve just seen. I hope more people will benefit from my perspective if it’s posted here. There is plenty of marketing behind most of the tools I mention here. It’s all rosy, see the “Life before Puppet” video. Let’s balance this marketing bullshit a bit.

Think twice before using declarative framework/tool

Edit 2017-12-06: note that this post is my opinion based on my experience with Puppet, Chef and CloudFormation. My opinion is based on my use cases and circumstances.

Terraform, CloudFormation, Puppet, Chef as any other declarative frameworks/tools take control away from you. They work fine for “hello world” examples. Then there is life where you need something these frameworks did not anticipate and you are sorry you have not coded everything yourself from the start. Now you are stuck with these tools and you will be paying for it in your time and money. Working around limitations of such tools is a pain.

I am using CloudFormation and have used Puppet and Chef in the past. These tools do have their place. In my opinion it’s a very limited set of scenarios. Terraform, CloudFormation, Puppet and Chef are used much more widely than they should be.

These tools have some value but too often people neglect the cost which in many cases outweighs the value. Most of the cost comes from inflexibility. Terraform and CloudFormation are so limited that people frequently use another tool for generating these. That adds another bit to the cost.

I’m hearing frequently from a friend (sorry, can’t name him) how much they suffer from Terraform’s inflexibility. Inflexibility can not be fixed because it’s a declarative framework. Unfortunately they are so invested in Terraform that they will continue to spend hundreds of hours to fight it. Chef is causing trouble there too, community Cookbooks proved to be a mismatch for the needs and sanity of the engineers there.

Edit 2017-12-06, following discussion on Reddit:

• I spoke to my friend regarding Terraform. Many issues they had are solved by more up-to-date version of TF.
• “Inflexibility” I’m talking about is my opinion on DSL. On the other hand, Terraform DSL contains features that apparently help cover most use cases.
• Just to make clear: CloudFormation has a lot flexibility using Lambda. That doesn’t make sense to me but it is there.

… and there is this gem

A key component of every successful Puppet implementation is access to a knowledgeable support team

That’s from https://puppet.com/support-services/customer-support/support-plans

Are you sure you want to use Puppet? Apparently you can’t do it well without their support… Just saying…

Is one of these tools right for you?

Regular considerations for choosing a tool apply. See my older post “Prove your tool is the right choice“.

Expected replies and my replies to those

You don’t get it.

OK

You don’t understand these tools.

OK

You are not using these tools right / as intended.

OK

Are you crazy? You want to code everything yourself?

Let’s take it to the extreme: no new code should be written. No libraries, no frameworks. Because everything already exists. Sounds about right.

People smarter than you have figured it all out, use their tools

Smarter people don’t always produce better solutions or solutions that fit your use case. Most of the time smart people will produce smart solutions… and then there are people that don’t usually think in graphs and are really puzzled when debugging Puppet cyclic dependency errors for example.

Most of the code you need is already written, don’t waste time and money, use it! Community Cookbooks and modules are great!

This is marketing bullshit. Don’t buy it! It’s often more expensive to adopt a code that does not meet your exact needs and is much more complex that you need (because it should support multiple platforms and use cases) than to write your own. I have seen suffering followed by usage of community Cookbooks/modules followed by in-house rewrite or fork.

Don’t you care about the next guy? Work with standard tools!

Let’s do some math. Team of two works for a year. They are (very modest estimation) 10% more productive because they have coded whatever they needed and were not fighting with the tools. Even when wrongfully assuming that custom solution is harder to understand for the 3rd guy that joined the team after one year, how much is it harder? Is it more than 300 hours harder?

Update following responses on Reddit

2017-04-28

2 totally different toolsets – infrastructure orchestration (Terraform, Cloudformation), and Configuration Management (Puppet, Chef)… — (/u/absdevops)

Yes. What is common to all these tools is declarative style and their usage: these tools are typically run using CLI.

All these tools have three axes that I consider:

1. “Input” axis: What’s the input of these tools?
   1. Configuration format
   2. Half-baked programming language that was probably never indented to be a programming language
   3. Real programming language
2. “Calling” axis: framework vs library (typical usage)
3. “TCO” axis: TCO vs other solutions, especially vs the other solution that is always available – code the subset of the functionality that you need yourself

I’d like to make sure that it’s clear that the tools mentioned in this article have different positions on the 3 axes and are not equal in the value they provide you in your specific situation.

The main point of the article is that while these tools differ on axes 1 and 3, they are all limiting because conceptually, they are all frameworks. You pass your execution into the tool and it does a lot. Here is where you loose your flexibility as opposed to using a library. You have relatively little control of what happens inside the tool.

I must strongly disagree with Terraform being put in the list – its a great base tool with limitations that can be worked around. — (/u/absdevops)

I don’t want to work around limitations. It seems to be the norm for these tools. I’d rather have a library that misses parts that I’d code myself. Working around limitations in my opinion is generally much worse than missing functionality (depends on specific circumstances of course).

Regarding inflexibility – it’s probably the most flexible tool of the bunch — (/u/absdevops)

Please note we are still comparing between the tools that all use limiting paradigm: frameworks

I will also duel anyone to the death for preference of Cloudformation syntax to Terraform — (/u/absdevops)

We are talking about the “Input” axis I mentioned above. Yes, Terraform syntax apart from being more aesthetically pleasing is somewhat closer to “Half-baked programming language that was probably never indented to be a programming language” while CloudFormation is somewhat closer to “Configuration format”.

I totally disagree with points made about having to generate Terraform manifests. … generate what you need specifically, and hand it off to Terraform, much like making an API call to a library. — /u/SlinkyAvenger

There is a huge difference in the amount of work done by typical API call and what these tools do once you call them. With more granular API calls you decide if and when you do specific calls and what do you do in between the calls – it’s much more flexible.

I’m also a big proponent of Puppet — /u/SlinkyAvenger

One of the low value tools from my perspective. I’ll explain. On the “Input” axis, it’s half-baked programming language. Better than configuration file but still loses to Chef for example. On the “TCO” access, I really think that Puppet and Chef are not good alternatives to custom scripts in most cases. Scripts by the way also win on the “Calling” axis, which means flexibility.

I’d really like to hear what you’re honestly going back to puppet support for. — /u/neoghostz

We don’t. When we suffered while working with Puppet, we knew that support will not solve our problems. Some crappy community modules can not be solved by support. Breakage on modules versions updates – same. Librarian, more complexity on top of complexity – same. The above quote about support (“A key component of every successful Puppet implementation is access to a knowledgeable support team”) was just to highlight that guys at Puppet think people can’t use it without support. This is just a humorous point and not really important.

What is the point of this article? It basically dumps on Terraform, CF, Puppet, Chef, etc., but offers no actual criticism (other than a vague ‘it takes away control’ statement) or, perhaps more importantly, alternatives. — /u/cryonine

The point is that all these tools would have been better if they would be implemented as libraries on top of real programming languages, where you call the parts that you need instead of one “do everything” call.

With the exception of Chef, these tools use either configuration files as input or configuration-file-almost-a-programming-language format. It’s always the same path:

1. We need small limited DSL it’s so academically beautiful, we can prove theorems about this.
2. Oh wait, there are real world scenarios where it’s not enough, damn these complaining engineers.
3. Let’s add stdlib
4. Let’s add proper loops
5. Now we have a half-baked programming language.

Elaborating on taking control away from you. You get convoluted things like this:

• https://gist.github.com/ryan0x44/18da672bb416046284ed
• https://gist.github.com/grubernaut/2dc93ea62f018444f10b#file-readme-md
• https://github.com/hashicorp/terraform/issues/1604

Alternatives

For Puppet and Chef, I have not seen a single system where my estimated TCO of these tools would be better than a bunch of idempotent modular bash scripts which I use. It did not take much time to write these. Some Python is used for configuration generation (json / jinja templates + environment data).

With Cloudformation and Terraform it’s not that simple. I’m mostly amazed that nobody does libraries which would just provide declarative primitives, not frameworks where you feed everything you need via one call. I am working on one but it is really strange for me that I haven’t heard already about such library for Python or Ruby.

Terraform … vastly superior to any other alternate out there — /u/cryonine

Not sure I agree 100% because it depends on situation but I can imagine many situations where it’s correct. The important thing here is that I think that all current alternatives are not so good.

How is it wrongful to assume that a custom solution is harder to understand? That’s completely accurate. — /u/cryonine

Custom solution is simpler. Do you really need documentation for 19 lines of bash code that installs Nginx and another 29 that do a restart that handles leaking file descriptors? You will definitely need documentation of 2000+ lines of Chef cookbook or Puppet module that install Nginx and … oh wait… how do I reload Nginx and then conditionally (if enough file descriptors leaked) restart it? Time to dive in 🙂

I do imagine how custom solution can be complicated (read harder to maintain and higher TCO) if done by unprofessional people. In some cases it might be better for them to use a framework. On the other hand, they might stuck when trying to do something advanced with the framework. Really depends on the situation.

While “use standard tools” generally sounds right, I have seen too much convoluted solutions using “standard tools” because of the inflexibility. People were trying to work around the limitations. Comparing top-down execution of simple script to workarounds for these tools, it’s much simpler to wrap your head around the scripts. I have recently passed one of my clients to the next guy. I have asked him how he is doing and he told me that he was happy to have simple custom solution over complex frameworks. TCO has many components. Choosing “standard tools” does not always outweigh other aspects.

 

---

Have a nice day and a productive life!