Strange comparison, I know.
Scripting vs declarative approaches
The aspect I’m looking at is scripting (aka imperative programming) vs declarative approach. In many situations I choose the scripting approach over declarative because the downsides of declarative approach outweigh the benefits in the situations that I have.
Declarative approach downsides
Downsides of Chef, Puppet and other declarative systems? Main downsides are complexity and more external dependencies. These lead to:
- Fragility
- More maintenance
- More setup for anything except for the trivial cases
I can’t stress enough the price of complexity.
Declarative approach advantages
When the imperative approach would mean too much work the declarative approach has the advantage. Think of SQL statements. It would be enormous amounts of work to code them by hand each time. Let’s summarize:
- Concise and meaningful code
- Much work done by small amount of code
Value of tools
I value the tools by TCO.
Example 1: making sure a file has specific content. It could be as simple as echo my_content > my_file in a script or it could be as complex as installing Chef/Puppet/Your-cool-tool-du-jour server and so on…
Example 2: making sure that specific load balancer is set up (AWS ELB). It could be writing a script that uses AWS CLI or using declarative tools such as CloudFormation or Terraform (haven’t used Terraform myself yet). Writing a script to idempotently configure security groups and the load balancer and it’s properties is much more work than echo ... from the previous example.
While the TCO greatly depends on your specific situation, I argue that the tools that reduce larger amounts of work, such as in example 2, are more likely to have better TCO in general than tools from example 1.
“… but Chef can manage AWS too, you know?”
Yes, I know… and I don’t like this solution. I would like to manage AWS from my laptop or from dedicated management machine, not where Chef client runs. Also, (oh no!) I don’t currently use Chef and bringing it just for managing AWS does not seem like a good idea.
Same for managing AWS with Puppet.
Summary
Declarative tools will always bring complexity and it’s a huge minus. The more complex the tool the more work it requires to operate. Make sure the amount of work saved is greater than the amount of work your declarative tool requires to operate.
Opinion: we can do better
I like the scripting solutions for their relative simplicity (when scripts are written professionally). I suggest combined approach. Let’s call it “declarative primitives”.
Imagine a scripting library that provides primitives AwsElb, AwsInstance, AwsSecGroup and such. Using this primitives does not force you to give up the flow control. No dependency graphs. You are still writing a script. Minimal complexity increase over regular scripting.
Such library is under development. Additional advantage of this library is that the whole state will be kept in the tags of the resources. Other solutions have additional state files and I don’t like that.
Sample (NGS language) censored code that uses the library follows:
my_vpc_ancor = {'aws:cloudformation:stack-name': 'my-vpc'}
elb = AwsElb(
"${ENV.ENV}-myservice",
{
'tags': %{
env ${ENV.ENV}
role myservice-elb
},
'listeners': [
%{
Protocol TCP
LoadBalancerPort 443
InstanceProtocol TCP
InstancePort 443
}.n()
]
'subnets': AwsSubnet(my_vpc_ancor).expect(2)
'health-check': %{
UnhealthyThreshold 5
Timeout 5
HealthyThreshold 3
Interval 10
Target 'SSL:443'
}.n()
'instances': AwsInstance({'env': ENV.ENV, 'role': 'myservice'}).expect()
}
)
elb.converge()
It creates a load balancer in an already existing VPC (which was created by CloudFormation) and connects existing instances to it. The example is not full as the library is work in progress but it does work.
Have fun and watch your TCO!
Ilya's blog 
The idea of “declarative primitives” is very interesting.
Does your argument that simple scripting tools reduce larger amounts of work are more likely to have better TCO in general than puppet/chef, takes into consideration agent-less tools like Ansible?
LikeLike
Thanks!
I only took a look at Ansible and never actually worked with it. I did not see many reasons to choose Ansible over a script.
Note that Chef and Puppet can also be used without the central server but this is not a common configuration.
LikeLike
Ilya, I shared he same perspective on “enterprisey” provisioning and configuration management systems like Chef / Puppet that you articulate here, primarily their lack of idempotency or at least the complexity required to implement it properly as well as the heaviness of the agent based systems. Of course those systems have a place in any engineers took it but it seems they were suited for smaller organizations or teams that are truly attempting to be agile .
I was incredibly surprised on the upside when I spent the time to begin working with Ansible. Dean is a learning curve was a little steep though not that bad given the power of its declarative syntax and the ability to extend it almost definitely be a python modules. I have not yet checked out the t have not yet checked out the tool ool you referenced in this post, but if it’s still under development, I would strongly recommend spending a few days getting to know Ansible well, especially version 2.0 that was recently released after Red Hat acquired them.
The feature set simply unmatched by any system with similar levels of complexity and even offers full device management for an entire network stack I think you’ll be pleasantly surprised as well at the effective TCO given the large community of developers contributing modules and extensions at a rapid pace as well as the invitations being driven by Red Hat post acquisition .
LikeLike
Sounds like Ansible commercial by Red Hat.
Chef / Puppet “lack of idempotency”? Where?
LikeLike
Ilya, your whole argument is based on the assumption “…(when scripts are written professionally)…”, which is easy to say but does not reflect reality since most people in the ops space are not professional software developers. That is the real value of tools such as Ansible. They simplify re-use of existing scripts which are essentially called modules in Ansible.
LikeLike
You have a point. I agree a bit. Still, today’s “system” work includes programming, preferably well. Unprofessional people (read can’t program) will cause problems sooner or later in any case. Tools such as Ansible are not the right solution for that in my opinion. I’d really prefer to work with 2 professional people that get twice the salary than team of 5 that can’t program. I think it would be more productive.
LikeLike
I understand you. But it still doesn’t the reflect the reality of most IT shops. I looked at your NGS project and UI poc. The shell in a browser is cool. How does the browser connect to the system?
LikeLike
Hi. Sorry did not see your comment. That’s just a POC and I’m focused on the language right now (next: CLI). At the time I was thinking HTTPS + some shared key (maybe randomly generated password file).
LikeLike
Script approach works fine until you start to grow, if you are a one man team and/or your infrastructure is small, then sure script works fine. But when you actually have to work on a large scale system and/or a larger team the script approach is neither maintainable nor feasible . You do have a point in regards to complexity of CM tool, however there are times when it is needed to introduce extra complexity to deal with a even larger complexity issue (scale, idempotency etc). Not to mention when you start integrating your infrastructure with CI tool chain, a CM is basically a must.
LikeLike
Idempotency is possible in scripts too. Most of my scripts are idempotent.
How is that CI is so much better with CM tool than scripts?
LikeLike