This article is aimed mostly at juniors. Lacking experience, you are a soft target for marketing bullshit. I encourage critical thinking for evaluation of products and services that are forced down your throat marketed to you.
The purpose of dishonest marketing is to mislead you into using a product and/or a service that you don’t really need. This can waste your time and money.
Separate the products from marketing. The products themselves are not inherently good or bad. The main question is whether to use product X or Y (and how) or code the solution yourself given your specific situation: team, skill levels, requirements, etc. This post will guide you how to see through marketing bullshit when evaluating these products.
The list below is not comprehensive and I might add items later.
Products Marketing Bullshit
( in no particular order )
Cool / coolness
If something is “cool” and you use it, you might write a blog post about it. It has some PR value to you or your company. Other than that, “coolness” of a product has nothing to do with its usefulness for your situation/use case.
Our product is so much better than X
Note that often X will be the worst possible alternative. For example, configuration management tools will most likely be compared to manual work rather to other configuration management tools or any kind of automation. Ignore and compare the suggested product to other viable alternatives.
Use our product or become irrelevant / Our product is [becoming] industry standard
The message is sometimes direct but often it is hidden between the lines. This is an attempt, many times successful, to exploit fear of missing out. Learn the underlying principles then decide whether you want to use one of the products, which come and go. Learning the underlying principles will take more time upfront but you will become more professional. If you are junior Ops, learn Linux and how to use it without any configuration management tools first; learn using the cloud without CloudFormation or Terraform first; see the problems which these tools trying to solve before using the tools.
Our customers include big companies such as X, Y and Z
You have to understand what these companies are using the product/service for. It can be a pilot for example. Check for common owners or investors. Remember that big companies make mistakes too. Anyhow, this is irrelevant for your use case and your situation until proven otherwise.
The formula is simple: deep shit + our product = great success. Look closely. Often deep shit + other sensible alternative solution would also be great success. For example, configuration management tools show manual process and then how it was automated. Chances are automating using scripts would be other viable alternative.
We abstract all the hard work away from you
You must understand what exactly is abstracted and how. After learning that, it might happen that the perceived value of the product will drop. Skip the learning step and start using the tool and you are in danger: you might need to learn whatever was abstracted in hurry when facing a bug in the tool or be at mercy of someone else to fix it. Remember: abstractions come at cost.
Don’t reinvent the wheel, we have figured out everything already
This exploits your fear of looking stupid. Would you use a spaceship for travelling from Moscow to Tokyo? I guess it would be cheaper to use a plane. Even if tool X solves your problem it might cost you time and complexity and it might be easier to code yourself or use simper tool. There are probably more legitimate reasons not to use tool X.
Companies marketing bullshit
Industry leading company
Who is not? Define your industry narrow enough and you are the leading company! Simply ignore.
Did I miss something? Let me know here in comments or on Reddit. Have a nice day!
The question “should I do it in bash or in Python?” is both frustrating and common. Why one even needs to choose between two alternatives which are both inadequate for the task at hand? Why try to pick one of the square pegs for the round hole? I believe that you should not be in this annoying situation when just trying to write a script and get back to your endless stream of other todos.
Both are Inadequate for Ops
bash does not meet any modern expectations for syntax, error handling nor has ability to work with structured data (beyond arrays and associative arrays which can not be nested). Let it go. You are not usually coding in assembly, FORTRAN, C, or C++, do you? They just don’t match the typical Ops tasks. Don’t make your life harder than it should be. Let it go. (Let’s not make it a blanket statement. Use your own judgement when to make an exception).
Python along with many other languages are general purpose programming languages which were not intended to solve specifically Ops problems. The consequence is longer and less readable scripts when dealing with files or running external programs, which are both pretty common for Ops. For example, try to check every status code of every program you run, see how your code looks like. Sure you can import 3rd party library for that. Is that as convenient as having automatic checking by default + list of known programs which don’t return zero + convenient syntax for specifying/overriding expected exit code? I guess not.
Your disorientation and frustration is completely legitimate.
Informative section is over. Shameless plug about an alternative that I am developing and how it is special follows.
Next Generation Shell as an Alternative
How Next Generation Shell differs from the alternatives? Reasonable question that I would ask too before investing any more time if I was the reader.
Current shells as well as proposed alternatives treat UI as nothing has happened since the 70-s: mostly typing commands and getting some text back.
How about real interaction with the objects on the screen? Oops. In a typical shell there are no objects on the screen, it’s just a dumpster of combined text (if you are lucky; could be binary) from stdout and from stderr from one or more processes (unless steps were taken). WTF? It doesn’t help you to win. It helps you to lose time. NGS does what is intended to make you productive. I have organized my thoughts about how the UI should look and behave on the wiki page.
It looks like alternative solutions have the “let’s make the shell better” approach and therefore are heavily based on the shell syntax and paradigms.
There is also “let’s make a library for existing language” approach which doesn’t hit the target either. Can’t have a syntax for common ops tasks for example.
And finally there is “let’s make a library and a syntax on top of existing language”. Not sure about this one. Sounds good in theory. Looked some time ago at something like this and the overall impression was … awkward (for the lack of better word).
Approach in NGS: let’s make a good programming language for Ops, which fits the use cases and has syntax and facilities for the most common tasks such as running external programs.
The language follows the principle that the most common tasks should have their own syntax or a library function (depending on usage frequency). Examples:
``external program`` – runs external program and parses the output (JSON is auto detected, easily extensible for anything else).
status(), log(), debug(), retry() – standard library functions. How many times an Ops person should write his/her own retry()? It’s insane.
Argv() facility for constructing command line parameters (for calling external program).
p=$(my_prog my_args &); ....; p.wait().
Small number of “big” core concepts in the language (types with inheritance, multiple dispatch and exceptions).
The UI (only recently started working on it) is in NGS. It doesn’t make sense that when a user of the shell wants to fix a bug in the UI and suddenly he/she needs to learn Go, Rust, C or whatever other language.
I am here to help. Do not hesitate to contact me with questions, suggestions, or feedback. If there is anything Ops-y you are trying to do seems to be easier in bash or Python – open an issue, because that’s a bug from my perspective. Something is inconvenient? Yep, also a bug.
If you are like me, you will find at least some satisfaction in using the most appropriate tool before continue to the myriad of other tasks that are in your todo queue.
Looking at some “exciting” features landing in Python 3.8, I’m still disappointed and frustrated by the language… like by quite a few other languages.
As an author of another programming language, I can’t stop thinking about how things “should have been done” from my perspective. I want to be explicit here. My perspective is biased towards correctness and “WTF are you doing?”. Therefore, take everything here with a appropriate amount of salt.
Yes, not talking about any “positive” changes here.
There is new syntax := that assigns values to variables as part of a larger expression.
A fix which couldn’t be the best because of previous design decision.
“Somebody” ignored the wisdom of Lisp, which was “everything is an expression and evaluates to a value” (no statements vs expressions), and made assignment a statement in Python years ago. Now this can not be fixed in a straightforward manner. It must be another syntax. Two different syntaxes for almost the same thing which is = for assignment as a statement and := for expression assignment.
There is a new function parameter syntax / to indicate that some function parameters must be specified positionally and cannot be used as keyword arguments:
def f(a, b, /, c, d, *, e, f):
print(a, b, c, d, e, f)
Trying to clean up a mess created by mixing positional and named parameters. Unfortunately I did not give it enough thought at the time and copied parameters handling behaviour from Python. Now NGS also has the same problem as Python had before 3.8. Hopefully, I will be able to fix it in some more elegant way than Python did.
functools.lru_cache() can now be used as a straight decorator rather than as a function returning a decorator. So both of these are now supported
OK. Bug fix. But … (functools.py)
if isinstance(maxsize, int):
# Negative maxsize is treated as 0
if maxsize < 0:
maxsize = 0
If you are setting LRU cache size to a negative number, it’s 99% by mistake. In NGS that would be an exception. That’s the approach that causes rm -rf $myfolder/ to remove / when myfolder is unset. Note that the maxsize code is not new but it’s still there in Python 3.8. I guess that is another mistake which can not be easily fixed now because that would break “working” code.
NGS had ordered maps from the start but that’s not a fair comparison because NGS project started in 2013, when the mistake was already understood.
How all that helps you, the reader? I encourage deeper thinking about the choice of programming languages that you use. From my perspective, all languages suck, while NGS aims to suck less than the rest for the intended use cases (tl;dr – for DevOps scripting).
It looks like the article above needs some clarification about my perspective: background, what I am doing and why.
The main points of the article are:
Everything still sucks, including Python. By sucks I mean does not fit well with the tasks I need to do neither aligned with how I think about these tasks.
I am trying to help the situation and the industry by developing my own programming language
Background about my Thinking
In general, I’m amazed with how bad the overall state of programming is. That includes:
All programming languages that I know including my own NGS. This is aggravated by inability to fix anything properly for any language with substantial amount of code written in it because you will be breaking existing code. And if you do break, you get the shitstorm like with Python 3 or Perl 6 (Raku).
Quality of available materials, which are sometimes plainly wrong.
Many of existing “Infrastructure as code” solutions, which in most cases follow the same path:
Invent a DSL or use YAML.
“figure out” later that it’s not powerful enough (by the way there is an elegant solution – a programming language, forgot the name)
Create pretty ugly programming language on top of a DSL that was intended for data.
I am creating new programming language and a shell out of frustration with current situation, especially with bash and Python. Why these two? Because that’s what I was and still using to get my tasks done.
From technical perspective, that’s probable: I am a skilled programmer in several languages and I have languages to look at more than everybody else had before. My disadvantage is not much experience in language design. I’m trying to offset that with thinking hard (about the language, the essence of what is being expressed, common patterns, etc), looking at other languages and experimenting.
From marketing perspective, I need to learn a lot. I am aware that “technically better” doesn’t matter as much as I would like to. Without community and users that would be a failed project.
Also don’t forget luck which I might or might not have.
What if NGS fails?
I think that the situation today is unbearable. I’m trying to fix it. I feel like I have to, despite the odds. I hope that even if NGS fails to move the industry forward it would be useful to somebody who will attempt that later.
First, we can get rid of “everybody” here because chances are that if you look carefully… it’s not everybody. Nice rhetoric though.
Second, the argument itself is invalid. The latter does not follow from the former.
Which problem are you solving?
Stop here and think before continuing. The more correct you define the problem the better are your chances of solving what you really need to be solving.
What’s the best solution to your problem?
When looking into alternative solutions, consider your circumstances: budget, people, knowledge, time frames, integration with existing products in use, etc.
In case that X is one of the alternatives to your problem, if “everybody” does X, there might be better documentation, resources, people available that know how to do X. That’s why you might want to consider X favourably.
JetBrains IDEs? No! They do too much! They are so into the code!
Vim? Emacs? Not pure enough! Who needs that stupid syntax highlighting?
Keep text editing pure! Any semantic understanding by the text editor is undesirable, other programs should handle that. You don’t want to complicate the text editor.
Developers are not saying that because user experience and productivity matter. Yet, “Use Dumb Shell” is considered to be an acceptable opinion. Is that so common that people fall on their heads so hard (alternatively, did not give it any thought)? WTF?
The solution (shell) should be as simple as possible but not simpler than possible. Current shells are simpler than required by good user experience. Wrong trade-off. Keeping something simple is important but not more important than the outcomes.
Additional food for thought:
Why use a car when bicycle is so much simpler?
Why use electricity when fire is so much simpler?
Why have water in your house when a wells are so much simpler?
I was doing consulting. The usual suspects: AWS, bash, Python, Puppet, Chef. Got to Terraform later. I had and I am still having subpar experiences with these tools. Anything I wanted to do, was overly burdensome, complicated and full of pitfalls.
Since I can’t attempt to fix everything, I picked the worst offender and started working on the alternative programming language and shell combo. The motivating opinion is that Ops have no good programming language nor adequate shell.
The dominant player is bash. It didn’t change much for decades: you type commands and get a dump of text on your screen. Most of the alternatives are essentially the same in this regard, for decades.
Is this because of the brilliant design? I would ask: which design? This? Quoting:
I wrote quite complex shell scripts and my first suggestion is “don’t”. The reason is that is fairly easy to make a small mistake that hinders your script, or even make it dangerous.
The “Dumb Shell” Approach
In this post I would like to address common thought that I hear from people regarding Next Generation Shell, a new programming language and a shell that I’m working on. Note that other shells which are more advanced than POSIX shells also get this. Quoting @cup from lobste.rs:
Wouldnt it be better to just have a dumb shell, that can call programs to do heavy lifting (read: programming languages). This way you have a “division of labor”. Shell works best for launching executables, and programming languages work best for handling data structures and algorithms.
No, it would not. I refuse to accept under-powered tools.
Dumbness is Fundamental Flaw
The “dumb shell” has no semantic understanding and doesn’t care about programs’ inputs nor outputs. Let’s see how it plays out.
Today, “Understanding” of programs’ inputs is covered by completion. Completion was added because “dumb shell” had horrible user experience. It’s slightly better now when the shell “understands” programs’ input to some degree. To some people completion is a scope creep. I think of it as better user experience and productivity gain.
“Understanding” of programs’ outputs? We are not there yet. It also seems that interacting with objects on the screen is too novel of an idea for the shell. Considering how much time this idea is out there: WTF?
Let’s see how this “dumbness” manifests as bad user experience even at the very basic, “intended” functionality:
Programs’ Output – Size
Do you know of any real world scenario when a human supposed to go over 10K lines on the screen? I mean just sit there and read it. Let me know. I’ve never seen such use case.
The shell is dumb, the shell “does not intervene” in programs’ outputs. Sounds good until you get unlimited number of lines dumped on your screen.
“Should have used less” you think later. Right. What if you forgot? The buffer is now filled with useless output and you can’t see outputs of previous programs. Are you being punished? No, just nobody cared about the UX. Alternatively, “it would be to complicated to implement”.
Programs’ Output – All Mixed
Want to know what’s on your screen is stdout and what is stderr? Well… you can’t. Your shell is dumb, it doesn’t deal with things like that.
Want to know from which program the output came from? Nope. Some programs cope with that to some degree by prepending their name to error messages: ls xxx gives you ls: xxx: No such file or directory. What a wonderful strategy! Keep the shell dumb and push the burden to all the programs.
You can’t type because some background job is continuing to dump text on the screen where you are trying to work? Too bad, should have used redirection because guess what … you shell doesn’t handle that either… and you can’t add redirection after the program is running; again not shell’s business.
Programs’ Output – Semantic Understanding
You just typed aws ec2 describe instances --filters ... and now you have some output.
You now see on your screen instance you would like to stop. The ID of the instance is right in front of your face. Now you type aws ec2 stop-instances --instance-ids. You would like to append the instance ID that you see on the screen. Nope. Your shell doesn’t do that. Too dumb. Select with the mouse and paste, because f*ck you!
Let’s imagine for a moment that the command output had some semantic meaning to the shell.
The shell would display the output as a table.
The table would be interactive (interactive output, what a heresy!) and one could navigate with arrow keys and have a shortcut for copy/paste the current cell value to the command line (for completion).
You could interact with the objects in the table with the mouse (very new concept, another heresy for the shell).
How about instead of typing aws ec2 stop-instances --instance-ids you navigate to the correct line, press enter, choose “stop” from the menu and the command is constructed for you? aws ec2 stop-instances --instance-ids i-123... amazing, ha? Well, your shell can’t do that.
Meaning, do you speak it mo***er?
How about after performing operations using the UI you would get as per your choice one of the below snippets which would re-create the operation:
Solution: UI for the Shell
Suppose I agree for a second, what do you suggest?
jq is a great tool. It does what bash can not – work with structured data. I use it. I would like not to use it.
In my opinion, working with structured data is such a basic thing that it makes much more sense to be handled by the language itself. I want my shell to be capable and I strongly disagree with the view that a shell “is not supposed to do that”. Shell is supposed to do whatever is needed to make my life easier. Handling structured data is one of these things.
If “shell is not supposed to do that”, by that logic, bash is not supposed to do anything except for running external commands and routing the data between them. Doesn’t it seem odd that bash does have builtin string manipulation then? Maybe bash shouldn’t have added associative arrays in version 4? … or arrays in version 2? How about if and while ? Maybe bash shouldn’t have them either?
jq is a symptom that bash can’t handle today’s reality: structured data. The world is increasingly more about APIs. APIs consume and return structured data. I do work with APIs from shell. Don’t you guys use AWS CLI or any other API that returns JSON?
The reality has changed. bash hasn’t. I’m working on bash alternative. Please help me with it. Or at least spread the word.
If you don’t like my project, join Elvish . Elvish is another shell that supports structured data.
I’m not saying that people that made Puppet and Ansible are not smart. It’s that we could learn from the mistakes they made… unless we don’t consider those being mistakes.
Puppet and Ansible went through very similar difficult situation. They have limited themselves to a declarative format and then they tried to accommodate the real life. Terraform has this situation right now.
The situation is:
Declarative format being used
People need something more powerful, like a programming language because … real life where conditionals, loops and data transformations make much more sense than working around declarative languages limitations.
Interestingly enough, they all did not switch to a proper programming language. Maybe because that would be at least partially admitting that the product should have been a library in the first place?
Terraform is actually in very crappy situation because even if they decide to expose everything as a library as the main interface, I don’t see people start using Go for “infrastructure as code”. Not as smooth as Ruby or Python anyway.
HCL to JSON one-to-one mapping. When I read “having a clean 1:1 mapping between HCL and JSON, and ensuring every feature of HCL is supported in JSON” I immediately thought that there must be converting tools then… and was not disappointed 🙂 “In future versions of Terraform, we will also support native tooling to convert HCL to JSON and JSON to HCL cleanly (including comments)”
Approach that in my eyes failed, again and again, is to start with your own declarative language and then with time grow the language. (SQL being among notable exceptions)
Puppet is the best example. map and each, added in Puppet 4.0.0 are, in my opinion, just two in a sea of evidence that the envisioned simple format has failed to handle the needs of the real world.
Ansible’s loop looks bad as the whole idea of making top levels of programs in YAML based syntax (and the rest in Python).
In my opinion, it makes more sense to create a language first and then libraries for it, not a library and then a language around it.
My hope for Terraform
I think Terraform guys are smart. Among other things, it manifests in implementing data sources. Data sources make Terraform much more flexible. I think it’s very clever.
Terraform, which started declarative, are now inventing their own programming language. They are going the way of Puppet and Ansible. I hope they can do better, in this awkward situation: there are quite a lot of constraints on the programming language because of the existing syntax and semantics.