Opening Rant
You don’t hear one developer saying “Just use Notepad” to a colleague with argumentation that goes roughly like this:
Why are you using this horrible Visual Studio Code? It has built-in debugger! No!
JetBrains IDEs? No! They do too much! They are so into the code!
Vim? Emacs? Not pure enough! Who needs that stupid syntax highlighting?
Keep text editing pure! Any semantic understanding by the text editor is undesirable, other programs should handle that. You don’t want to complicate the text editor.
Developers are not saying that because user experience and productivity matter. Yet, “Use Dumb Shell” is considered to be an acceptable opinion. Is that so common that people fall on their heads so hard (alternatively, did not give it any thought)? WTF?
The solution (shell) should be as simple as possible but not simpler than possible. Current shells are simpler than required by good user experience. Wrong trade-off. Keeping something simple is important but not more important than the outcomes.
Image is a link to http://www.workcompass.com/
Additional food for thought:
- Why use a car when bicycle is so much simpler?
- Why use electricity when fire is so much simpler?
- Why have water in your house when a wells are so much simpler?
Background
I was doing consulting. The usual suspects: AWS, bash, Python, Puppet, Chef. Got to Terraform later. I had and I am still having subpar experiences with these tools. Anything I wanted to do, was overly burdensome, complicated and full of pitfalls.
Since I can’t attempt to fix everything, I picked the worst offender and started working on the alternative programming language and shell combo. The motivating opinion is that Ops have no good programming language nor adequate shell.
The absence of good programming language for ops was covered in another post. In this post I will cover some of the things that are wrong with the interactive shell.
The Shell
The dominant player is bash. It didn’t change much for decades: you type commands and get a dump of text on your screen. Most of the alternatives are essentially the same in this regard, for decades.
Is this because of the brilliant design? I would ask: which design? This? Quoting:
I wrote quite complex shell scripts and my first suggestion is “don’t”. The reason is that is fairly easy to make a small mistake that hinders your script, or even make it dangerous.
The “Dumb Shell” Approach
In this post I would like to address common thought that I hear from people regarding Next Generation Shell, a new programming language and a shell that I’m working on. Note that other shells which are more advanced than POSIX shells also get this. Quoting @cup from lobste.rs:
Wouldnt it be better to just have a dumb shell, that can call programs to do heavy lifting (read: programming languages). This way you have a “division of labor”. Shell works best for launching executables, and programming languages work best for handling data structures and algorithms.
No, it would not. I refuse to accept under-powered tools.
Dumbness is Fundamental Flaw
The “dumb shell” has no semantic understanding and doesn’t care about programs’ inputs nor outputs. Let’s see how it plays out.
Today, “Understanding” of programs’ inputs is covered by completion. Completion was added because “dumb shell” had horrible user experience. It’s slightly better now when the shell “understands” programs’ input to some degree. To some people completion is a scope creep. I think of it as better user experience and productivity gain.
“Understanding” of programs’ outputs? We are not there yet. It also seems that interacting with objects on the screen is too novel of an idea for the shell. Considering how much time this idea is out there: WTF?
Let’s see how this “dumbness” manifests as bad user experience even at the very basic, “intended” functionality:
Programs’ Output – Size
Do you know of any real world scenario when a human supposed to go over 10K lines on the screen? I mean just sit there and read it. Let me know. I’ve never seen such use case.
The shell is dumb, the shell “does not intervene” in programs’ outputs. Sounds good until you get unlimited number of lines dumped on your screen.
“Should have used less” you think later. Right. What if you forgot? The buffer is now filled with useless output and you can’t see outputs of previous programs. Are you being punished? No, just nobody cared about the UX. Alternatively, “it would be to complicated to implement”.
Programs’ Output – All Mixed
- Want to know what’s on your screen is stdout and what is stderr? Well… you can’t. Your shell is dumb, it doesn’t deal with things like that.
- Want to know from which program the output came from? Nope. Some programs cope with that to some degree by prepending their name to error messages:
ls xxxgives youls: xxx: No such file or directory. What a wonderful strategy! Keep the shell dumb and push the burden to all the programs. - You can’t type because some background job is continuing to dump text on the screen where you are trying to work? Too bad, should have used redirection because guess what … you shell doesn’t handle that either… and you can’t add redirection after the program is running; again not shell’s business.
Programs’ Output – Semantic Understanding
You just typed aws ec2 describe instances --filters ... and now you have some output.
You now see on your screen instance you would like to stop. The ID of the instance is right in front of your face. Now you type aws ec2 stop-instances --instance-ids. You would like to append the instance ID that you see on the screen. Nope. Your shell doesn’t do that. Too dumb. Select with the mouse and paste, because f*ck you!
Side note: amazing AWS engineers did not include any human readable output format so you get JSON dumped on your screen (or any other format which is still non-human-compatible).
Let’s imagine for a moment that the command output had some semantic meaning to the shell.
- The shell would display the output as a table.
- The table would be interactive (interactive output, what a heresy!) and one could navigate with arrow keys and have a shortcut for copy/paste the current cell value to the command line (for completion).
- You could interact with the objects in the table with the mouse (very new concept, another heresy for the shell).
- How about instead of typing
aws ec2 stop-instances --instance-idsyou navigate to the correct line, press enter, choose “stop” from the menu and the command is constructed for you?aws ec2 stop-instances --instance-ids i-123...amazing, ha? Well, your shell can’t do that.
Meaning, do you speak it mo***er?
How about after performing operations using the UI you would get as per your choice one of the below snippets which would re-create the operation:
- CLI commands
- CloudFormation tempalte
- Terraform “code”
Solution: UI for the Shell
Suppose I agree for a second, what do you suggest?
https://github.com/ngs-lang/ngs/wiki/UI-Design
I personally don’t see how the described features could be implemented as external programs, keeping the shell “dumb”.
We Can Do Better Today
The reality has changed. What was once amazing is subpar by today’s standards. The world outside of the shell moved forward while the shell stayed almost the same. Brilliant design? Brilliant what?
Let’s move this industry together from the stone age of bash shell to the bronze age of something a bit less subpar – Next Generation Shell.
Closing Rant
Imaginary UNIX people:
We wanted to separate things because they are semantically different so we split the things into stdout and stderr. Well… stderr was is actually for everything that is not stdout.
One bit of metadata (stdout vs stderr) for semantic meaning of the output should be enough for everyone forever. Well… at least it’s simple for us to implement.
Update: discussion on lobste.rs
Ilya's blog 
Please excuse my French, but it seems like a number of facts about bash have been ignored.
0. “I refuse to accept under-powered tools.” – Why does every tool need to be all-powerful? That results in bloatware. Build your set of tools instead! A hammer is a hammer, and it should not be lightweight for it is a hammer, and not a paperclip. You may try to use the paperclip as a hammer, but it is not the tool for the task. A paperclip always asks if it could help you, like in Office. 🙂
[ ] Don’t show this message again
1. 10k lines. Using the dumb shell approach, there is grep (and egrep and zgrep, etc), which can do the dirty filtering and pattern matching work for you. It is not the job of the shell. What is the job of the shell? To return those lines to you (Neo), or another program. (Agent Smithish smile.)
2. There is a thing called command completion. Use the TAB key! And that is a scriptable thing, so you
can make it work with your freshly compiled binary as well. Using the dialog utility for making choices would result in just what you have described above. It also works with half-typed options. Like:
openssl x5 {Wtf was that number again?} [TAB] [TAB] {zero-nine, yeah}
3. Redirections. stdin, stdout, and stderr are just some defaults. You can create as many as you wish. Also, dupx is a utility to manage redirects of already running processes, following the dumb shell approach. Not part of the default installation. Think about it: should manipulating already running separate processes be the task of a dumb shell? There are task managers for that.
4. “Solution: UI for the Shell” – there is one. It is text based. aka TUI. Which also includes CLI. It has some invisible features as well.
JSON is still human readable. Take a look at a binary stream! That is not human readable, however, I personally know certain human beings, who can read and interpret a hexdump of network traffic, and that is not an urban legend. There are much more things that are possible to do, than which are convenient to do.
“Is that so common that people fall on their heads so hard (alternatively, did not give it any thought)? WTF?” – No. They have not fallen on their heads. It is a different paradigm. Developers often think they understand everything better than others. Well, it is not true.
LikeLike
> a number of facts about bash have been ignored.
Which exactly?
> Why does every tool need to be all-powerful?
Not all powerful but it should allow me to be productive.
> Build your set of tools instead!
That’s what I’m doing … and that’s a shell. I can’t build intended functionality on top of any existing shell.
> 10k lines.
What exactly did you count?
Is that too many? Too few? Is it good? Bad?
Relative to what? To 13961 lines of .h files and 143710 lines of .c in bash 4.4?
And the most important: do you understand the provided functionality?
> There is a thing called command completion
No shit, dude 🙂
> Using the dialog utility for making choices would result in just what you have described above
Nope. I don’t see any way to populate that dialog with contents from the sreen, taking into account semantic types of the things. Do you?
> dupx
While the script does show some bash skill…
“use gdb to attach to the process and then do a dup(2) call on its file descriptors.”
I would really prefer cleaner solution and one which was not only “has been tested on Fedora systems” and for sure not work on MacOS because it uses /proc … and gdb is not installed typically.
> JSON is still human readable.
Not when it’s coming out of AWS describe instances command for example. It’s too big to have remotely good UX.
> It is a different paradigm
Yep, the Notepad paradigm, with which too many people are comfortable.
The simplicity for the sake of simplicity can not be objective. When simplicity butchers the UX, I’m choosing the software to be more complex. I’m using IDEs and only sometimes text editors. Do you use Notepad most of the time?
LikeLike
I started off with Bash scripting before I got into OOP. I just kind of accepted the cringing task of parsing string output and working with utilities like grep and sed. Once I learned actual programming, I loathed having to create any kind of script as they always ended up feeling kind of hacky. I considered writing programs to do the logic that I would need, but it seemed like such overkill to have to create a service or a program to execute what seemed like simple tasks.
With that said, I’ve just recently started learning some PowerShell (after dragging my feet) and I’ve really taken a liking to it. Versions 6 and 7 are cross platform and will work on both Mac and Linux. Everything returns .Net objects and can easily be converted to human-readable formats, such as tables where you can specify which properties to display. You can also pipe outputs directly into other cmdlets too. There’s also error-handling built right in which has been improved with version 7.
LikeLike