Why NGS has no “undefined”

Since I know JavaScript, some Ruby and a bit of Perl which all have the concept of undefined it was a decision I had to make whether I implement undefined in NGS. This article shows why I decided not to have the undefined value/data type.

Update (thanks /u/EldritchSundae): what you observe in Ruby example below is nil, not undefined. In bash the undefined value is empty string. Ruby does not have undefined but it has the ability to read non-existing hash keys without causing an exception like JavaScript and Perl. In Ruby’s case the result is nil, not undef (Perl) or undefined (JavaScript).

Undefined in other languages

Showing here few common cases, not all possible usages.


> nodejs -e 'const a; console.log(a)'

> nodejs -e 'const h={}; console.log(h["xyz"])'

> nodejs -e '(function f(a,b) { console.log(a,b) })(1)'
1 undefined


> ruby -e 'h={}; puts h["xyz"]' # outputs empty line


> perl -e '%h=(); print $h{"xyz"}' # outputs nothing


> bash -c 'echo $a' # outputs empty line

Absence of undefined in NGS

Adding yet another data type to NGS needs justification. I can’t find any justification for undefined. I do consider the usages above bugs. Accessing a variable or a place that were not assigned any value is an error.

Conveying absence of a value in NGS is done similar to other languages with the special null value. There are also somewhat experimental Box, FullBox and EmptyBox types, similar to Option, Some and None in Scala.

Undefined as a hash value for non-existing keys

Having undefined returned when looking up non-existing hash key is a trade-off. It’s more convenient and more error-prone. I have chosen Python-like approach: it’s an error.

> ngs -e 'h={}; h["xyz"]'
... Exception of type KeyNotFound ...

# and added convenience method "get"
> ngs -p 'h={}; h.get("xyz", "NONE")'

Undefined when accessing unset variable

While bash gives you an empty string by default and Perl gives you undef, I do think accessing unset variable is definitely a bug. I guess it was understood at some point by the creators of bash and Perl so bash has -u flag that makes accessing undefined variable an error and Perl has use strict mode which does the same among other things.

> bash -c 'echo $a' # no error
> bash -c 'set -u; echo $a'
bash: a: unbound variable

> bash -c 'a=(); echo ${a[0]}' # no error, just horrible syntax :)
> bash -c 'set -u; a=(); echo ${a[0]}'
bash: a[0]: unbound variable

> perl -e 'print $a' # no error
> perl -e 'use strict; print $a;'
# no error - I have no idea why, probably some "special" variable

# Perl - take number two:
> perl -e 'print $abc' # no error
> perl -e 'use strict; print $abc;'
Global symbol "$abc" requires explicit package name
(did you forget to declare "my $abc"?) at -e line 1.
Execution of -e aborted due to compilation errors.

Undefined as value for parameters without arguments

Calling an NGS function with less arguments than it expects is an error as in most languages:

> ngs -e '(F (a,b) 10)(1)'
... Exception of type ArgsMismatch ..

By the way, I do cringe every time I see JavaScript code that explicitly uses undefined:

function f(optional_a, optional_b) { }
f(undefined, 10)

The programmer took a decision not to pass a value. How in the world is this undefined? Use null for f*ck sake!

Have a nice day!

Unicode characters as operators in a programming language¿

Wouldn’t it be cool to use Unicode characters as operators and maybe even function names?

if a ≠ b { ... }

range = 0…10

all_above_ten = myarr ∀ F(x) x > 10

Looks good, concise, expressive. Perl 6, Julia, Scala appear to support Unicode operators.

Why don’t I add Unicode operators and function names to NGS then?

If NGS would allow Unicode, it would be optional as I don’t want additional entry barrier or possible problems typing Unicode using remote connection. If I do add optional Unicode to NGS, here is what I think will happen next:

Some people start using Unicode in NGS while others don’t. Mixed code style emerges. It’s easy to imagine such mixed code style even in the same file as someone without the Unicode setup on his/her keyboard  is doing a quick fix. ViralBShah sums it up pretty well.

What do you think? Your comments are welcome.


Ruby’s do block in NGS experiment

Ruby has a very nice feature – a special syntax to pass a block of code as an argument to a method. I did want something like this NGS. Background, reasons and what this feature became follow.



NGS – Next Generation Shell is a language and a shell I’m working on for a few years now.

This article is about new NGS syntax inspired by and compared to the counterpart  Ruby’s do ... end syntax. The new syntax provides more convenient way of passing anonymous function to another function. Passing functions to other functions is (sometimes) considered to be a “functional programming” feature of a language.

In NGS, up until recently you had to pass all of your arguments (except for the first argument which can be passed before .method() ) inside the parenthesis of the method call syntax:

mymethod(arg1, arg2, ... argN)

or syntactically equivalent

arg1.mymethod(arg2, arg3, ... argN)

Passing an anonymous function could look like the following in NGS:

required_props.each(F(prop_name) {
    prop_name not in rd.anchor throws InvalidArgument("...")

The problem

In the code snippet above I don’t like the closing }) part. A single } would look much better. You could use the each operator % to get rid of the closing ).  So the code above becomes

required_props % F(prop_name) {
    prop_name not in rd.anchor throws InvalidArgument("...")

Which is shorter but not very clear (at least for newcomers) because of the % operator. I prefer to use the each operator in small expressions such as myarr % echo and not to use it in bigger expressions such as the snippet above. The closing ) syntax issue is not solved for the general case then so I wanted something like Ruby’s do block syntax.

Importing the “do” block

I’ve made quite a few modifications to fit NGS and to fit some OCD-ness:

The keyword

I prefer “with” because in my opinion, it’s much clearer that it links left-hand-side and right-hand-side. mymethod() with { ... some code ... } looks more clear to me than mymethod() do { ... some code ... }. It might be that I was affected by constructing a mini test framework where test("my feature") with { test code } looks very good.

Not only for code blocks

I don’t like “special” things so whatever comes after with can be any expression.

required_props.each() with F(prop_name) {
    prop_name not in rd.anchor throws InvalidArgument("...")


results = mylist.map() with my_super_mapper_func_name

Note than when using each it could probably be clearer to have the do keyword. I’m still unsure that with is the right choice

Code blocks syntax

You might ask how test("my feature") with { test code } works if what’s after with is any expression.

The test("my feature") with { test code } syntax works because { ... } is a syntax for anonymous function (with 3 optional parameters called A, B and C) and not because it’s part of the with syntax. You could get the same result using test("my feature", { test code }) .

Not just one

I don’t see any reason why with would be limited to one occurrence.

    with {
        while entry = c_readdir(d) {
    with {
        r = c_closedir(d)
        r != 0 throws DirFail('...')

Not a special parameter

As compared to Ruby, the finally() method does not use the special yield keyword to run the given arguments nor parameter should be declared with &. Here is the definition of finally() method (including the bug that cleanup can run twice, which I intend to fix).

F finally(body:Fun, cleanup:Fun) {
    try {
        ret = body()
    } catch(e) {
        throw e

Not special at all, just an additional syntax for arguments

The with syntax allows to write method call arguments outside the parenthesis. That’s all. What follows from this definition is the possibility of using keyword arguments:

        with body = {
                while entry = c_readdir(d) {
        with cleanup = {
                r = c_closedir(d)
                r != 0 throws DirFail('...')

Let’s shorten the syntax

The with name = val argument syntax looked a bit too verbose for me and with was stealing attention focus from the name. I’ve added the fat arrow syntax to combat that.

    body => {
        while entry = c_readdir(d) {
    cleanup => {
        r = c_closedir(d)
        r != 0 throws DirFail('...')

How the feature looks like now?

The snippet above refers to real code in stdlib. Additional examples of the with syntax follow:

Stdlib’s retry()


F body_missing_in_retry() throw InvalidArgument("...")

F retry(times=60, sleep=1, ..., progress_cb={null}, success_cb={A},
    fail_cb=null, body:Fun=body_missing_in_retry) {


F assert_resolvable(h:Str, title="Resolve host", times=45, sleep=2) {
	retry(times=times, sleep=sleep, title=title)
		body       => { `dig "+short" $h`.lines() }
		success_cb => { log_test_ok("...") }
		fail_cb    => { throw TestFail("...") }

test() mini framework for … tests!


F test(name:Str, f:Fun) {


test("Resolving gateway FQDN $gw_fqdn") with {
    assert_resolvable(gw_fqdn, "Resolve gateway FQDN")

Have your own ideas?

Please comment with your ideas regarding this, other and proposed NGS features, they can make it into the language. Or fork/improve/pull request.

Have a nice weekend!

Creating a language is easier now

I do look at other languages when designing and implementing a new shell and a language called NGS. As time goes by, there are more languages to look at and learn from. Hence, I think creating a language in recent years is easier than ever before. I would like to thank authors and contributors of all the languages!cubes-677092_640

NGS is heavily based on ideas that already existed before NGS. Current languages provide many of them and work on NGS includes sorting out which of these ideas resonate with my way of thinking and which don’t.

Of course NGS does have it’s own original ideas (read: I’m not aware of any other language that implemented these ideas) and they are critical to NGS, creating NGS is not just filtering others’ ideas 🙂

Some stolen ideas inspired by other languages

  1. Multi-methods & kind-of-object-orientation in NGS – mostly from CLOS
  2. External commands execution and redirection syntax – bash (while changing behaviour of the $ expansion, now known to be source of many bugs).
  3. Anonymous functions and closures – not sure where I got it from, I’d guess Lisp.
  4. Declarative primitives concept – heavily based on basic concept behind configuration management tools
  5. inspect() – Ruby‘s inspect()
  6. code() – the .perl property from Perl. Not fully implemented in NGS. Should return the code that would evaluate to the given value. Probably comes from Lisp originally which usually prints the values in a way it can read them back.
  7. Ordered hash (When you iterate a Hash, the items are in the same order as they were inserted) – PHP, also implemented in Ruby and if I’m correct in V8.
  8. The in and not in operators – Python.
  9. Many higher order functions such as map and filter probably originated in Scheme or Lisp.
  10. Translate anything that possible into function calls. + operator for example is actually a function call – Python, originally probably from Scheme or Lisp.

Ideas trashed before implementation

While working on NGS I think about features I can add. I have to admit that many of these candidate features seem fine for a few seconds to a few minutes… till I realize that something similar is already implemented in language X and it does not resonate with my way of thinking (also known as “This does not look good”).

Square brackets syntax for creating a list

Suppose we have this common piece of code:

list = []
some_kind_of_loop {

In this case it would advantageous to express the concept of building a list with a special syntax so it would be obvious from a first glance what’s happening. Since array literal already had [1,2,3,...] syntax, the new building-a-list syntax would be [ something here ]. And that’s how we get list comprehensions in Python (also in other languages). The idea is solid but when I see it implemented, it’s very easy for me to realize that I don’t like the syntax. List comprehension syntax is the preferred way to construct lists in Python and seems like it’s being widely used. I’d like to make clear that I do like Python and list comprehensions is one of the very few things I’m not fond of.

So what NGS has for building lists? Let’s see:

# Python
[x*2 for x in range(10) if x > 5]

# NGS straightforward translation
    if x > 5
      collect(x * 2)

Doesn’t look like much of an improvement. That’s because straightforward translations are not representative. The collector‘s use above is not a good example of it’s usage. More about collector later in this article.

Here are the NGS-y ways:

# Python
[x*2 for x in range(10) if x > 5]

# NGS alternatives, pick your compactness vs clarity
(0..10).filter(F(elt) elt>5).map(F(elt) elt*2)
10.filter(F(elt) elt>5).map(F(elt) elt*2)

NGS does not have special syntax for building lists. NGS does have special syntax for building anything and it’s called collector.

Syntax for everything

Just say “no”. It is tempting at first to make syntax for all the concepts in the language or at least for most of them. When taken to it’s extreme, we have Perl and APL.

I don’t like Perl’s syntax, especially sigils, especially that there is more than one sigil, (unlike $ in bash). Note that Perl kind of admitted sigils were used inconsistently (or confusingly? not sure) so sigils syntax was partially changed in Perl 6. On the other hand, Perl 6 added sigil modifiers, called “twigil“s. I counted 9 of them.

APL has many mathematical symbols in it’s syntax. Many of them are not on a keyboard. While the language is very terse and expressive, I don’t like the syntax and the fact that typing in a program in APL is not a straightforward task. [Update: Clarification: I’m not familiar with APL as opposed to other languages mentioned in this article. I was pointed out that I was quick to judge APL. It does have mathematical symbols but it actually doesn’t have much syntax. Most of the symbols are just built-in functions and operators. I’ve also found out one symbol for “go to line” and one for “define function”/”end function”. Actually APL syntax rules are interesting “think different” approach as they are implementation of mathematical notation. Thanks for the link, geocar.]

I do agree with Python’s developers reluctant attitude towards adding new syntax, especially new characters. Perl appears to be on the other end of the spectrum.

I’d like to come back to the collector syntax and show that even when adding a syntax, it’s advantageous to add less of it and reuse existing language concepts if possible.

The initial thought was to base the collector syntax on Lisp’s (loop …) macro , specifically the macro’s collect keyword: (loop ... (collect ...) ...). I was about do add two new keywords: collector and collect. The code would look like this:

mylist = collector
    if x > 5
      collect x * 2

I’ve started considering implementation strategies and came up with the idea that it would be simpler to have one keyword only, collector. collector wraps single expression to the right (which can be a { code block }) in a function with one argument named collect. The code becomes this:

mylist = collector
    if x > 5
      collect(x * 2)

When the collector expression is evaluated, the wrapped code runs with provided collect function. When collect is a function and not a special syntax it allows to use all the facilities that work with functions. Example: somelist.filter(...).each(collect) .

There is more to collector! It has optional initialization argument. Let’s see it in the following example:

    arr.each(F(elt) {
        if predicate(elt)

The code above counts number of elements in arr which satisfy the predicate. That’s roughly how standard library’s count() is implemented.

When collector is not followed by / (slash) and initialization value, the default initialization argument is newly created empty list. The default initial value was chosen to be empty list as it appears that most uses of collector

Advantages of the implementation where collect is a function and not a keyword:

  1. Simpler implementation
  2. Collector behaviour customization – there is a way to provide your own collect function for your types which I will not describe here. Standard library defines such functions for array, hash and integer types.
  3. Functional facilities available for the collect function
  4. collect function can have more than one argument making it possible to collect key-value pairs if constructing a Hash: collector ... collect(k, v) ... (this is from filter() implementation for Hash in standard library).

I consider collector to be a good example of “less syntax is better”.

Easier, not easy

I do enjoy the situation that I have many other languages to look at for ideas, good or bad.

NGS needs original features for it’s domain (systems administration tasks niche). These domain-specific features make NGS stand out. Other languages don’t have them because they are either not domain-specific or have wider domain or are too old to have features for the today’s systems administration domain. Every original feature has a risk attached to it. If you haven’t seen it work before there is practically no way to predict how good or bad a feature will turn out. More about original features in NGS in another post.

While creating a language is now easier than before it’s still not an easy task 🙂 … but you can help. Fork, code, make a pull request.

Have a nice weekend!

Update: discussion on reddit

Bashing bash – unexceptional

This is the third post in the “bashing bash” series. The aim of the series is to highlight problems in bash and convince systems and software engineers to help building a better alternative.

The problem

What is the output of the following script?


set -eu

echo Here
echo Not here

Right, the output is


Imagine that instead of false you have a lot of code. Since there are no exceptions, you have no idea where the error occurred.


Solutions using bash:

  1. Use set -x to trace the code.
  2. Add echo something every here and there to know between which two echo‘s the error occurred.
  3. Catch the error using trap and print the line number as suggested on StackOverflow . Writing this additional catching snippet at the top of every script is not really convenient.

This problem of unclear error location is unimaginable in any normal programming language.

“There is no problem, just don’t do it”

Bash was not intended to be a “normal” programming language. Some people say it’s an abuse of bash to use it as such. Looking at the code written in Bash I can tell it really is an abuse in many cases.

The reality though is that bash is still (ab)used for programming. In some cases Bash has positive aspects which outweigh the need to use other languages. In other cases a program starts as a small Bash script and is just not rewritten in another language after the script grows.

I suggest making a better shell rather than convincing people not to abuse Bash. People will keep on doing what they are doing. Let’s make their lives easier by providing them with a better shell.

The suggested solution

Use NGS. In NGS, any failed process throws an exception. Let’s take a look at the script below

#!/usr/bin/env ngs
echo Here
echo Not here

What’s the output?

Not here


Well, actually false returning an exit code of 1 is not an exception, it’s normal. If any command returning non-zero code would cause an exception, you wouldn’t be able to write for example if $(test -e myfile) do_something .

Failed process is a process that returns an unexpected exit code. Here is the part of stdlib that defines what’s a fail and what’s not:

F finished_ok(p:Process) p.exit_code == 0

F finished_ok(p:Process) {
    guard p.executable.path == '/bin/false'
    p.exit_code == 1

F finished_ok(p:Process) {
    guard p.executable.path == '/usr/bin/test'
    p.exit_code in [0, 1]

Such definitions also mean that you can easily extend NGS to work properly with any other command, simply by adding another finished_ok function. (Or add it to stdlib if it’s a common command so everyone would benefit).

So where are the exceptions?

We’ll have to modify the code to get an unexpected exit code. Example:

#!/usr/bin/env ngs
echo Here
ls nosuchfile
echo Not here


ls: cannot access 'nosuchfile': No such file or directory
========= Uncaught exception of type 'ProcessFailed' =========
====== Exception of type 'ProcessFailed' ======
=== [ backtrace ] ===
[Frame #0] /etc/ngs/bootstrap.ngs:158:1 - 158:10 [in <anonymous>]
[Frame #1] /etc/ngs/bootstrap.ngs:154:17 - 154:29 [in bootstrap]
[Frame #2] ./2.ngs:3:4 - 3:14 [in <anonymous>]
[Frame #3] /usr/share/ngs/stdlib.ngs:1116:11 - 1116:15 [in $()]
[Frame #4] /usr/share/ngs/stdlib.ngs:1050:29 - 1050:42 [in wait]
[Frame #5] /usr/share/ngs/stdlib.ngs:1006:7 - 1006:20 [in ProcessFailed]
=== [ dump process ] ===
(a lot of not very well formatted output with info about the process)

Please help building a better alternative

Go to https://github.com/ilyash/ngs/ and contribute some code.

Fundamental flaws of bash and its alternatives


Quoting Steve Bourne from  An in-depth interview with Steve Bourne, creator of the Bourne shell, or sh:

I think you are going to see, as new environments are developed with new capabilities, scripting capabilities developed around them to make it easy to make them work.

Cloud happened since then. Unfortunately, I don’t see any shells that could be an adequate response to that. Such shell should at the very least have data structures. No, not like bash, I mean real data structures, nested, able to represent a JSON response from an API and having a sane syntax.

In many cases bash is the best tool for the job. Still, I do think that current shells are not a good fit for today’s problems I’m solving. It’s like the time has frozen for shells while everything else advanced for decades.

As a systems engineer, I feel that there is no adequate shell nor programming language exist for me to use.


Bash was designed decades ago. Nothing of what we expect from any modern programming language is not there and somehow I get the impression that it’s not expected from a shell. Looks like years changed many things around us and bash is not one of them. It changed very little.

Looks like it was designed to be an interactive shell while the language was a bit of afterthought. In practice it’s used not just as an interactive shell but as a programming language too.

What’s wrong with bash?

Sometimes I’m told that there is nothing wrong with Bash and another shell is not needed.

Even if we assume that nothing is wrong with Bash, there is nothing wrong with assembler and C languages either. Yet we have Ruby, Python, Java, Tcl, Perl, etc… . Productivity and other concerns might something to do with that I guess.

… except that there are so many things wrong with bash: syntax, error handling, completion, prompt, inadequate language, pitfalls, lack of data structures, and so on.

While jq is trying to compensate for lack of data structures can you imagine any “normal” programming language that would outsource handling of data? It’s insane.

Silently executing the rest of your code after an error by default. I don’t think this requires any further comments.

Do you really think that bash is the global maximum and we can’t do better decades later?

Over the years there were several attempts to make a better alternative to bash.

Project Focus on shell and shell UX Powerful programming language
Bash No No
Csh No No
Fish shell Yes No
Plumbum No Yes, Python
RC shell No No
sh No Yes, Python
Tclsh No Yes, Tcl
Zsh Yes No

You can take a look at more comprehensive comparison at Wikipedia.

Flaw in the alternatives

A shell or a programming language? All the alternatives I’ve seen till this day focus either on being a good interactive shell with a good UX or on using a powerful language. As you can see there is no “yes, yes” row in the table above and I’m not aware of any such project. Even if you find one, I bet it will have one of the problems I mention below.

Focusing on the language

The projects above that focus on the language choose existing languages to build on. This is understandable but wrong. Shell language was and should be a domain-specific language. If it’s not, the common tasks will be either too verbose or unnecessarily complex to express.

Some projects (not from the list above) choose bash compatible domain-specific language. I can not categorize these projects as “focused on the language” because I don’t think one can build a good language on top of bash-compatible syntax. In addition these projects did not do anything significant to make their language powerful.

Focusing on the interactive experience

Any projects that I have seen that focus on the shell and UX do neglect the language, using something inadequate instead of real, full language.

What’s not done

I haven’t seen any domain-specific language developed instead of what we have now. I mean a language designed from ground up to be used as a shell language, not just a domain-specific language that happened to be an easy-to-implement layer on top of an existing language.

Real solution

Do both good interactive experience and a good domain-specific language (not bash-compatible).

List of features I think should be included in a good shell: https://github.com/ilyash/ngs/blob/master/readme.md

Currently I’m using bash for what it’s good for and Python for the rest. A good shell would eliminate the need to use two separate tools.

The benefits of using a good shell over using one of the current shells plus a scripting language are:

Development process

With a good shell, you could start from a few commands and gradually grow your script. Today, when you start with a few commands you either rewrite everything later using some scripting language or get a big bash/zsh/… script which uses underpowered language and usually looks pretty bad.


Same libraries available for both interactive and scripting tasks.

Error handling and interoperability

Having one language for your tasks simplifies greatly both the integration between pieces of your code and error handling.

Help needed

Please help to develop a better shell. I mean not an easy-to-implement, a good one, a shell that would make people productive and a joy to use. Contribute some code or tell your friend developers about this project.


I’m using Linux. I’m not using Windows and hope I will never have to do it. I don’t really know what’s going on there with shells and anyhow it is not very relevant to my world. I did take a brief look at Power Shell it it appears to have some good ideas.

Bashing bash – undefined variables

This is the second post in the “bashing bash” series. The aim of the series is to highlight problems in bash and convince systems and software engineers to help building a better alternative.

The problem

What does the following command do?

rm -rf $mydir/

Looks like the author would like to delete $mydir directory and everything in it. Actually it may do unexpected things because of missing quotes. The rant about quotes is in the previous post. This post is about yet another issue.

The correct commands should be:

set -u
rm -rf "$mydir/"

The important thing here is set -u . Without it, when $mydir is undefined for some reason, such as a bug in code preceding the rm command, there is a chance to brick the machine because an undefined variable becomes an empty string so the command is silently expanded to

rm -rf /


While more experienced engineers will usually use set -eu at the beginning of the script, omitting this declaration is a big trap for others.

Side note. You could ask why the original command has a trailing slash. The trailing slash is common and is used to signify a directory. While to the best of my knowledge the rm should work the same without the slash, some commands are actually more correct with trailing slash. For example cp myfile mydir/ would copy the file into the directory if it exists and would cause error if it doesn’t. On the other hand,  cp myfile mydir would behave the same if directory exists but would create a mydir file if there is no such directory nor file, which was not intended. Other commands such as rsync also behave differently with and without the slash. So it is common to use the slash.

See also: http://www.tldp.org/LDP/abs/html/options.html – bash options

The suggested solution

In NGS, any use of an undefined variable is an exception.

ngs -e 'echo(a)'

It’s going to look prettier but even in current implementation you have all the information about what happened:

========= Uncaught exception of type 'GlobalNotFound' =========
====== Exception of type 'GlobalNotFound' ======
=== [ dump name ] ===
* string(len=1) a
=== [ dump index ] ===
* int 321
=== [ backtrace ] ===
[Frame #0] /etc/ngs/bootstrap.ngs:156:1 - 156:10 [in <anonymous>]
[Frame #1] /etc/ngs/bootstrap.ngs:152:17 - 152:29 [in bootstrap]
[Frame #2] <command line -e switch>:1:8 - 1:9 [in <anonymous>]

While bash options probably have historical justification, a new language should not have such a mechanism. It complicates things a lot. In addition, the programmer should always be aware what are the current options.

Please help building a better alternative

Go to https://github.com/ilyash/ngs/ and contribute some code.

Bashing bash – variable substitution

This is the first post in the “bashing bash” series to highlight problems in bash and convince systems and software engineers to help building a better alternative.

Bash is a very powerful and useful tool, doing a better job than many other shells and programming languages when used for the intended tasks. Still, it’s hard to believe that writing a software decades later can not be done better.

The problem

What does the following command do?

cp $src_file $dst_file

One might think it copies the given file to the specified destination. Looking at the code we can say it was the intention. What would actually happen? It can not be known from the line above. Each $src_file and $dst_file expand to zero to N arguments so unexpected things that could happen. The correct command would be

cp "$src_file" "$dst_file"

Forgetting the quotes or leaving them out assuming  $src_file and $dst_file will always contain one bash word, expanding to exactly one argument each is dangerous.

Keeping quoting everything makes code cluttered.

The suggested solution

In NGS, $var expands to exactly one argument similar to "$var" in bash. The new syntax $*var, consistent with similar syntax in other NGS parts, would expand to zero to N arguments.

Please help building a better alternative

Go to https://github.com/ilyash/ngs/ and contribute some code.

AWS CLI inhuman output and what I suggest

The problem

If you use AWS you probably use AWS CLI. This tool gives you control over your resources from the command line. AWS CLI can output the results of your commands in several formats. None of the formats seem to be useful for humans.

AWS CLI can output the data in the following formats:

  1. JSON, the default – good for processing with jq or programming languages, definitely not good for a human looking at list of few or more instances.
  2. text – good for processing with command line tools such as awk, very hard to read for a human.
  3. table – WTF? Probably meant for humans. Cluttered and difficult for a human to process. See the screenshot:
AWS CLI. List of instances. –output table. Is it for humans?

You can make it look better but what you see above is in no way reasonable default and making it look prettier should not be as unpleasing as described in the user guide towards the end of the page.

My takes on solving this

Take 1 – old script

I have a script which uses outdated Ruby AWS client which does not work with eu-central regions because it’s old. It was originally built for EC2 classic so it fails to show VPC security groups.


If newer regions and VPC security groups issues are solved this is a fine tool for human output.

Take 2 – JQ

While the old script is not fixes I have this temporary solution: jq transformation of AWS CLI output JSON (from my .bashrc_aws). The output is one line per instance.

DESC () 
    aws ec2 describe-instances | jq -r '.Reservations[].Instances[] | "\(.InstanceId)%\(.KeyName)%\(.PublicIpAddress)%\(.LaunchTime)%\(.InstanceType)%\(.State.Name)%\(.Tags)"' | sed 's/.000Z//g' | column -t -s '%'

It works well when there are no tags or small amount of tags per instance. With few tags on an instance, the line becomes too long and wraps. Reformatting sample output here for your convenience. Original is one logical line, wrapped, takes almost two lines in the terminal window.


Take 3 – ec2din.ngs

This is one of the demo scripts in the NGS project that I’m working on.

The column after state (“running”) is the key name.

ec2din.ngs – human output

It would be better if actual available screen width would be considered so there would be no need to arbitrary trim security groups’ names and tags’ keys and values (the ... in the screenshot above).

How I think it should be solved

NGS (the new shell I’m working on) implements the Table type which is used to do the layout you see on “ec2din.ngs – human output” screenshot. Table is part of the standard library in NGS. As you see, it’s already useful in it’s current state. There is plenty of room for improvement and the plan is to continue improving Table beyond simply “useful”.

This is the right direction with the following advantages:

  1. The script that needs to output a table (for example ec2din.ngs) does not deal with it’s layout at all, making it smaller and more focused on the data.
  2. The script can set defaults but does not handle which columns are actually displayed and in which order. The Table type handles it using another NGS facility: config which currently gets the configuration from environment variables but should be able to read it from files and maybe other sources in future.
  3. Output of tabular data is a very common task so factoring it out to a library has all the advantages of code deduplication.

I would also like the shell to automatically detect heuristically or otherwise which data comes out of the programs that you run and display accordingly. Most of the data that is manipulated in shells is tabular and could be displayed appropriately, roughly as ec2din.ngs displays it. Maybe ec2din.ngs will become unnecessary one day.

In my dream, the output is semantic so that the shell knows what’s on the screen and allows you to navigate for example to i-xxxxx part, press a key and choose which operation you would like to perform on the instance. When you pick an operation, the shell constructs a command for such operation and executes it exactly as if you typed it in (so it’s in history and all other CLI advantages over GUI apply).