Section Syntax – Next Generation Shell

Problem

Using comments to denote code sections feels like subpar solution.

One starts with something like the following:

// workaround for API stupidity
if(result === null) {
  result = [];
}

Then somebody adds another bit so it becomes:

// workaround for API stupidity
if(result === null) {
  result = [];
}
if(result === [1]) {
  foo();
}

Now you are not sure whether the second if is still workaround. You don’t want that. What I usually do in this situation and recommend to others is clearly mark start and end:

// workaround for API stupidity - start
if(result === null) {
  result = [];
}
// workaround for API stupidity - end

if(result === [1]) {
  foo();
}

Now you have duplicated text and subpar programming experience.

Solution

Today (2019-10-21) I have added section syntax (to dev branch) to the language I am working on, Next Generation Shell. I think it solves the problem in a clean way, consistent with syntax and semantics of the language:

section "workaround for API stupidity" {
  if result is Null {
    result = []
  }
}

Or:

result = section "Use algorithm X to calculate blah" {
  a = 1
  b = 2
  a + b
}

In future, for programmer’s convenience backtraces could be augmented with sections’ names.

Update: discussion

  1. https://www.reddit.com/r/ProgrammingLanguages/comments/dkzcls/section_syntax_next_generation_shell/
  2. https://lobste.rs/s/gert97/section_syntax_next_generation_shell

Have a nice week!

On Information Loss in Software

“Information Loss” is a way to look at the world. The topic is very broad. This blog post will focus on information loss during development and operation of computer software.

This post discusses why Information Loss is bad and gives some examples.

My hope is that after reading this post, you will be able to spot information loss more easily. This should help you avoiding information loss, eliminating the need for costly information recovery phase. Some examples include specific recommendations how to avoid that particular case of information loss.

Information Loss Definition

Information Loss for the purposes of this blog is the situation where information I is available and is easily accessible at point in time t1 but later, when it’s needed at point in time t2, it is either not available or not easily accessible.

The post will present various categories of information loss with examples. The list is not exhaustive; it’s not meant to be. The intention is to give some examples to help you get the feel and start looking at things from the information loss perspective.

Why Information Loss is Bad?

In many cases of Information Loss, the missing information can be recovered but that requires resources to be thrown at the issue (time and/or money). That is the situation I would like to help you to avoid.

Between the Head and the Code

When working on software, the first place the information loss occurs is when the programmer translates thoughts into code. Information loss at this stage will manifest itself as increased WTF-per-minute during code review or just code reading. Each time the code is read, there will be additional cognitive load while the reader reconstructs the programmer’s idea behind the code.

I have identified two main causes for information loss at the head-to-code stage:

  • Programmer’s fault
  • Programming language imposed

Information Loss due to Programmer’s Fault

The more a programmer is experienced, the less likely is the occurrence of information loss at this stage.

Misnamed Variable

In programmers head: number of servers running the ETL task. Name of the variable in the code: n. WTFs at code review – guaranteed.

Misnamed Function

I’m pretty sure getUser() should not update say last name of the user in database. Such naming is criminal but unfortunately I’ve seen code similar to that.

Use of Magic Numbers

if (result == 126) .... The person who wrote 126 knew what that number means. The person reading the code will need to spend time checking what that number means. One should use constants or enums instead: if (result == NOT_EXECUTABLE) ....

Missing Comments in Code

Most important comments are about why something is being done as opposed to how. If ones code is in a high-level language and of a good quality, it’s a rare occasion one needs to comment about what or how something is being done. On the other hand comments like “Working around API bug: it returns false instead of empty array” are very valuable.

Incorrect Usage of Data Types

A list of people, for example, is not just a list. It has semantic meaning. It’s much easier to understand a program when correct types are used for the data. Java has generics to convey such information, for example List<Person>. Some other languages have type systems that are powerful enough to convey such information too.

Programming Language Imposed Information Loss

Limitations of programming languages lead to less expressive code because the idea in programmer’s head can not be expressed in a straightforward manner. The readers of the code will struggle more (read waste time) to understand the code.

Unnamed Function Parameters

bash and perl5 (not sure about perl5 anymore, there was something experimental) do not have the syntax for specifying function parameter names. This makes the code less expressive. Sometimes programmers will do “the right thing”:

myfunc() {
    local target_file=$1
    ...
}

… but when they don’t, you finish with unnamed parameter, wondering what it could mean:

myfunc() {
    if [[ -f $1 ]];then
        ...
    fi
}

Is that a file to generate or a source file? You don’t know, you have to read on in myfunc hoping for the answer.

Recommendation: even if your language does not support named parameters, emulate them.

Expansion of Strings into Several Arguments (bash)

rm $x

Does that remove one file or several? What the programmer meant? You simply don’t know. It depends on the contents of x, which is typically split into arguments by spaces. You are lucky if you can deduce from the variable name whether it’s one or several files.

From today’s perspective this is just bad design. Back at the day I guess it was the most practical way to implement arrays.

Recommendation: use one of the two alternatives blow and do not use rm $x form.

  • Single file: rm "$x" (proper quoting)
  • Multiple files: rm "${my_files[@]}" (bash arrays)

Side note: this “feature” caused so much pain over the years when x would contain a spaces by accident. Even when x is meant to be used as an array, elements of that array can also contain spaces by accident.

Error Handling

In languages that do not support exceptions (bash, C, Go), the programmer is forced into one of two situations:

  • Write incorrect code that ignores the errors (on purpose or by mistake, go figure which one)
  • Write verbose code that handles the errors. When the code handles every possible error, it becomes cluttered with error handling and it takes more time to understand the code. That’s the case where information loss occurs because the reader is overwhelmed by the code.

In NGS, since typical use case is scripting, I wanted to have the option for the code to be concise. That rules out returning status code along with the result because the caller is then forced to check it. It does make more sense for NGS to have exceptions and for scripts to decide whether to catch them or let the whole script terminate with error because of an uncaught exception.

Unordered Hash/Map/dict Data Structure

Hash data structure is implemented in a non-order-preserving manner in some languages. That means that the programmer can not express the intention freely in situations where the order of key/value pairs is important. That pushes towards less readable code as the programmer fights the language by implementing his/her own ordered dictionary.

Information loss in this case is again losing the sight of programmer’s intention.

Fortunately many modern languages solved the issue by now:

Recommendation: check whether your language has the data structure you really want to use, either built-in or in a library.

Limited Data Structures (bash)

Working with data structures in bash results more or less convoluted code, depending on the data structures one need to work with. This is direct consequence of bash supporting exactly three data structures:

  • Scalar (strings which can sometimes be treated as numbers or arrays)
  • Array
  • Associative array

These data structures can not be nested.

The result is much less readable code where the original intent of the author is harder to recover as opposed to data manipulation in other popular languages (Python, Ruby, etc).

Recommendation: consider using other languages besides bash for heavy data manipulation code.

Absence of non-nullable Types

In some languages there is no straightforward way to specify non-nullable parameters. The programmers are then required to check whether each passed parameter is null. That results more boilerplate code. Let’s look at the following bit of Java code from the popular Apache Flink project:

// flink/flink-java/src/main/java/org/apache/flink/api/java/DataSet.java

protected DataSet(ExecutionEnvironment context, TypeInformation<T> typeInfo) {
    if (context == null) {
        throw new NullPointerException("context is null");
    }
    if (typeInfo == null) {
        throw new NullPointerException("typeInfo is null");
    }

    this.context = context;
    this.type = typeInfo;
}

Asynchronous Computing Model (JavaScript)

In JavaScript for example, progressively more readable code uses:

Again, information loss occurs when programmer’s intention is lost in the code because the code looks like a big struggle against asynchronicity and the language.

Recommendation: prefer async/await over Promises and prefer Promises over callbacks.

Loss of semantic information (JavaScript)

console.log() vs debug('my-module')('my message') in JavaScript. When a programmer chooses to use log() instead of debug(), loss of semantic information occurs. In this case it means more effort in finding the needed information in the output as opposed to simpler turning on and off the relevant debug sections.

Recommendation: use the debug module.

Information Loss at Runtime

Information loss at runtime will manifest as harder debugging.

Empty Catch Clause

This is borderline criminal. Except for very few cases when empty catch clause is really appropriate, by placing empty catch clause in the code, you are setting up a bomb for your colleagues. They will pay with their time, tears and mental health, not to mention they will be hating you. Where is the information loss? At the time the exception is generated, there is useful information about what happened. Empty catch clause loses that information. Result: hard to find exceptions and their causes.

In NGS, there are clear ways to express that you didn’t just forgot to handle the exception (try ... catch(e) { }) but you actually don’t care (or know exactly) what happened:

  • try EXPR without the catch clause at all. If EXPR throws exception, try EXPR evaluates to null, otherwise evaluates to EXPR.
  • EXPR tor DFLT if EXPR throws an exception, evaluates to DFLT, otherwise evaluates to EXPR.

Writing to stdout Instead of stderr

stdout has semantic meaning (result of the computation) and stderr also has semantic meaning (errors description). It will make harder for any wrapper script to deal with a program that outputs errors to stdout or outputs the result to stderr. The semantic information about the text is lost and then needs to be recovered by the caller if the two outputs are mixed.

Wrong exit codes reporting

This one really hinders automation.

if ... then {
    ...
    error("error occurred")
    exit(0) # incorrect error code reported
}

Since it’s easy to forget about exit code, and the common case is that exit() means abnormal termination of the program, in NGS exit() that does not provide an exit code defaults to exit code 1.

Wrong exit codes handling

if [ -e MY_FILE ] ...

This is all over bash scripts… and it’s wrong. Which exit codes [ program/built-in returns? Zero for “yes”, one for “no”, and two for “An error occurred”. Guess what. You can’t handle three distinct cases with two if branches; “An error occurred” is causing the “false” branch of the if to be taken. If you are lucky, you will spot error message on stderr. If you are not lucky, your script will just work incorrectly in some circumstances.

At this point the tradeoff in NGS was made in favor of correctness, not simplicity. if $(test -e MY_FILE) ... in NGS can go three ways: “yes” branch, “no” branch and an exception. After any external process is finished, NGS checks the exit code. For unknown process, non-zero exit code cases an exception. For test and a few others, zero and one are not causing an exception. The exit code checking facility is extensible and one can easily “teach” NGS about new programs.

Broaden your Horizon – Extras

I’ll mention here non-strictly software development related information loss cases.

Untagged Cloud Resources (AWS)

Have you just created an EC2 instance and named it Server or maybe you haven’t tagged it at all? Congratulations, semantic information has just been lost. You colleagues will strugle to understand what is the role of instance.

Recommendation: rigorously tag the resources, have alerts for untagged or improperly tagged resources. In AWS you can also know who created the resource by looking at CloudTrail.

Side note: In Azure, any resource must belong to a “Resource Group” which makes it much easier to track the resources.

GUI

You just performed operation in GUI. The information of what happened was just lost the minute you performed the operation. Good luck reproducing or documenting it.

The plan to combat this in NGS is to have textual representation for each operation that is performed via GUI.

String Concatenation

Every time two strings are concatenated into one, there is some information loss.

Recommendation: instead of parsing unstructured text (result of concatenation) later, consider using structured data format when producing the output. (Example: JSON).


Hope that helps. Have fun!

What I did not steal from Perl 6

I’m curious about programming languages. Not because I’m creating one right now. I always was. This post is about ideas and features that I have seen in Perl 6 and found interesting. If you are curious about programming languages in general, you should take a look at these.

There are various reasons for not stealing the interesting ideas from Perl 6:

  1. I’m trying to keep number of concepts in NGS as small as possible. If I’m not seeing huge immediate value in a concept – I skip it.
  2. Not taking anything that I think can confuse me or other programmers. I’m not talking here because someone is a beginner. I’m talking about confusing concepts.
  3. Simply because I don’t have enough resources to implement it at the moment.

Here are the interesting Perl 6 features, in no particular order (except the first one). There are also my comments whether I would like the feature in NGS or why not.

  1. Syntax. Very expressive an terse. Perl6 has even more of it than Perl 5. Now that we got rid of the $ and friends in the room:
  2. Grammars. Would actually be nice to have something like that in NGS.
  3. Lots of operators. The most interesting concept is Metaoperators. I’m trying to keep the amount of syntax elements in NGS relatively low. There are already two syntaxes in NGS: commands and expressions. Not taking more syntax without serious need.
  4. How the “pointy block” syntax mixes with “for” syntax: for @list -> @element . NGS already has several syntaxes for Lambdas.
  5. Flow control
    1. when” flow control. The closest NGS has is “cond” and friends, stolen from Lisp.
    2. repeat while / repeat until . It would be nice to have something like that in NGS.
    3. once . Not sure about this one. The functionality might be needed.
  6. Slips. The behaviour is frightening me: if it does expand, how do I pass a Slip if I just want to pass it, say as an item of an array? NGS uses syntax for slips: [1, 2, *myitems, 3, 4] which I think is cleaner. You know you can’t pass it because it’s syntax.
  7. .WHAT method. I stole something similar from Ruby: the inspect method.

As a special note, I have seen a welcome change from $arr[0] to @arr[0] . I think it removes confusion. (That was Perl 5 vs Perl 6).

Please don’t be offended if you are a Perl 6 hacker and you see that there is amazing feature that I have not mentioned. It could be that I’ve seen this in several other languages already or maybe I did not find it interesting or … maybe I just missed it. Don’t hesitate to leave a comment anyway.


Happy coding, in whatever language rocks your boat! Except for bash. Coding in bash will never be happy.

List of JSON tools for command line

I am considering making a JSON parsing and generating command line tool. Started with looking around a bit. Below is a list of existing JSON command line tools. Numbers are [GitHub stars] at the time of writing this post. (… contributed by …) means that this post was updated with the item.

  • jq [11126] – filter, extract, modify and output JSON or text using DSL
  • jid [4426] – “You can drill down JSON interactively by using filtering queries like jq.” (item contributed by /u/Tacticus)
  • gron [4103] – convert JSON or JSON lines (from file/stdin/url) to text (path=value) which can be processed with grep/sed/diff; the tool also supports converting back to JSON after such processing
  • jo [2209] – generate JSON based on command line arguments and stdin; can read data from files and place it as base64 encoded values
  • JSON.sh [1635] – written in shell/gawk; “traverses the JSON objects and prints out the path to the current object (as a JSON array) and then the object, without whitespace”
  • underscore-cli [1588] ‘THE “Swiss Army knife” tool for processing JSON data – can be used as a simple pretty-printer, or as a full-powered JavaScript command-line’. Added on 2019-09-30 following comment from @joeytwiddle.
  • jsawk [1239] – focused primarily on filtering and transforming a list (or an object). Update 2019-09-30: as @joeytwiddle suggested in comment, the project appears to be unmaintained and doesn’t work with recent Node.js versions. Latest commit and latest closed issue are from 2015.
  • json (by trentm) [1218] – “massaging JSON on your Unix command line”; JS-like syntax for extracting values; in-place file editing
  • rq [1007] – awk/sed-like tool for structured data; supports several formats, including JSON
  • TickTick [469] – use JSON syntax directly in bash; “This is just a fun hack”
  • jshon [309] – very CLI-ish way to extract, manipulate and output the data
  • jl [308] – “a tiny functional language for querying and manipulating JSON”; visually reminds Haskell
  • jsonpp [244] – JSON pretty printer (item contributed by /u/ferbass)
  • fx [227] – conveniently run your JS code to manipulate JSON.
  • RecordStream [224] – create, manipulate and output records; supports JSON; Perl-based so grep expressions for example are in Perl.
  • JSON.awk [186] – JSON.sh fork in awk; after fork the projects added different features.
  • jp [184] – “command line interface to JMESPath” (link contributed by Evgeny Zislis)
  • json-command [143] – conveniently manipulate JSON using JS.
  • jsonv.sh [130] – convert JSON to CSV; specify paths in JSON to
  • jgrep (aka “JSON-grep”) [78] – “Command line tool and API for parsing JSON documents” in Ruby (item contributed by /u/tophlammiepie)
  • jsed [48] – manipulate and extract data; somewhat similar to jsawk in mindset
  • jayin [10] “Piping with js at terminal”. Added on 2019-09-30 following comment from @joeytwiddle.
  • jsongrep [9] (by dsc) – extract data at given path using shell globs and output one per line
  • jc [2] – “jc is used to JSONify the output of many standard linux cli tools”. Added on 2019-10-29 following comment from Kelly Brazil.
  • jsongrep [0] (by terrycojones) – easily extract data at given path

Honorable mentions

Update 2018-09-10:

I’ve added related post in which I argue that jq functionality belongs to a shell.


If you feel that some project is missing from the list, please let me know in comments below.

The missing link of Ops tools

It’s like we went from horse to spaceship, skipping everything in between.

Background

Let’s say you are managing your system in AWS. Amazon provides you with API to do that. What are your options for consuming that API?

Option 1: CLI or library for API access

AWS CLI let’s us access the API from the command line and bash scripts. Python/Ruby/Node.js and other languages can access the API using appropriate libraries.

Option 2: Declarative tools

You declare how the system should look like, the tool figures out dependencies and performs any API calls that are needed to achieve the declared state.

Problem with using CLI or API libraries

Accessing API using CLI or libraries is fine for one off tasks. In many cases, automation is needed and we would like to prepare scripts. Ideally, these scripts would be idempotent (can be run multiple times, converging to the desired state and not ruining it). We then quickly discover how clunky these scripts are:

# Script "original"
if resource_a exists then
  if resource_a_property_p != desired_resource_a_property_p then
    set resource_a_property_p to desired_resource_a_property_p
  end
  if resource_a_property_q != desired_resource_a_property_q then
    ...
  end
else
  # resource_a does not exist
  create resource_a
  set resource_a_property_p to desired_resource_a_property_p
  ...
end
# more chunks like the above

It’s easy to see why you wouldn’t want to write and maintain a script such as above.

How the problem was solved

What happened next: jump to “Option 2”, declarative tools such as CloudFormation, Terraform, etc.

rocket-1374248_640

Other possible solution that never happened

If you have developed any code, you probably know what refactoring is: making the code more readable, deduplicate shared code, factoring out common patterns, etc… without changing the meaning of the code. The script above is an obvious candidate for refactoring, which would be improving “Option 1” (CLI or a library for API access) above, but that never happened.

All the ifs should have been moved to a library and the script could be transformed to something like this:

# Script "refactored"
create_or_update(resource_a, {
  property_p = desired_resource_a_property_p
  property_q = desired_resource_a_property_q
})
# more chunks like the above

One might say that the “refactored” script looks pretty much like input file of the declarative tools mentioned above. Yes, it does look similar; there is a huge difference though.

Declarative tools vs declarative primitives library

By “declarative primitives library” I mean a programming language library that provides idempotent functions to create/update/delete resources. In our cases these resource are VPCs, load balancers, security groups, instances, etc…

Differences of declarative tools vs declarative primitives library

  1. Declarative tools (at least some of them) do provide dependency resolution so they can sort out in which order the resources should be created/destroyed.
  2. Complexity. The complexity of mentioned tools can not be ignored; it’s much higher than one of  declarative primitives library. Complexity means bugs and higher maintenance costs. Complexity should be considered a negative factor when picking a tool.
  3. Some declarative tools track created resources so they can easily be destroyed, which is convenient. Note that on the other hand this brings more complexity to the tool as there must be yet another chunk of code to manage the state.
  4. Interacting with existing resources. Between awkward to impossible with declarative tools; easy with correctly built declarative primitives library. Example: delete all unused load balancers (unused means no attached instances): AWS::Elb().reject(X.Instances).delete()
  5. Control. Customizing behaviour of your script that uses declarative primitives library is straightforward. It’s possible but harder with declarative tools. Trivial if in a programming language can look like count = "${length(var.public_subnets) > 0 ? 1 : 0}" (approved Terraform VPC module).
  6. Ease of onboarding has declarative tools as a clear winner – you don’t have to program and don’t even need to know a programming language, but you can get stuck without knowing it:
  7. Getting stuck. If your declarative tool does not support a property or a resource that you need, you might need to learn a new programming language because the DSL used by your tool is not the programming language of the tool itself (Terraform, Puppet, Ansible). When using declarative primitives library on the other hand you can always either extend it when/if you wish (preferable) or make your own easy workaround.
  8. Having one central place where potentially all resources are described as text (Please! don’t call that code, format is not a code!). It should be easier done with declarative tools. In practice, I think it depends more on your processes and how you work.

As you can see, it’s not black and white, so I would expect both solutions be available so that we, Ops, could choose according to our use case and our skills.

My suggestion

I don’t only suggest to have something between a horse and a spaceship; I work on a car. As part of the Next Generation Shell (a shell and a programming language for ops tasks) I work on declarative primitives library. Right now it covers some parts of AWS. Please have a look. Ideally, join the project.

Next Generation Shell – https://github.com/ilyash/ngs

Feedback

Do you agree that the jump between API and declarative tools was too big? Do you think that the middle ground, declarative primitives approach, would be useful in some cases? Comment here or on Reddit.


Have a nice day!

Why I have no favorite programming language

TL;DR – because for me there is no good programming language.

I’m doing mostly systems engineering tasks. I manage resources in Cloud and on Linux machines mostly. I can almost hear your neurons firing half a dozen names of programming languages. I do realize that they are used by many people for systems engineering tasks:

  • Go
  • Python
  • Ruby
  • Perl
  • bash

The purpose of this post is not to diminish the value of these languages; the purpose is to share why I don’t want to use any of the languages above when I write one of my systems-engineering-task scripts. My hope is that if my points resonate with you, the reader, you might want to help spread the word about or even help with my suggested solution described towards the end.

man-2756206_640

So let’s go over and see why I don’t pick one of the languages:

Why not language X?

All languages

  • Missing smart handling of exit codes of external processes. Example in bash: if test -f my_file (file is not there, exit code 1) vs if test --f my_file (syntax error, exit code 2). If you don’t spot the syntax error with your eyes, everything behaves as if the file does not exist.
  • Missing declarative primitives libraries (for Cloud resources and local resources such as files and users). Correction: maybe found one, in Perl – (R)?ex ; unfortunately it’s not clear from the documentation how close it is to my ideas.

All languages except bash

  • Inconvenient/verbose work with files and processes. Yes, there are libraries for that but there is no syntax for that, which would be much more convenient. Never seen something that could compare to my_process > my_file or echo my_flag > my_file .

Go

  • Compiled
  • Error handling is a must. When I write a small script, it’s more important for me for it to be concise than to handle all possible failures; in many cases I prefer an exception over twice-the-size script. I do understand how mandatory and explicit error handling can be a good thing for larger programs or programs with greater stability requirements.
  • Dependencies problem seem to be unresolved issue

Python

  • Functional programming is second level citizen. In particular list/dictionary comprehension is the Pythonic way while I prefer map and filter. Yes, that’s probably one of the features that makes Python easier to learn and suggested first language. Not everything that’s optimized for beginners must be good for more experienced users. It’s OK.
  • Mixed feelings about array[slice:syntax] . It’s helpful but slice:syntax is only applicable inside [ ] , in other places you must use slice(...) function to create the same slice object

Ruby and Perl

  • The Sigils syntax does not resonate with me.

Ruby

I can’t put my finger on something specific but Ruby does not feel right for me.

Perl

  • Contexts and automatic flattening of lists in some cases make the language more complicated than it should.
  • Object orientation is an afterthought.
  • Functions that return success status. I prefer exceptions. Not the default behaviour in Perl but an afterthought: autodie.
  • Overall syntax feeling (strictly matter of personal taste).

bash

Note that bash was created in a world that was vastly different from the world today: different needs, tasks, languages to take inspiration from.

  • Missing data structures (flat arrays and hashes is not nearly enough). jq is a workaround, not a solution in my eyes.
  • Awkward error handling with default of ignoring the errors completely (proved to be bad idea)
  • Expansion of undefined variable to empty string (proved to be bad idea)
  • -e ,  -u and other action at a distance options.
  • Unchecked facts but my feelings:
    • When bash was created, it was not assumed that bash will be used for any complex scripting.
    • bash was never “designed” as a language, started with simple commands execution and other features were just bolted on as time goes by while complete redesign and rewrite were off the table, presumably for compatibility.
  • Syntax
  • No widely used libraries (except few for init scripts) and no central code repository to search for modules (Correct me if I’m wrong here. I haven’t heard of these).

My suggested solution

I would like to fill the gap. We have systems-engineering-tasks oriented language: bash. We have quite a few modern programming languages. What we don’t have is a language that is both modern and systems-engineering-tasks oriented. That’s exactly what I’m working on: Next Generation Shell. NGS is a fully fledged programming language with domain specific syntax and features. NGS tries to avoid the issues listed above.

Expected questions and my answers

People work with existing languages and tools. Why do you need something else?

  • I assume I have lower bullshit tolerance than many others. Some people might consider it to be normal to build more and more workarounds (especially around anemic DSLs) where I say “fuck this tool, I would already finish the task without it (preferably using appropriate scripting language)”. I don’t blame other people for understandable desire to work with “standard” tools. I think it’s not worth it when the solutions become too convoluted.
  • I am technically able to write a new programming language that solves my problems better than other languages.

Another programming language? Really? We have plenty already.

  • I would like to remind you that most of the programming languages were born out of dissatisfaction with existing ones.
  • Do you assume that we are at global maximum regarding the languages that we have and no better language can be made?

Feedback

Would you use NGS? Which features it must have? What’s the best way to ease the adoption? Please comment here, on Reddit (/r/bash , /r/ProgrammingLanguages) or on Hacker News.


Update: following feedback roughly of the form “Yes, I get that but many Ops tasks are done using configuration management tools and tools like CloudFormation and Terraform. How NGS compares to these tools” – there will be a blog post comparing NGS to the mentioned tools. Stay tuned!


Have a nice day!

NGS unique features – Argv command line arguments builder

Background: what is NGS?

NGS LOGO

NGS, the Next Generation Shell is a (work in progress) shell and a programming language built ground up for systems engineering tasks. You can think of it as bash that’s designed today: sane syntax, data structures, functional programming, extensibility, cloud in mind, declarative primitives.

What’s the problem with constructing command line arguments?

The problem affects only more “advanced” cases of constructing command line arguments when some arguments might or might not be present. Let’s consider this example:

# Made-up syntax, resembling NGS
args = []
if 'Subnets' in props {
  args += '--subnets'
  args += props['Subnets']
}
if ... {
  args += ...
}
if ... {
  args += ...
}
...
aws elb create-load-balancer ... $args

Wouldn’t it be cleaner to get rid of all the ifs? … and what happens if props['Subnets'] is an empty array?

How Argv facility in NGS solves the problem?

Argv is a result of factoring out the common code bits involved in constructing command line arguments. The ifs above were also factored out. They are now in Argv.

Let’s look at usage example (real NGS code, from AWS library)

argv = Argv({
  '--load-balancer-name': rd.anchor.name
  '--listeners': props.ListenerDescriptions.encode_json()
  '--subnets': rd.opt_prop('Subnets', props).map(only(ResDef, ids))
})
rd.run('create ELB', %(aws elb create-load-balancer $*argv))

The important points here are:

  1. Argv is a function with a single parameter which must be of type Hash (also called “dictionary” in some languages)
  2. The keys of the Hash are switches’ names (--load-balancer-name, --listeners, --subnets)
  3. The values of the Hash are values for the switches

The “if” that decides whether a switch is present in the resulting argv is inside Argv implementation and your code is clean of it. The values of the Hash are considered when Argv decides whether a switch should be present. null, empty array and instances of type EmptyBox are considered by Argv as missing values and it discards the switch. For convenience, instances of type FullBox are unboxed when constructing the result of Argv.

The Argv facility is yet another point among others that shows why NGS and systems engineering tasks are best fit.


Have a nice weekend!