NGS unique features – Argv command line arguments builder

Background: what is NGS?


NGS, the Next Generation Shell is a (work in progress) shell and a programming language built ground up for systems engineering tasks. You can think of it as bash that’s designed today: sane syntax, data structures, functional programming, extensibility, cloud in mind, declarative primitives.

What’s the problem with constructing command line arguments?

The problem affects only more “advanced” cases of constructing command line arguments when some arguments might or might not be present. Let’s consider this example:

# Made-up syntax, resembling NGS
args = []
if 'Subnets' in props {
  args += '--subnets'
  args += props['Subnets']
if ... {
  args += ...
if ... {
  args += ...
aws elb create-load-balancer ... $args

Wouldn’t it be cleaner to get rid of all the ifs? … and what happens if props['Subnets'] is an empty array?

How Argv facility in NGS solves the problem?

Argv is a result of factoring out the common code bits involved in constructing command line arguments. The ifs above were also factored out. They are now in Argv.

Let’s look at usage example (real NGS code, from AWS library)

argv = Argv({
  '--listeners': props.ListenerDescriptions.encode_json()
  '--subnets': rd.opt_prop('Subnets', props).map(only(ResDef, ids))
})'create ELB', %(aws elb create-load-balancer $*argv))

The important points here are:

  1. Argv is a function with a single parameter which must be of type Hash (also called “dictionary” in some languages)
  2. The keys of the Hash are switches’ names (--load-balancer-name, --listeners, --subnets)
  3. The values of the Hash are values for the switches

The “if” that decides whether a switch is present in the resulting argv is inside Argv implementation and your code is clean of it. The values of the Hash are considered when Argv decides whether a switch should be present. null, empty array and instances of type EmptyBox are considered by Argv as missing values and it discards the switch. For convenience, instances of type FullBox are unboxed when constructing the result of Argv.

The Argv facility is yet another point among others that shows why NGS and systems engineering tasks are best fit.

Have a nice weekend!


NGS unique features – improving NodeJS require()

Background: what is NGS?

NGS, the Next Generation Shell is a (work in progress) shell and a programming language built ground up for systems engineering tasks. You can think of it as bash that’s designed today: sane syntax, data structures, functional programming, extensibility, cloud in mind, declarative primitives.

What’s good in NodeJS’ require()

I like most of how require() works in JavaScript. I’m not talking in this post about npm, just NodeJS require() function. require() does not pollute your namespace, you just get a reference, it’s simple to use and easy to reason about.

const a = require('cool-aws-wrapper');
// Can not be done easily with AWS SDK:

What’s there to improve in require() ?

NodeJS modules are usually fall into one of the categories:

  1. Class definition / big library that manages it’s own namespace. These usually end with module.exports = MyClass. No problem here.
  2. Group of functions or classes. These usually end with module.exports = { func1, func2, func3, ...} lists (ES6 syntax, otherwise written as module.exports =  { func1: func1, ... } ) which I think are cumbersome.

How require() and modules look in NGS?

Note that require() in NGS is work in progress and it doesn’t have much of the functionality that NodeJS provides. I just started with things that bothered me the most.

Consistent with other places in NGS, require() returns the last evaluated expression. NodeJS for example returns module.exports which you must explicitly set as the result of require().

I think of modules primary as a namespaces. Creating a namespace in NGS has a syntax: ns { ... } .

Combining require() behaviour of returning last evaluated expression and namespace syntax, typical NGS module consists of single top level expression which evaluates to a namespace. The whole module file can look like this:

ns {

  global init

  type Vpc
  type Subnet

  F init(v:Vpc) {

  F _helper_func(s:Str) { ... }

  MY_CONST = 42

  F ok() {


Let’s ignore the global for now, it’s about how methods and types’ instances creation are implemented in NGS. Anything defined inside the ns { ... } is exposed as namespace member so usage of the above module could look like this:

  m = require('mymodule.ngs')
  vpc = m::Vpc()

As you probably guessed, the :: operator is the namespace member access operator.

There is no need to explicitly state what module/namespace exports. That’s the improvement over NodeJS’ require().

How ns works and more options for the curios

ns { … } returns a Hash

As stolen from NodeJS, the namespace syntax (ns { ... }) returns a Hash. In NodeJS, require() typically returns JavaScript Object which is close enough for the purpose of this post.

About :: operator

The namespace member access operator :: is actually a Hash key access operator. It is helpful because the regular syntax for accessing members is not always a good fit for namespaces. The regular member access syntax is dot (.) but the dot syntax is also a function call: myobj.field – is a field/key/attribute access but myobj.func() is equivalent to func(myobj). For example, m::ok() will call the ok function defined in the module, m.ok() will call the function ok in current lexical environment with m as parameter.

As a bonus, since :: is an operator, it is implemented as function call. This means you can define how :: works with types that you define and modify how :: works with existing types.

ns { … } syntax implementation

For simplicity of implementation and absence of obvious reasons against, ns { ... } syntax is just a syntactic sugar for defining anonymous function without parameters and calling it immediately. The though behind this decision was simple: “I need to implement namespaces. Let’s see where I have them already. Oh, namespaces are already implemented in functions. This is so convenient, I can use this mechanism with minimal effort”.

How ns knows what to return?

ns is mostly a syntactic hack:

  1. Inside the ns body, the first statement, before any use-supplied statements is _exports = {} which sets the local variable _exports to an empty Hash.
  2. Any assignment and function definition also set _exports["something"]. MY_CONST = 42 becomes MY_CONST = 42;  _exports["MY_CONST"] = MY_CONST;
  3. Exception to the rule above are variables and functions with names starting with underscore (_). They are not automatically added to _exports. This for example is why _exports itself is not exported.
  4. Last statement, after all user-supplied statements is _exports.

The behavior I just described looks like sane defaults to me. As we all know, the life is usually more complex than hello world examples and customizations are need. Here are two ways to customize the resulting namespace.

  1. return your_expr – since ns is just a function, you can use return at any point to return your own custom namespace.
  2. manipulate _exports however you want towards the end of ns body. For example after _exports .= filterv(Type) only types will be exported. _exports.filterk(/^pub_/) will only export symbols (keys) that have names that start with pub_ .

Improvement suggestions are welcome! Have a nice day!

NGS unique features – Hash methods I wish I had in other languages

NGS is a language and a shell that I am building for systems administration tasks. Enough of the language is implemented to enable writing some useful scripts. The shell is not there yet.

Some of the Hash methods in NGS

The methods for working with Hash I have not seen all at once in other languages are:

  1. filterk – filter Hash by key (produces Hash)
  2. filterv – filter Hash by value (produces Hash)
  3. mapk – map Hash keys (produces Hash)
  4. mapv – map Hash values (produces Hash)
  5. mapkv – map Hash keys and values (produces Hash as opposed to map which produces an array)
  6. without – filters out specific key

How these are actually used? Following is an excerpt from the pollute method (function), which is a part of the AWS module. It uses several of the Hash methods I mentioned, making the method a good example. pollute method (as in “pollute global namespace”) enables using Vpc variable for example instead of AWS::Vpc and so on. I would like to have this behaviour for small quick-and-dirty scripts but not as default so it’s in a method that one can optionally call.

F pollute(do_warn=true) {

    vars =
        _exports.filterk(/^AMI_OWNER/) +
        _exports.filterv(Type).without('Res').without('ResDef') +

    if do_warn {
        warn("Polluting ...: ${vars.keys().join(', ')}")


Let’s go over the code above step by step:

F pollute(do_warn=true) { ... } defines the pollute method with optional parameter do_warn that has default value of true.

_exports is a Hash containing all of the AWS’ module variables and functions, similar to NodeJS module.exports but members are added automatically rather than explicitly. Only the methods and variables that do not start with _ (underscore) are added. One can modify _exports in any way before the end of the module. I will write more in detail about require() and modules in NGS in another post.

filterk(/^AMI_OWNER/) filters all the variables that match the given RegExp

filterv(Type) filters all the variables that are of type Type. These are AWS types’ definitions, such as Vpc, Subnet or Instance.

without('...') filters out the types I don’t like to override.

+ between and after _exports.filterk(...) and _exports.filterv(...) joins the hashes.

mapk translates variables’ names into their index (using resolve_global_variable)

each runs set_global_variable with variable index and the value to set

Hash methods in other languages

I am aware that some of the methods above are present in other languages or libraries. Some examples:

  1. Ruby has mapv (transform_values) method.
  2. Rails has mapk (transform_keys) and mapv.
  3. Perl 6 can modify values in a convenient manner:for %answers.values -> $v is rw { $v += 10 };.

What I have not seen is a language which has all the methods above out of the box. I have a feeling that arrays get all the fame methods while hashes (dictionaries) often get less attention in other languages.

Why NGS has all these methods?

NGS is aiming to be convenient for systems administration tasks. More often than not these tasks include data manipulation. NGS has many functions (methods) for data manipulation, including the ones listed in this post.

Update: reddit discussion

Have a nice day!

NGS unique features – exit code handling


How other languages treat exit codes?

Most languages that I know do not care about exit codes of processes they run. Some languages do care … but not enough.

Update / Clarification / TL;DR

  1. Only NGS can throw exceptions based on fine grained inspection of exit codes of processes it runs out of the box. For example, exit code 1 of test will not throw an exception while exit code 1 of cat will throw an exception by default. This allows to write correct scripts which do not have explicit exit codes checking and therefore are smaller (meaning better maintainability).
  2. This behaviour is highly customizable.
  3. In NGS, it is OK to write if $(test -f myfile) ... else ... which will throw an exception if exit code of test is 2 (test expression syntax error or alike) while for example in bash and others you should explicitly check and handle exit code 2 because simple if can not cover three possible exit codes of test (zero for yes,  one for no, two for error). Yes, if /usr/bin/test ...; then ...; fi in bash is incorrect! By the way, did you see scripts that actually do check for three possible exit codes of test? I haven’t.
  4. When -e switch is used, bash can exit (somewhat similar to uncaught exception) when exit code of a process that it runs is not zero. This is not fine grained and not customizable.
  5. I do know that exit codes are accessible in other languages when they run a process. Other languages do not act on exit codes with the exception of bash with -e switch. In NGS exit codes are translated to exceptions in a fine grained way.
  6. I am aware that $? in the examples below show the exit code of the language process, not the process that the language runs. I’m contrasting this to bash (-e) and NGS behaviour (exception exits with non-zero exit code from NGS).

Let’s run “test” binary with incorrect arguments.


> perl -e '`test a b c`; print "OK\n"'; echo $?
test: ‘b’: binary operator expected


> ruby -e '`test a b c`; puts "OK"'; echo $?
test: ‘b’: binary operator expected


> python
>>> import subprocess
>>> subprocess.check_output(['test', 'a', 'b', 'c'])
... subprocess.CalledProcessError ... returned non-zero exit status 2
>>> subprocess.check_output(['test', '-f', 'no-such-file'])
... subprocess.CalledProcessError: ... returned non-zero exit status 1


> bash -c '`/usr/bin/test a b c`; echo OK'; echo $?
/usr/bin/test: ‘b’: binary operator expected

> bash -e -c '`/usr/bin/test a b c`; echo OK'; echo $?
/usr/bin/test: ‘b’: binary operator expected

Used /usr/bin/test for bash to make examples comparable by not using built-in test in bash.

Perl and Ruby for example, do not see any problem with failing process.

Bash does not care by default but has -e switch to make non-zero exit code fatal, returning the bad exit code when exiting from bash.

Python can differentiate zero and non-zero exit codes.

So, the best we can do is distinguish zero and non-zero exit codes? That’s just not good enough. test for example can return 0 for “true” result, 1 for “false” result and 2 for exceptional situation. Let’s look at this bash code with intentional syntax error in “test”:

if /usr/bin/test --f myfile;then
  echo OK
  echo File does not exist

The output is

/usr/bin/test: missing argument after ‘myfile’
File does not exist

Note that -e switch wouldn’t help here. Whatever follows if is allowed to fail (it would be impossible to do anything if -e would affect if and while conditions)

How NGS treats exit codes?

> ngs -e '$(test a b c); echo("OK")'; echo $?
test: ‘b’: binary operator expected
... Exception of type ProcessFail ...

> ngs -e '$(nofail test a b c); echo("OK")'; echo $?
test: ‘b’: binary operator expected

> ngs -e '$(test -f no-such-file); echo("OK")'; echo $?

> ngs -e '$(test -d .); echo("OK")'; echo $?

NGS has easily configurable behaviour regarding how to treat exit codes of processes. Built-in behaviour knows about false, test, fuser and ping commands. For unknown processes, non-zero exit code is an exception.

If you use a command that returns non-zero exit code as part of its normal operation you can use nofail prefix as in the example above or customize NGS behaviour regarding the exit code of your process or even better, make a pull request adding it to stdlib.

How easy is to customize exit code checking for your own command? Here is the code from stdlib that defines current behaviour. You decide for yourself (skipping nofail as it’s not something typical an average user is expected to do).

F finished_ok(p:Process) p.exit_code == 0

F finished_ok(p:Process) {
    guard p.executable.path == '/bin/false'
    p.exit_code == 1

F finished_ok(p:Process) {
    guard p.executable.path in ['/usr/bin/test', '/bin/fuser', '/bin/ping']
    p.exit_code in [0, 1]

Let’s get back to the bash if test ... example and rewrite the it in NGS:

if $(test --f myfile)
    echo("File does not exist")

… and run it …

... Exception of type ProcessFail ...

For if purposes, zero exit code is true and any non-zero exit code is false. Again, customizable. Such exit code treatment allows the if ... test ... NGS example above to function properly, somewhat similar to bash but with exceptions when needed.

NGS’ behaviour makes much more sense for me. I hope it makes sense for you.

Update: Reddit discussion.

Have a nice weekend!

NGS unique features – execute and parse

I am developing a shell and a language called NGS. I keep repeating it’s domain specific. What are the unique features that make NGS most suitable for today’s system administration tasks (a.k.a “Operations” or hype-compatible word “DevOps”)?

This post is first in series that show what makes NGS unique.


Execute and parse operator

Execute-and-parse operator … executes a command and parses it’s output. This one proved to be central in working with AWS API. Citing ec2din.ngs demo script:

``aws ec2 describe-instances $*filters``

The expression above returns a data structure. The command is run, the output is captured and then fed to parse() method. Whatever the parse() method returns is the result of the ``exec-and-parse syntax`` expression above.

Built-in parsing

By default, NGS parses any JSON output when running a command using ``exec-and-parse`` syntax. (TODO: parse YAML too)

In case with AWS CLI commands additional processing takes place to make the data structure coming out of exec-and-parse operator more useful:

  1. The top level of AWS responses is usually a hash that has one key which has an array as value: {"LoadBalancerDescriptions": [NGS, returns, this] } . While I can guess few reasons for such format, I find it much more useful to have an array as a result of running an AWS CLI command and that’s what NGS returns if you run ``aws ...`` commands.
  2. Specifically for aws ec2 describe-instances I’ve removed the annoyance of having Reservations list with instances as sub-lists. NGS returns flat instances list. Sorry, Amazon, this is much more productive.

Customizable parsing

What if you have your own special command with it’s own special output format?

The parsing is customizable via defining your own parse(s:Str, hints:Hash) method implementation. That means you can define how your command is parsed.

 No parsing

Don’t want parsed data? No problem, stick with the `command` syntax instead of ``command``. In case you need original data structure you can use `command`.decode_json() for example.

Why exec-and-parse is an operator?

Why adding an exec_parse() function would not be sufficient?

  1. Execute-and-parse is common operation in system tasks so it should be short. NGS has taken the pragmatic approach: the more common the operation, the shorter the syntax.
  2. Execute-and-parse should look similar `execute-and-capture-output` syntax which already existed when I was adding execute-and-parse.
  3. Making it an operator allows the command to be executed to be written in “commands syntax” (a bit bash-like) which is a better fit.

“I can add this as a function to any language!”

Sure but:

  1. Your chances of getting same brevity are not very good
  2. Making exec-and-parse as flexible as in NGS in other languages would be an additional effort
  3. ``some-command arg1 arg2`` – would it be exec_parse(['some-command', 'arg1', 'arg2']) ? How do you solve the syntax of the passed command? The array syntax does not look good here. Not many languages will allow you to have special syntax for commands to be passed to exec_parse().

If your language is not domain-specific for system tasks, adding exec-and-parse to it will be a task with dubious benefit.

How extreme opposite looks like

Just came across build configuration file of Firefox: settings.gradle (sorry, could not find a link to this file on a web in a sane amount of time). Here is excerpt with lines wrapped for convenience.

def commandLine = ["${topsrcdir}/mach", "environment", "--format",
    "json", "--verbose"]
def proc = commandLine.execute(null, new File(topsrcdir))
def standardOutput = new ByteArrayOutputStream()
proc.consumeProcessOutput(standardOutput, standardOutput)


import groovy.json.JsonSlurper
def slurper = new JsonSlurper()
def json = slurper.parseText(standardOutput.toString())


if (json.substs.MOZ_BUILD_APP != 'mobile/android') {

Here is how roughly equivalent code looks in NGS (except for the new File(topsrcdir) which I don’t understand):

json = ``"${topsrcdir}/mach" environment --format json --verbose``
if json.substs.MOZ_BUILD_APP != 'mobile/android' {

Yes, there are many languages where exec-and-parse functionality looks like something in between Gradle and NGS. I don’t think there is one that can do what NGS does in this regard out of the box. I’m not saying NGS is better than other languages for all tasks. NGS is aiming to be better at some tasks. Dealing with I/O and data structures is definitely a target area.

Have a nice day!

Why NGS has no “undefined”

Since I know JavaScript, some Ruby and a bit of Perl which all have the concept of undefined it was a decision I had to make whether I implement undefined in NGS. This article shows why I decided not to have the undefined value/data type.

Update (thanks /u/EldritchSundae): what you observe in Ruby example below is nil, not undefined. In bash the undefined value is empty string. Ruby does not have undefined but it has the ability to read non-existing hash keys without causing an exception like JavaScript and Perl. In Ruby’s case the result is nil, not undef (Perl) or undefined (JavaScript).

Undefined in other languages

Showing here few common cases, not all possible usages.


> nodejs -e 'const a; console.log(a)'

> nodejs -e 'const h={}; console.log(h["xyz"])'

> nodejs -e '(function f(a,b) { console.log(a,b) })(1)'
1 undefined


> ruby -e 'h={}; puts h["xyz"]' # outputs empty line


> perl -e '%h=(); print $h{"xyz"}' # outputs nothing


> bash -c 'echo $a' # outputs empty line

Absence of undefined in NGS

Adding yet another data type to NGS needs justification. I can’t find any justification for undefined. I do consider the usages above bugs. Accessing a variable or a place that were not assigned any value is an error.

Conveying absence of a value in NGS is done similar to other languages with the special null value. There are also somewhat experimental Box, FullBox and EmptyBox types, similar to Option, Some and None in Scala.

Undefined as a hash value for non-existing keys

Having undefined returned when looking up non-existing hash key is a trade-off. It’s more convenient and more error-prone. I have chosen Python-like approach: it’s an error.

> ngs -e 'h={}; h["xyz"]'
... Exception of type KeyNotFound ...

# and added convenience method "get"
> ngs -p 'h={}; h.get("xyz", "NONE")'

Undefined when accessing unset variable

While bash gives you an empty string by default and Perl gives you undef, I do think accessing unset variable is definitely a bug. I guess it was understood at some point by the creators of bash and Perl so bash has -u flag that makes accessing undefined variable an error and Perl has use strict mode which does the same among other things.

> bash -c 'echo $a' # no error
> bash -c 'set -u; echo $a'
bash: a: unbound variable

> bash -c 'a=(); echo ${a[0]}' # no error, just horrible syntax :)
> bash -c 'set -u; a=(); echo ${a[0]}'
bash: a[0]: unbound variable

> perl -e 'print $a' # no error
> perl -e 'use strict; print $a;'
# no error - I have no idea why, probably some "special" variable

# Perl - take number two:
> perl -e 'print $abc' # no error
> perl -e 'use strict; print $abc;'
Global symbol "$abc" requires explicit package name
(did you forget to declare "my $abc"?) at -e line 1.
Execution of -e aborted due to compilation errors.

Undefined as value for parameters without arguments

Calling an NGS function with less arguments than it expects is an error as in most languages:

> ngs -e '(F (a,b) 10)(1)'
... Exception of type ArgsMismatch ..

By the way, I do cringe every time I see JavaScript code that explicitly uses undefined:

function f(optional_a, optional_b) { }
f(undefined, 10)

The programmer took a decision not to pass a value. How in the world is this undefined? Use null for f*ck sake!

Have a nice day!

Unicode characters as operators in a programming language¿

Wouldn’t it be cool to use Unicode characters as operators and maybe even function names?

if a ≠ b { ... }

range = 0…10

all_above_ten = myarr ∀ F(x) x > 10

Looks good, concise, expressive. Perl 6, Julia, Scala appear to support Unicode operators.

Why don’t I add Unicode operators and function names to NGS then?

If NGS would allow Unicode, it would be optional as I don’t want additional entry barrier or possible problems typing Unicode using remote connection. If I do add optional Unicode to NGS, here is what I think will happen next:

Some people start using Unicode in NGS while others don’t. Mixed code style emerges. It’s easy to imagine such mixed code style even in the same file as someone without the Unicode setup on his/her keyboard  is doing a quick fix. ViralBShah sums it up pretty well.

What do you think? Your comments are welcome.


Ruby’s do block in NGS experiment

Ruby has a very nice feature – a special syntax to pass a block of code as an argument to a method. I did want something like this NGS. Background, reasons and what this feature became follow.



NGS – Next Generation Shell is a language and a shell I’m working on for a few years now.

This article is about new NGS syntax inspired by and compared to the counterpart  Ruby’s do ... end syntax. The new syntax provides more convenient way of passing anonymous function to another function. Passing functions to other functions is (sometimes) considered to be a “functional programming” feature of a language.

In NGS, up until recently you had to pass all of your arguments (except for the first argument which can be passed before .method() ) inside the parenthesis of the method call syntax:

mymethod(arg1, arg2, ... argN)

or syntactically equivalent

arg1.mymethod(arg2, arg3, ... argN)

Passing an anonymous function could look like the following in NGS:

required_props.each(F(prop_name) {
    prop_name not in rd.anchor throws InvalidArgument("...")

The problem

In the code snippet above I don’t like the closing }) part. A single } would look much better. You could use the each operator % to get rid of the closing ).  So the code above becomes

required_props % F(prop_name) {
    prop_name not in rd.anchor throws InvalidArgument("...")

Which is shorter but not very clear (at least for newcomers) because of the % operator. I prefer to use the each operator in small expressions such as myarr % echo and not to use it in bigger expressions such as the snippet above. The closing ) syntax issue is not solved for the general case then so I wanted something like Ruby’s do block syntax.

Importing the “do” block

I’ve made quite a few modifications to fit NGS and to fit some OCD-ness:

The keyword

I prefer “with” because in my opinion, it’s much clearer that it links left-hand-side and right-hand-side. mymethod() with { ... some code ... } looks more clear to me than mymethod() do { ... some code ... }. It might be that I was affected by constructing a mini test framework where test("my feature") with { test code } looks very good.

Not only for code blocks

I don’t like “special” things so whatever comes after with can be any expression.

required_props.each() with F(prop_name) {
    prop_name not in rd.anchor throws InvalidArgument("...")


results = with my_super_mapper_func_name

Note than when using each it could probably be clearer to have the do keyword. I’m still unsure that with is the right choice

Code blocks syntax

You might ask how test("my feature") with { test code } works if what’s after with is any expression.

The test("my feature") with { test code } syntax works because { ... } is a syntax for anonymous function (with 3 optional parameters called A, B and C) and not because it’s part of the with syntax. You could get the same result using test("my feature", { test code }) .

Not just one

I don’t see any reason why with would be limited to one occurrence.

    with {
        while entry = c_readdir(d) {
    with {
        r = c_closedir(d)
        r != 0 throws DirFail('...')

Not a special parameter

As compared to Ruby, the finally() method does not use the special yield keyword to run the given arguments nor parameter should be declared with &. Here is the definition of finally() method (including the bug that cleanup can run twice, which I intend to fix).

F finally(body:Fun, cleanup:Fun) {
    try {
        ret = body()
    } catch(e) {
        throw e

Not special at all, just an additional syntax for arguments

The with syntax allows to write method call arguments outside the parenthesis. That’s all. What follows from this definition is the possibility of using keyword arguments:

        with body = {
                while entry = c_readdir(d) {
        with cleanup = {
                r = c_closedir(d)
                r != 0 throws DirFail('...')

Let’s shorten the syntax

The with name = val argument syntax looked a bit too verbose for me and with was stealing attention focus from the name. I’ve added the fat arrow syntax to combat that.

    body => {
        while entry = c_readdir(d) {
    cleanup => {
        r = c_closedir(d)
        r != 0 throws DirFail('...')

How the feature looks like now?

The snippet above refers to real code in stdlib. Additional examples of the with syntax follow:

Stdlib’s retry()


F body_missing_in_retry() throw InvalidArgument("...")

F retry(times=60, sleep=1, ..., progress_cb={null}, success_cb={A},
    fail_cb=null, body:Fun=body_missing_in_retry) {


F assert_resolvable(h:Str, title="Resolve host", times=45, sleep=2) {
	retry(times=times, sleep=sleep, title=title)
		body       => { `dig "+short" $h`.lines() }
		success_cb => { log_test_ok("...") }
		fail_cb    => { throw TestFail("...") }

test() mini framework for … tests!


F test(name:Str, f:Fun) {


test("Resolving gateway FQDN $gw_fqdn") with {
    assert_resolvable(gw_fqdn, "Resolve gateway FQDN")

Have your own ideas?

Please comment with your ideas regarding this, other and proposed NGS features, they can make it into the language. Or fork/improve/pull request.

Have a nice weekend!

Creating a language is easier now

I do look at other languages when designing and implementing a new shell and a language called NGS. As time goes by, there are more languages to look at and learn from. Hence, I think creating a language in recent years is easier than ever before. I would like to thank authors and contributors of all the languages!cubes-677092_640

NGS is heavily based on ideas that already existed before NGS. Current languages provide many of them and work on NGS includes sorting out which of these ideas resonate with my way of thinking and which don’t.

Of course NGS does have it’s own original ideas (read: I’m not aware of any other language that implemented these ideas) and they are critical to NGS, creating NGS is not just filtering others’ ideas 🙂

Some stolen ideas inspired by other languages

  1. Multi-methods & kind-of-object-orientation in NGS – mostly from CLOS
  2. External commands execution and redirection syntax – bash (while changing behaviour of the $ expansion, now known to be source of many bugs).
  3. Anonymous functions and closures – not sure where I got it from, I’d guess Lisp.
  4. Declarative primitives concept – heavily based on basic concept behind configuration management tools
  5. inspect() – Ruby‘s inspect()
  6. code() – the .perl property from Perl. Not fully implemented in NGS. Should return the code that would evaluate to the given value. Probably comes from Lisp originally which usually prints the values in a way it can read them back.
  7. Ordered hash (When you iterate a Hash, the items are in the same order as they were inserted) – PHP, also implemented in Ruby and if I’m correct in V8.
  8. The in and not in operators – Python.
  9. Many higher order functions such as map and filter probably originated in Scheme or Lisp.
  10. Translate anything that possible into function calls. + operator for example is actually a function call – Python, originally probably from Scheme or Lisp.

Ideas trashed before implementation

While working on NGS I think about features I can add. I have to admit that many of these candidate features seem fine for a few seconds to a few minutes… till I realize that something similar is already implemented in language X and it does not resonate with my way of thinking (also known as “This does not look good”).

Square brackets syntax for creating a list

Suppose we have this common piece of code:

list = []
some_kind_of_loop {

In this case it would advantageous to express the concept of building a list with a special syntax so it would be obvious from a first glance what’s happening. Since array literal already had [1,2,3,...] syntax, the new building-a-list syntax would be [ something here ]. And that’s how we get list comprehensions in Python (also in other languages). The idea is solid but when I see it implemented, it’s very easy for me to realize that I don’t like the syntax. List comprehension syntax is the preferred way to construct lists in Python and seems like it’s being widely used. I’d like to make clear that I do like Python and list comprehensions is one of the very few things I’m not fond of.

So what NGS has for building lists? Let’s see:

# Python
[x*2 for x in range(10) if x > 5]

# NGS straightforward translation
    if x > 5
      collect(x * 2)

Doesn’t look like much of an improvement. That’s because straightforward translations are not representative. The collector‘s use above is not a good example of it’s usage. More about collector later in this article.

Here are the NGS-y ways:

# Python
[x*2 for x in range(10) if x > 5]

# NGS alternatives, pick your compactness vs clarity
(0..10).filter(F(elt) elt>5).map(F(elt) elt*2)
10.filter(F(elt) elt>5).map(F(elt) elt*2)

NGS does not have special syntax for building lists. NGS does have special syntax for building anything and it’s called collector.

Syntax for everything

Just say “no”. It is tempting at first to make syntax for all the concepts in the language or at least for most of them. When taken to it’s extreme, we have Perl and APL.

I don’t like Perl’s syntax, especially sigils, especially that there is more than one sigil, (unlike $ in bash). Note that Perl kind of admitted sigils were used inconsistently (or confusingly? not sure) so sigils syntax was partially changed in Perl 6. On the other hand, Perl 6 added sigil modifiers, called “twigil“s. I counted 9 of them.

APL has many mathematical symbols in it’s syntax. Many of them are not on a keyboard. While the language is very terse and expressive, I don’t like the syntax and the fact that typing in a program in APL is not a straightforward task. [Update: Clarification: I’m not familiar with APL as opposed to other languages mentioned in this article. I was pointed out that I was quick to judge APL. It does have mathematical symbols but it actually doesn’t have much syntax. Most of the symbols are just built-in functions and operators. I’ve also found out one symbol for “go to line” and one for “define function”/”end function”. Actually APL syntax rules are interesting “think different” approach as they are implementation of mathematical notation. Thanks for the link, geocar.]

I do agree with Python’s developers reluctant attitude towards adding new syntax, especially new characters. Perl appears to be on the other end of the spectrum.

I’d like to come back to the collector syntax and show that even when adding a syntax, it’s advantageous to add less of it and reuse existing language concepts if possible.

The initial thought was to base the collector syntax on Lisp’s (loop …) macro , specifically the macro’s collect keyword: (loop ... (collect ...) ...). I was about do add two new keywords: collector and collect. The code would look like this:

mylist = collector
    if x > 5
      collect x * 2

I’ve started considering implementation strategies and came up with the idea that it would be simpler to have one keyword only, collector. collector wraps single expression to the right (which can be a { code block }) in a function with one argument named collect. The code becomes this:

mylist = collector
    if x > 5
      collect(x * 2)

When the collector expression is evaluated, the wrapped code runs with provided collect function. When collect is a function and not a special syntax it allows to use all the facilities that work with functions. Example: somelist.filter(...).each(collect) .

There is more to collector! It has optional initialization argument. Let’s see it in the following example:

    arr.each(F(elt) {
        if predicate(elt)

The code above counts number of elements in arr which satisfy the predicate. That’s roughly how standard library’s count() is implemented.

When collector is not followed by / (slash) and initialization value, the default initialization argument is newly created empty list. The default initial value was chosen to be empty list as it appears that most uses of collector

Advantages of the implementation where collect is a function and not a keyword:

  1. Simpler implementation
  2. Collector behaviour customization – there is a way to provide your own collect function for your types which I will not describe here. Standard library defines such functions for array, hash and integer types.
  3. Functional facilities available for the collect function
  4. collect function can have more than one argument making it possible to collect key-value pairs if constructing a Hash: collector ... collect(k, v) ... (this is from filter() implementation for Hash in standard library).

I consider collector to be a good example of “less syntax is better”.

Easier, not easy

I do enjoy the situation that I have many other languages to look at for ideas, good or bad.

NGS needs original features for it’s domain (systems administration tasks niche). These domain-specific features make NGS stand out. Other languages don’t have them because they are either not domain-specific or have wider domain or are too old to have features for the today’s systems administration domain. Every original feature has a risk attached to it. If you haven’t seen it work before there is practically no way to predict how good or bad a feature will turn out. More about original features in NGS in another post.

While creating a language is now easier than before it’s still not an easy task 🙂 … but you can help. Fork, code, make a pull request.

Have a nice weekend!

Update: discussion on reddit

JSON vs data structure

When you see var j = {"x": 1} in JavaScript, it is plainly wrong to say that j is now JSON or references JSON or holds JSON for that matter.


That’s because j now references a data structure in memory. A data structure is not JSON nor YAML nor any other serialization format.

A data structure can sometimes be serialized to JSON or to YAML or to other formats. A data structure can sometimes be deserialized from these formats.

The code on the right side of the assignment looks like JSON. Don’t let this confuse you. It’s a JavaScript code and it evaluates to a data structure as many other JavaScript expressions do. It could easily be var j = {'a': 1, 'f': function() {} } . You wouldn’t say it’s JSON, right? There is not much difference between the two JavaScript expressions var j = {"x": 1} and var j = {'a': 1, 'f': function() {} } for this matter.

Data structure vs its serialized form

Data structure is the layout of the data. In our case it is in memory. Data structures can also be on disk, think data file of a database. Data structures are “good for” accessing and modifying the data that they hold. In our case it means one can use the expression j.x to access the field x or j.x = 7 to modify it.

Serialized form of a data is a string of characters that can be saved to a file, read from such file or sent over the network. There is no easy way to manipulate such data directly. Modifying serialized data usually involves deserializing it, modifying and serializing back.

Serialization limitations

Not any data structure can be serialized at all. Example (specific to JSON format):

var j = {}
j['circ'] = j
TypeError: Converting circular structure to JSON

Not any data structure can be serialized in a way that would ensure that deserialization would produce similar data structure. Example (specific to JavaScript + JSON):

var j = {'a': 1, 'f': function() {} }

Have a nice week!