NGS is a language and a shell that I am building for systems administration tasks. Enough of the language is implemented to enable writing some useful scripts. The shell is not there yet.

Some of the Hash methods in NGS

The methods for working with Hash I have not seen all at once in other languages are:

1. filterk – filter Hash by key (produces Hash)
2. filterv – filter Hash by value (produces Hash)
3. mapk – map Hash keys (produces Hash)
4. mapv – map Hash values (produces Hash)
5. mapkv – map Hash keys and values (produces Hash as opposed to map which produces an array)
6. without – filters out specific key

How these are actually used? Following is an excerpt from the pollute method (function), which is a part of the AWS module. It uses several of the Hash methods I mentioned, making the method a good example. pollute method (as in “pollute global namespace”) enables using Vpc variable for example instead of AWS::Vpc and so on. I would like to have this behaviour for small quick-and-dirty scripts but not as default so it’s in a method that one can optionally call.

F pollute(do_warn=true) {

    vars =
        _exports.filterk(/^AMI_OWNER/) +
        _exports.filterv(Type).without('Res').without('ResDef') +
        ...

    if do_warn {
        warn("Polluting ...: ${vars.keys().join(', ')}")
    }

    vars.mapk(resolve_global_variable).each(set_global_variable)
}

Let’s go over the code above step by step:

F pollute(do_warn=true) { ... } defines the pollute method with optional parameter do_warn that has default value of true.

_exports is a Hash containing all of the AWS’ module variables and functions, similar to NodeJS module.exports but members are added automatically rather than explicitly. Only the methods and variables that do not start with _ (underscore) are added. One can modify _exports in any way before the end of the module. I will write more in detail about require() and modules in NGS in another post.

filterk(/^AMI_OWNER/) filters all the variables that match the given RegExp

filterv(Type) filters all the variables that are of type Type. These are AWS types’ definitions, such as Vpc, Subnet or Instance.

without('...') filters out the types I don’t like to override.

+ between and after _exports.filterk(...) and _exports.filterv(...) joins the hashes.

mapk translates variables’ names into their index (using resolve_global_variable)

each runs set_global_variable with variable index and the value to set

Hash methods in other languages

I am aware that some of the methods above are present in other languages or libraries. Some examples:

1. Ruby has mapv (transform_values) method.
2. Rails has mapk (transform_keys) and mapv.
3. Perl 6 can modify values in a convenient manner:for %answers.values -> $v is rw { $v += 10 };.

What I have not seen is a language which has all the methods above out of the box. I have a feeling that arrays get all the fame methods while hashes (dictionaries) often get less attention in other languages.

Why NGS has all these methods?

NGS is aiming to be convenient for systems administration tasks. More often than not these tasks include data manipulation. NGS has many functions (methods) for data manipulation, including the ones listed in this post.

Update: reddit discussion

---

Have a nice day!

Wouldn’t it be cool to use Unicode characters as operators and maybe even function names?

if a ≠ b { ... }

range = 0…10

all_above_ten = myarr ∀ F(x) x > 10

Looks good, concise, expressive. Perl 6, Julia, Scala appear to support Unicode operators.

Why don’t I add Unicode operators and function names to NGS then?

If NGS would allow Unicode, it would be optional as I don’t want additional entry barrier or possible problems typing Unicode using remote connection. If I do add optional Unicode to NGS, here is what I think will happen next:

Some people start using Unicode in NGS while others don’t. Mixed code style emerges. It’s easy to imagine such mixed code style even in the same file as someone without the Unicode setup on his/her keyboard  is doing a quick fix. ViralBShah sums it up pretty well.

What do you think? Your comments are welcome.

 

Ruby has a very nice feature – a special syntax to pass a block of code as an argument to a method. I did want something like this NGS. Background, reasons and what this feature became follow.

Background

NGS – Next Generation Shell is a language and a shell I’m working on for a few years now.

This article is about new NGS syntax inspired by and compared to the counterpart  Ruby’s do ... end syntax. The new syntax provides more convenient way of passing anonymous function to another function. Passing functions to other functions is (sometimes) considered to be a “functional programming” feature of a language.

In NGS, up until recently you had to pass all of your arguments (except for the first argument which can be passed before .method() ) inside the parenthesis of the method call syntax:

mymethod(arg1, arg2, ... argN)

or syntactically equivalent

arg1.mymethod(arg2, arg3, ... argN)

Passing an anonymous function could look like the following in NGS:

required_props.each(F(prop_name) {
    prop_name not in rd.anchor throws InvalidArgument("...")
})

The problem

In the code snippet above I don’t like the closing }) part. A single } would look much better. You could use the each operator % to get rid of the closing ).  So the code above becomes

required_props % F(prop_name) {
    prop_name not in rd.anchor throws InvalidArgument("...")
}

Which is shorter but not very clear (at least for newcomers) because of the % operator. I prefer to use the each operator in small expressions such as myarr % echo and not to use it in bigger expressions such as the snippet above. The closing ) syntax issue is not solved for the general case then so I wanted something like Ruby’s do block syntax.

Importing the “do” block

I’ve made quite a few modifications to fit NGS and to fit some OCD-ness:

The keyword

I prefer “with” because in my opinion, it’s much clearer that it links left-hand-side and right-hand-side. mymethod() with { ... some code ... } looks more clear to me than mymethod() do { ... some code ... }. It might be that I was affected by constructing a mini test framework where test("my feature") with { test code } looks very good.

Not only for code blocks

I don’t like “special” things so whatever comes after with can be any expression.

required_props.each() with F(prop_name) {
    prop_name not in rd.anchor throws InvalidArgument("...")
}

or

results = mylist.map() with my_super_mapper_func_name

Note than when using each it could probably be clearer to have the do keyword. I’m still unsure that with is the right choice

Code blocks syntax

You might ask how test("my feature") with { test code } works if what’s after with is any expression.

The test("my feature") with { test code } syntax works because { ... } is a syntax for anonymous function (with 3 optional parameters called A, B and C) and not because it’s part of the with syntax. You could get the same result using test("my feature", { test code }) .

Not just one

I don’t see any reason why with would be limited to one occurrence.

finally()
    with {
        while entry = c_readdir(d) {
            ...
        }
    }
    with {
        r = c_closedir(d)
        r != 0 throws DirFail('...')
    }

Not a special parameter

As compared to Ruby, the finally() method does not use the special yield keyword to run the given arguments nor parameter should be declared with &. Here is the definition of finally() method (including the bug that cleanup can run twice, which I intend to fix).

F finally(body:Fun, cleanup:Fun) {
    try {
        ret = body()
        cleanup()
        ret
    } catch(e) {
        cleanup()
        throw e
    }
}

Not special at all, just an additional syntax for arguments

The with syntax allows to write method call arguments outside the parenthesis. That’s all. What follows from this definition is the possibility of using keyword arguments:

finally()
        with body = {
                while entry = c_readdir(d) {
                        ...
                }
        }
        with cleanup = {
                r = c_closedir(d)
                r != 0 throws DirFail('...')
        }

Let’s shorten the syntax

The with name = val argument syntax looked a bit too verbose for me and with was stealing attention focus from the name. I’ve added the fat arrow syntax to combat that.

finally()
    body => {
        while entry = c_readdir(d) {
            ...
        }
    }
    cleanup => {
        r = c_closedir(d)
        r != 0 throws DirFail('...')
    }

How the feature looks like now?

The snippet above refers to real code in stdlib. Additional examples of the with syntax follow:

Stdlib’s retry()

Definition:

F body_missing_in_retry() throw InvalidArgument("...")

F retry(times=60, sleep=1, ..., progress_cb={null}, success_cb={A},
    fail_cb=null, body:Fun=body_missing_in_retry) {
...
}

Usage:

F assert_resolvable(h:Str, title="Resolve host", times=45, sleep=2) {
	retry(times=times, sleep=sleep, title=title)
		body       => { `dig "+short" $h`.lines() }
		success_cb => { log_test_ok("...") }
		fail_cb    => { throw TestFail("...") }
}

test() mini framework for … tests!

Definition:

F test(name:Str, f:Fun) {
...
}

Usage:

test("Resolving gateway FQDN $gw_fqdn") with {
    assert_resolvable(gw_fqdn, "Resolve gateway FQDN")
}

Have your own ideas?

Please comment with your ideas regarding this, other and proposed NGS features, they can make it into the language. Or fork/improve/pull request.

---

Have a nice weekend!

I do look at other languages when designing and implementing a new shell and a language called NGS. As time goes by, there are more languages to look at and learn from. Hence, I think creating a language in recent years is easier than ever before. I would like to thank authors and contributors of all the languages!

NGS is heavily based on ideas that already existed before NGS. Current languages provide many of them and work on NGS includes sorting out which of these ideas resonate with my way of thinking and which don’t.

Of course NGS does have it’s own original ideas (read: I’m not aware of any other language that implemented these ideas) and they are critical to NGS, creating NGS is not just filtering others’ ideas 🙂

Some stolen ideas inspired by other languages

1. Multi-methods & kind-of-object-orientation in NGS – mostly from CLOS
2. External commands execution and redirection syntax – bash (while changing behaviour of the $ expansion, now known to be source of many bugs).
3. Anonymous functions and closures – not sure where I got it from, I’d guess Lisp.
4. Declarative primitives concept – heavily based on basic concept behind configuration management tools
5. inspect() – Ruby‘s inspect()
6. code() – the .perl property from Perl. Not fully implemented in NGS. Should return the code that would evaluate to the given value. Probably comes from Lisp originally which usually prints the values in a way it can read them back.
7. Ordered hash (When you iterate a Hash, the items are in the same order as they were inserted) – PHP, also implemented in Ruby and if I’m correct in V8.
8. The in and not in operators – Python.
9. Many higher order functions such as map and filter probably originated in Scheme or Lisp.
10. Translate anything that possible into function calls. + operator for example is actually a function call – Python, originally probably from Scheme or Lisp.

Ideas trashed before implementation

While working on NGS I think about features I can add. I have to admit that many of these candidate features seem fine for a few seconds to a few minutes… till I realize that something similar is already implemented in language X and it does not resonate with my way of thinking (also known as “This does not look good”).

Square brackets syntax for creating a list

Suppose we have this common piece of code:

list = []
some_kind_of_loop {
    ...
    list.push(my_new_item)
    ...
}

In this case it would advantageous to express the concept of building a list with a special syntax so it would be obvious from a first glance what’s happening. Since array literal already had [1,2,3,...] syntax, the new building-a-list syntax would be [ something here ]. And that’s how we get list comprehensions in Python (also in other languages). The idea is solid but when I see it implemented, it’s very easy for me to realize that I don’t like the syntax. List comprehension syntax is the preferred way to construct lists in Python and seems like it’s being widely used. I’d like to make clear that I do like Python and list comprehensions is one of the very few things I’m not fond of.

So what NGS has for building lists? Let’s see:

# Python
[x*2 for x in range(10) if x > 5]

# NGS straightforward translation
collector
  for(x;10)
    if x > 5
      collect(x * 2)

Doesn’t look like much of an improvement. That’s because straightforward translations are not representative. The collector‘s use above is not a good example of it’s usage. More about collector later in this article.

Here are the NGS-y ways:

# Python
[x*2 for x in range(10) if x > 5]

# NGS alternatives, pick your compactness vs clarity
(0..10).filter(F(elt) elt>5).map(F(elt) elt*2)
10.filter(F(elt) elt>5).map(F(elt) elt*2)
10.filter(X>5).map(X*2)
10?(X>5)/(X*2)

NGS does not have special syntax for building lists. NGS does have special syntax for building anything and it’s called collector.

Syntax for everything

Just say “no”. It is tempting at first to make syntax for all the concepts in the language or at least for most of them. When taken to it’s extreme, we have Perl and APL.

I don’t like Perl’s syntax, especially sigils, especially that there is more than one sigil, (unlike $ in bash). Note that Perl kind of admitted sigils were used inconsistently (or confusingly? not sure) so sigils syntax was partially changed in Perl 6. On the other hand, Perl 6 added sigil modifiers, called “twigil“s. I counted 9 of them.

APL has many mathematical symbols in it’s syntax. Many of them are not on a keyboard. While the language is very terse and expressive, I don’t like the syntax and the fact that typing in a program in APL is not a straightforward task. [Update: Clarification: I’m not familiar with APL as opposed to other languages mentioned in this article. I was pointed out that I was quick to judge APL. It does have mathematical symbols but it actually doesn’t have much syntax. Most of the symbols are just built-in functions and operators. I’ve also found out one symbol for “go to line” and one for “define function”/”end function”. Actually APL syntax rules are interesting “think different” approach as they are implementation of mathematical notation. Thanks for the link, geocar.]

I do agree with Python’s developers reluctant attitude towards adding new syntax, especially new characters. Perl appears to be on the other end of the spectrum.

I’d like to come back to the collector syntax and show that even when adding a syntax, it’s advantageous to add less of it and reuse existing language concepts if possible.

The initial thought was to base the collector syntax on Lisp’s (loop …) macro , specifically the macro’s collect keyword: (loop ... (collect ...) ...). I was about do add two new keywords: collector and collect. The code would look like this:

mylist = collector
  for(x;10)
    if x > 5
      collect x * 2

I’ve started considering implementation strategies and came up with the idea that it would be simpler to have one keyword only, collector. collector wraps single expression to the right (which can be a { code block }) in a function with one argument named collect. The code becomes this:

mylist = collector
  for(x;10)
    if x > 5
      collect(x * 2)

When the collector expression is evaluated, the wrapped code runs with provided collect function. When collect is a function and not a special syntax it allows to use all the facilities that work with functions. Example: somelist.filter(...).each(collect) .

There is more to collector! It has optional initialization argument. Let’s see it in the following example:

collector/0
    arr.each(F(elt) {
        if predicate(elt)
            collect(1)
    })

The code above counts number of elements in arr which satisfy the predicate. That’s roughly how standard library’s count() is implemented.

When collector is not followed by / (slash) and initialization value, the default initialization argument is newly created empty list. The default initial value was chosen to be empty list as it appears that most uses of collector

Advantages of the implementation where collect is a function and not a keyword:

1. Simpler implementation
2. Collector behaviour customization – there is a way to provide your own collect function for your types which I will not describe here. Standard library defines such functions for array, hash and integer types.
3. Functional facilities available for the collect function
4. collect function can have more than one argument making it possible to collect key-value pairs if constructing a Hash: collector ... collect(k, v) ... (this is from filter() implementation for Hash in standard library).

I consider collector to be a good example of “less syntax is better”.

Easier, not easy

I do enjoy the situation that I have many other languages to look at for ideas, good or bad.

NGS needs original features for it’s domain (systems administration tasks niche). These domain-specific features make NGS stand out. Other languages don’t have them because they are either not domain-specific or have wider domain or are too old to have features for the today’s systems administration domain. Every original feature has a risk attached to it. If you haven’t seen it work before there is practically no way to predict how good or bad a feature will turn out. More about original features in NGS in another post.

While creating a language is now easier than before it’s still not an easy task 🙂 … but you can help. Fork, code, make a pull request.

---

Have a nice weekend!

Update: discussion on reddit