Creating a language is easier now

I do look at other languages when designing and implementing a new shell and a language called NGS. As time goes by, there are more languages to look at and learn from. Hence, I think creating a language in recent years is easier than ever before. I would like to thank authors and contributors of all the languages!cubes-677092_640

NGS is heavily based on ideas that already existed before NGS. Current languages provide many of them and work on NGS includes sorting out which of these ideas resonate with my way of thinking and which don’t.

Of course NGS does have it’s own original ideas (read: I’m not aware of any other language that implemented these ideas) and they are critical to NGS, creating NGS is not just filtering others’ ideasūüôā

Some stolen ideas inspired by other languages

  1. Multi-methods & kind-of-object-orientation in NGS – mostly from CLOS
  2. External commands execution and redirection syntax – bash (while changing behaviour of the $ expansion, now known to be source of many bugs).
  3. Anonymous functions and closures – not sure where I got it from, I’d guess Lisp.
  4. Declarative primitives concept – heavily based on basic concept behind configuration management tools
  5. inspect() – Ruby‘s inspect()
  6. code() – the .perl property from Perl. Not fully implemented in NGS. Should return the code that would evaluate to the given value. Probably comes from Lisp originally which usually prints the values in a way it can read them back.
  7. Ordered hash (When you iterate a Hash, the items are in the same order as they were inserted) – PHP, also implemented in Ruby and if I’m correct in V8.
  8. The in and not in operators – Python.
  9. Many higher order functions such as map and filter probably originated in Scheme or Lisp.
  10. Translate anything that possible into function calls. + operator for example is actually a function call – Python, originally probably from Scheme or Lisp.

Ideas trashed before implementation

While working on NGS I think about features I can add. I have to admit that many of these candidate features seem fine for a few seconds to a few minutes… till I realize that something similar is already implemented in language X and it does not resonate with my way of thinking (also known as “This does not look good”).

Square brackets syntax for creating a list

Suppose we have this common piece of code:

list = []
some_kind_of_loop {

In this case it would advantageous to express the concept of building a list with a special syntax so it would be obvious from a first glance what’s happening. Since array literal already had [1,2,3,...] syntax, the new building-a-list syntax would be [ something here ]. And that’s how we get list comprehensions in Python (also in other languages). The idea is solid but when I see it implemented, it’s very easy for me to realize that I don’t like the syntax. List comprehension syntax is the preferred way to construct lists in Python and seems like it’s being widely used. I’d like to make clear that I do like Python and list comprehensions is one of the very few things I’m not fond of.

So what NGS has for building lists? Let’s see:

# Python
[x*2 for x in range(10) if x > 5]

# NGS straightforward translation
    if x > 5
      collect(x * 2)

Doesn’t look like much of an improvement. That’s because straightforward translations are not representative. The collector‘s use above is not a good example of it’s usage. More about collector later in this article.

Here are the NGS-y ways:

# Python
[x*2 for x in range(10) if x > 5]

# NGS alternatives, pick your compactness vs clarity
(0..10).filter(F(elt) elt>5).map(F(elt) elt*2)
10.filter(F(elt) elt>5).map(F(elt) elt*2)

NGS does not have special syntax for building lists. NGS does have special syntax for building anything and it’s called collector.

Syntax for everything

Just say “no”. It is tempting at first to make syntax for all the concepts in the language or at least for most of them. When taken to it’s extreme, we have Perl and APL.

I don’t like Perl’s syntax, especially sigils, especially that there is more than one sigil, (unlike $ in bash). Note that Perl kind of admitted sigils were used inconsistently (or confusingly? not sure) so sigils syntax was partially changed in Perl 6. On the other hand, Perl 6 added sigil modifiers, called “twigil“s. I counted 9 of them.

APL has many mathematical symbols in it’s syntax. Many of them are not on a keyboard. While the language is very terse and expressive, I don’t like the syntax and the fact that typing in a program in APL is not a straightforward task.

I do agree with Python’s developers reluctant attitude towards adding new syntax, especially new characters. Perl appears to be on the other end of the spectrum.

I’d like to come back to the collector syntax and show that even when adding a syntax, it’s advantageous to add less of it and reuse existing language concepts if possible.

The initial thought was to base the collector syntax on Lisp’s (loop …)¬†macro , specifically the macro’s collect keyword: (loop ... (collect ...) ...). I was about do add two new keywords: collector and collect. The code would look like this:

mylist = collector
    if x > 5
      collect x * 2

I’ve started considering implementation strategies and came up with the idea that it would be simpler to have one keyword only, collector. collector wraps single expression to the right (which can be a { code block }) in a function with one argument named collect. The code becomes this:

mylist = collector
    if x > 5
      collect(x * 2)

When the collector expression is evaluated, the wrapped code runs with provided collect function. When collect is a function and not a special syntax it allows to use all the facilities that work with functions. Example: somelist.filter(...).each(collect) .

There is more to collector! It has optional initialization argument. Let’s see it in the following example:

    arr.each(F(elt) {
        if predicate(elt)

The code above counts number of elements in arr which satisfy the predicate. That’s roughly how standard library’s count() is implemented.

When collector is not followed by / (slash) and initialization value, the default initialization argument is newly created empty list. The default initial value was chosen to be empty list as it appears that most uses of collector

Advantages of the implementation where collect is a function and not a keyword:

  1. Simpler implementation
  2. Collector behaviour customization – there is a way to provide your own collect function for your types which I will not describe here. Standard library defines such functions for array, hash and integer types.
  3. Functional facilities available for the collect function
  4. collect function can have more than one argument making it possible to collect key-value pairs if constructing a Hash: collector ... collect(k, v) ... (this is from filter() implementation for Hash in standard library).

I consider collector to be a good example of “less syntax is better”.

Easier, not easy

I do enjoy the situation that I have many other languages to look at for ideas, good or bad.

NGS needs original features for it’s domain (systems administration tasks niche). These domain-specific features make NGS stand out. Other languages don’t have them because they are either not domain-specific or have wider domain or are too old to have features for the today’s systems administration domain. Every original feature has a risk attached to it. If you haven’t seen it work before there is practically no way to predict how good or bad a feature will turn out. More about original features in NGS in another post.

While creating a language is now easier than before it’s still not an easy taskūüôā … but you can help. Fork, code, make a pull request.

Have a nice weekend!

Update: discussion on reddit

JSON vs data structure

When you see var j = {"x": 1} in JavaScript, it is plainly wrong to say that j is now JSON or references JSON or holds JSON for that matter.


That’s because j now references a data structure in memory. A data structure is not JSON nor YAML nor any other serialization format.

A data structure can sometimes be serialized to JSON or to YAML or to other formats. A data structure can sometimes be deserialized from these formats.

The code on the right side of the assignment looks like JSON. Don’t let this confuse you. It’s a JavaScript code and it evaluates to a data structure as many other JavaScript expressions do. It could easily be var j = {'a': 1, 'f': function() {} } . You wouldn’t say it’s JSON, right? There is not much difference between the two JavaScript expressions¬†var j = {"x": 1} and var j = {'a': 1, 'f': function() {} } for this matter.

Data structure vs it’s serialized form

Data structure is the layout of the data. In our case it is in memory. Data structures can also be on disk, think data file of a database. Data structures are “good for” accessing and modifying the data that they hold. In our case it means one can use the expression j.x to access the field x or j.x = 7 to modify it.

Serialized form of a data is a string of characters that can be saved to a file, read from such file or sent over the network. There is no easy way to manipulate such data directly. Modifying serialized data usually involves deserializing it, modifying and serializing back.

Serialization limitations

Not any data structure can be serialized at all. Example (specific to JSON format):

var j = {}
j['circ'] = j
TypeError: Converting circular structure to JSON

Not any data structure can be serialized in a way that would ensure that desirialization would produce similar data structure. Example (specific to JavaScript + JSON):

var j = {'a': 1, 'f': function() {} }

Have a nice week!

dnsimple SSL defaults to simple, not secure


When purchasing an SSL certificate at dnsimple I was amazed to discover that by default it’s dnsimple who generates the private key for you. The checkbox that says “I want to provide a custom CSR” is unchecked by default. Not checking it causes dnsimple to generate your private key and a CSR for you. Simple – yes. Secure – no.

Generating a private key for anyone but yourself is a big security no-no. This is common knowledge for people that are selling certificates. Still somehow it’s default at dnsimple.

Please stop doing these stupid things!

Declarative primitives or mkdir -p for the cloud

After some positive feedback regarding the concept of declarative primitives I would like to elaborate about it.

Defining declarative primitives

Declarative primitives is just a description of existing techniques. I gave it a name because I’m not aware of any other term describing these techniques. The idea behind declarative approach is to describe the desired state or result and not particular command or operations to achieve it.

Example: mkdir -p dir1/dir2/dir3

The outcome of the command does not depend on current state (whether the directory exists or not). You describe the desired state: directories dir1, dir2 and dir3 should exist after the command is run. Note that mkdir dir1/dir2/dir3 does not have the same effect: it fails if dir1 does not exist or dir2 does not exist or if dir3 exists.

The phrase declarative primitives emphasizes granularity. Existing declarative tools for the cloud operate on many described resources, build dependency graphs and run in order that they decide. Declarative primitives provide a very flexible way to control a single resource or a group of few resources of the same type. The flexibility comes from granularity. You decide how you combine the resources. You can easily integrate existing resources. You can modify just the properties of your interest on the resources you choose. This approach is ideal of scripting in my opinion.

Where are declarative primitives for the cloud?


I believe that when writing a script, using mkdir -p should be similar to using AwsElb(...).converge() for example. I’m working on implementing it (as a library for the Next Generation Shell) and I’m not aware of any other project that does it.

There are many projects for managing the cloud, how are they different?

Here are the solutions that I’m aware of and how familiar I am with each one:

  1. CloudFormation Рusing frequently (I prefer YAML syntax for it)
  2. Terraform – I’ve read the documentation and bits of source code
  3. Cloudify Рfamiliar with the product, made modules for it
  4. Puppet Рwas using it intensively on few different projects
  5. Chef Рwas using it intensively in many projects
  6. Ansible – unfamiliar with this one (only took a look at documentation) so not reviewing it below
  • All take the declarative approach. You describe many resources or the entire system and feed the description to the tool which in turn does all the work. None of these solutions was designed to provide you with the primitives that could be easily used in your scripts. These tools just don’t match my view regarding scripting.
  • These tools can do a rollback on error for example. They can do that precisely because they have the description of the entire system or big parts of it. It will take some additional work to implement rolling back using declarative primitives. The question is whether you need the rollback functionality …
  • Some of these tools can be made to work with different clouds relatively easily. Working with different clouds easily may also possible with declarative primitives but the library I’m currently working on does not have such goal.
  • Except for Chef, the tools in the list above use formats or DSLs not based on real programming languages. This means that except for trivial cases you will be using some additional tool to generate the descriptions of desired states. Limited DSLs do not work. See Puppet and Ansible that started with simple description languages and now they are almost real programming languages … which where never designed as programming languages, which has consequences.
  • I’m not aware of any option in the tools above that lets you view definitions of existing resources, which prevents you from starting managing existing resources with these tools and from cloning existing resources. I have started implementing the functionality that lets you generate the script that would build an existing resource: SomeResource(...).code() . This will allow easy modification or cloning.
  • A feature missing both from these tools and from my library is generating a code to start with for a given resource type (say security group or load balancer). Writing CloudFormation definition for a type with many properties is a nightmare. Nobody should start from scratch. Apache or Nginx configuration files are good example of starting points. Similar should be done for the cloud resources.
  • Note that Chef and Puppet were originally designed to manage servers. I don’t have any experience using them for managing the cloud but I can guess it would be less optimal than dedicated tools (the first three tools).

Scripting the cloud – time to do it right!

Why CloudFormation is better than Chef and Puppet

Strange comparison, I know.


Scripting vs declarative approaches

The aspect I’m looking at is scripting (aka imperative programming) vs declarative approach. In many situations I choose the scripting approach over declarative because the downsides of declarative approach outweigh the benefits in the situations that I have.

Declarative approach downsides

Downsides of Chef, Puppet and other declarative systems? Main downsides are complexity and more external dependencies. These lead to:

  1. Fragility
  2. More maintenance
  3. More setup for anything except for the trivial cases

I can’t stress enough the price of complexity.

Declarative approach advantages

When the imperative approach would mean too much work the declarative approach has the advantage. Think of SQL statements. It would be enormous amounts of work to code them by hand each time. Let’s summarize:

  1. Concise and meaningful code
  2. Much work done by small amount of code

Value of tools

I value the tools by TCO.

Example 1: making sure a file has specific content. It could be as simple as echo my_content > my_file in a script or it could be as complex as installing Chef/Puppet/Your-cool-tool-du-jour server and so on…

Example 2: making sure that specific load balancer is set up (AWS ELB). It could be writing a script that uses AWS CLI or using declarative tools such as CloudFormation or Terraform (haven’t used Terraform myself yet). Writing a script to idempotently configure security groups and the load balancer and it’s properties is much more work than echo ... from the previous example.

While the TCO greatly depends on your specific situation, I argue that the tools that reduce larger amounts of work, such as in example 2, are more likely to have better TCO in general than tools from example 1.

“… but Chef can manage AWS too, you know?”

Yes, I know… and I don’t like this solution. I would like to manage AWS from my laptop or from dedicated management machine, not where Chef client runs. Also, (oh no!) I don’t currently use Chef and bringing it just for managing AWS does not seem like a good idea.

Same for managing AWS with Puppet.


Declarative tools will always bring complexity and it’s a huge minus. The more complex the tool the more work it requires to operate. Make sure the amount of work saved is greater than the amount of work your declarative tool requires to operate.

Opinion: we can do better

I like the scripting solutions for their relative simplicity (when scripts are written professionally). I suggest combined approach. Let’s call it “declarative primitives”.

Imagine a scripting library that provides primitives AwsElb, AwsInstance, AwsSecGroup and such. Using this primitives does not force you to give up the flow control. No dependency graphs. You are still writing a script. Minimal complexity increase over regular scripting.

Such library is under development. Additional advantage of this library is that the whole state will be kept in the tags of the resources. Other solutions have additional state files and I don’t like that.

Sample (NGS language) censored code that uses the library follows:

my_vpc_ancor = {'aws:cloudformation:stack-name': 'my-vpc'}

elb = AwsElb(
        'tags': %{
            env ${ENV.ENV}
            role myservice-elb
        'listeners': [
                Protocol TCP
                LoadBalancerPort 443
                InstanceProtocol TCP
                InstancePort 443
        'subnets': AwsSubnet(my_vpc_ancor).expect(2)
        'health-check': %{
            UnhealthyThreshold 5
            Timeout 5
            HealthyThreshold 3
            Interval 10
            Target 'SSL:443'
        'instances': AwsInstance({'env': ENV.ENV, 'role': 'myservice'}).expect()


It creates a load balancer in an already existing VPC (which was created by CloudFormation) and connects existing instances to it. The example is not full as the library is work in progress but it does work.

Have fun and watch your TCO!

Bash – opinionated review


  1. Huge immediately available “library” of external commands, providing lots of out-of-the-box functionality. You don’t even need to “import” or “require” to get it.
  2. Easy manipulation of files and processes with good syntax for these tasks
  3. Pipes make combining programs very easy
  4. Always installed (OS X comes with version 3 for some reason, version 4 is easily installable).

The pros above make bash the best language for many system tasks (and not Ruby, Perl or Python for example).


  1. Horrible syntax for general purpose programming tasks (read anything that is not a process or file manipulation), probably consequence of bash not being designed as a programming language. Language features were added with time in a backwards compatible way. The syntax looks really bad.
  2. No nested data structures
  3. No exceptions
  4. No named function parameters
  5. Subshells are forks, preventing global variables changes to be visible outside of the subshell. Combined with very “interesting” rules of what is a subshell and what is not, this behaviour can be surprising.
  6. Default behaviour is not to exit if one of the commands returns an error (use set -e to change)
  7. Default behaviour is to treat unset variables as empty strings (use set -u to change)
  8. Behaviour switches with setAction at a distance anti-pattern
  9. Default prompt does not include critical information: exit code of the last command
  10. No semantic understanding of the output and hence no command line completion based on output of previous commands. What makes me angry the most, completing apt-get install HERE will use the whole available packages list, not the output of the apt-cache search ... you just used to find your package while most of the time completing based on the output would be the right thing.
  11. History includes the commands and sometimes timestamps but no relevant context such as working directory, relevant variables’ values, exit code, etc…
  12. Inconvenient programming

What’s being done about the numerous cons?

I don’t see much being done about these in bash. Backwards compatibility and the fear of potentially breaking huge amount of existing code is probably the reason.

There are alternative shells being/were developed. Unfortunately for compatibility and other reasons they don’t seem to address all of the cons. Some of this projects ruin the simple syntax for process and file manipulation by basing themselves on an existing general purpose programming language.


NGS, a new and completely different shell, does aim to fix all the cons while keeping the positive aspects of shell programming.

The bigger part of the language is already implemented. See how it looks like.

You are welcome to join the project.

Recommended tools – tmux

Project URL:

Tmux is a newer take on screen.

tmux is a terminal multiplexer: it enables a number of terminals to be created, accessed, and controlled from a single screen. tmux may be detached from a screen and continue running in the background, then later reattached.

My usage pattern


  1. Locally in one or more of terminal tabs. This keeps number of terminal tabs sane. I mostly use splits and occasionally tabs inside tmux.
  2. Remotely for avoiding multiple ssh connections, critical tasks, poor connection.

I’d recommend these navigation keys:

bind-key -n C-Left select-pane -L
bind-key -n C-Right select-pane -R
bind-key -n C-Up select-pane -U
bind-key -n C-Down select-pane -D

bind-key -n C-S-Left switch-client -p
bind-key -n C-S-Right switch-client -n
bind-key -n S-Left prev
bind-key -n S-Right next
bind-key -n M-j prev
bind-key -n M-k next
bind-key -n C-M-z resize-pane -Z

For a remote machine I suggest setting pane navigation keys to M-Left/Right/Up/Down. This way instead of hitting Ctrl+b C-Left or worse Ctrl+b Ctrl+b Left you just hit Alt+Left (aka Meta+Left).


  1. Zoom mode.
  2. Saner default prefix key Ctrl+b as opposed to screen’s Ctrl+a. Outside screen Ctrl+a usually jumps to beginning of a line and I use this shortcut frequently.
  3. Multiple copy buffers. See the prefix = shortcut in the manual.


  1. Mouse selection and scrolling behaves different than in plain terminal. The regular mouse selection can be done with Shift but still…
  2. I have not automated setting up remote machines’ tmux.conf automatically yet. That would be a huge win.