Why NGS has no “undefined”

Since I know JavaScript, some Ruby and a bit of Perl which all have the concept of undefined it was a decision I had to make whether I implement undefined in NGS. This article shows why I decided not to have the undefined value/data type.

Update (thanks /u/EldritchSundae): what you observe in Ruby example below is nil, not undefined. In bash the undefined value is empty string. Ruby does not have undefined but it has the ability to read non-existing hash keys without causing an exception like JavaScript and Perl. In Ruby’s case the result is nil, not undef (Perl) or undefined (JavaScript).

Undefined in other languages

Showing here few common cases, not all possible usages.

JavaScript

> nodejs -e 'const a; console.log(a)'
undefined

> nodejs -e 'const h={}; console.log(h["xyz"])'
undefined

> nodejs -e '(function f(a,b) { console.log(a,b) })(1)'
1 undefined

Ruby

> ruby -e 'h={}; puts h["xyz"]' # outputs empty line

Perl

> perl -e '%h=(); print $h{"xyz"}' # outputs nothing

bash

> bash -c 'echo $a' # outputs empty line

Absence of undefined in NGS

Adding yet another data type to NGS needs justification. I can’t find any justification for undefined. I do consider the usages above bugs. Accessing a variable or a place that were not assigned any value is an error.

Conveying absence of a value in NGS is done similar to other languages with the special null value. There are also somewhat experimental Box, FullBox and EmptyBox types, similar to Option, Some and None in Scala.

Undefined as a hash value for non-existing keys

Having undefined returned when looking up non-existing hash key is a trade-off. It’s more convenient and more error-prone. I have chosen Python-like approach: it’s an error.

> ngs -e 'h={}; h["xyz"]'
... Exception of type KeyNotFound ...

# and added convenience method "get"
> ngs -p 'h={}; h.get("xyz", "NONE")'
NONE

Undefined when accessing unset variable

While bash gives you an empty string by default and Perl gives you undef, I do think accessing unset variable is definitely a bug. I guess it was understood at some point by the creators of bash and Perl so bash has -u flag that makes accessing undefined variable an error and Perl has use strict mode which does the same among other things.

> bash -c 'echo $a' # no error
> bash -c 'set -u; echo $a'
bash: a: unbound variable

> bash -c 'a=(); echo ${a[0]}' # no error, just horrible syntax :)
> bash -c 'set -u; a=(); echo ${a[0]}'
bash: a[0]: unbound variable

> perl -e 'print $a' # no error
> perl -e 'use strict; print $a;'
# no error - I have no idea why, probably some "special" variable

# Perl - take number two:
> perl -e 'print $abc' # no error
> perl -e 'use strict; print $abc;'
Global symbol "$abc" requires explicit package name
(did you forget to declare "my $abc"?) at -e line 1.
Execution of -e aborted due to compilation errors.

Undefined as value for parameters without arguments

Calling an NGS function with less arguments than it expects is an error as in most languages:

> ngs -e '(F (a,b) 10)(1)'
... Exception of type ArgsMismatch ..

By the way, I do cringe every time I see JavaScript code that explicitly uses undefined:

function f(optional_a, optional_b) { }
f(undefined, 10)

The programmer took a decision not to pass a value. How in the world is this undefined? Use null for f*ck sake!


Have a nice day!

Unicode characters as operators in a programming language¿

Wouldn’t it be cool to use Unicode characters as operators and maybe even function names?

if a ≠ b { ... }

range = 0…10

all_above_ten = myarr ∀ F(x) x > 10

Looks good, concise, expressive. Perl 6, Julia, Scala appear to support Unicode operators.

Why don’t I add Unicode operators and function names to NGS then?

If NGS would allow Unicode, it would be optional as I don’t want additional entry barrier or possible problems typing Unicode using remote connection. If I do add optional Unicode to NGS, here is what I think will happen next:

Some people start using Unicode in NGS while others don’t. Mixed code style emerges. It’s easy to imagine such mixed code style even in the same file as someone without the Unicode setup on his/her keyboard  is doing a quick fix. ViralBShah sums it up pretty well.

What do you think? Your comments are welcome.

 

Ruby’s do block in NGS experiment

Ruby has a very nice feature – a special syntax to pass a block of code as an argument to a method. I did want something like this NGS. Background, reasons and what this feature became follow.

file-534178_640

Background

NGS – Next Generation Shell is a language and a shell I’m working on for a few years now.

This article is about new NGS syntax inspired by and compared to the counterpart  Ruby’s do ... end syntax. The new syntax provides more convenient way of passing anonymous function to another function. Passing functions to other functions is (sometimes) considered to be a “functional programming” feature of a language.

In NGS, up until recently you had to pass all of your arguments (except for the first argument which can be passed before .method() ) inside the parenthesis of the method call syntax:

mymethod(arg1, arg2, ... argN)

or syntactically equivalent

arg1.mymethod(arg2, arg3, ... argN)

Passing an anonymous function could look like the following in NGS:

required_props.each(F(prop_name) {
    prop_name not in rd.anchor throws InvalidArgument("...")
})

The problem

In the code snippet above I don’t like the closing }) part. A single } would look much better. You could use the each operator % to get rid of the closing ).  So the code above becomes

required_props % F(prop_name) {
    prop_name not in rd.anchor throws InvalidArgument("...")
}

Which is shorter but not very clear (at least for newcomers) because of the % operator. I prefer to use the each operator in small expressions such as myarr % echo and not to use it in bigger expressions such as the snippet above. The closing ) syntax issue is not solved for the general case then so I wanted something like Ruby’s do block syntax.

Importing the “do” block

I’ve made quite a few modifications to fit NGS and to fit some OCD-ness:

The keyword

I prefer “with” because in my opinion, it’s much clearer that it links left-hand-side and right-hand-side. mymethod() with { ... some code ... } looks more clear to me than mymethod() do { ... some code ... }. It might be that I was affected by constructing a mini test framework where test("my feature") with { test code } looks very good.

Not only for code blocks

I don’t like “special” things so whatever comes after with can be any expression.

required_props.each() with F(prop_name) {
    prop_name not in rd.anchor throws InvalidArgument("...")
}

or

results = mylist.map() with my_super_mapper_func_name

Note than when using each it could probably be clearer to have the do keyword. I’m still unsure that with is the right choice

Code blocks syntax

You might ask how test("my feature") with { test code } works if what’s after with is any expression.

The test("my feature") with { test code } syntax works because { ... } is a syntax for anonymous function (with 3 optional parameters called A, B and C) and not because it’s part of the with syntax. You could get the same result using test("my feature", { test code }) .

Not just one

I don’t see any reason why with would be limited to one occurrence.

finally()
    with {
        while entry = c_readdir(d) {
            ...
        }
    }
    with {
        r = c_closedir(d)
        r != 0 throws DirFail('...')
    }

Not a special parameter

As compared to Ruby, the finally() method does not use the special yield keyword to run the given arguments nor parameter should be declared with &. Here is the definition of finally() method (including the bug that cleanup can run twice, which I intend to fix).

F finally(body:Fun, cleanup:Fun) {
    try {
        ret = body()
        cleanup()
        ret
    } catch(e) {
        cleanup()
        throw e
    }
}

Not special at all, just an additional syntax for arguments

The with syntax allows to write method call arguments outside the parenthesis. That’s all. What follows from this definition is the possibility of using keyword arguments:

finally()
        with body = {
                while entry = c_readdir(d) {
                        ...
                }
        }
        with cleanup = {
                r = c_closedir(d)
                r != 0 throws DirFail('...')
        }

Let’s shorten the syntax

The with name = val argument syntax looked a bit too verbose for me and with was stealing attention focus from the name. I’ve added the fat arrow syntax to combat that.

finally()
    body => {
        while entry = c_readdir(d) {
            ...
        }
    }
    cleanup => {
        r = c_closedir(d)
        r != 0 throws DirFail('...')
    }

How the feature looks like now?

The snippet above refers to real code in stdlib. Additional examples of the with syntax follow:

Stdlib’s retry()

Definition:

F body_missing_in_retry() throw InvalidArgument("...")

F retry(times=60, sleep=1, ..., progress_cb={null}, success_cb={A},
    fail_cb=null, body:Fun=body_missing_in_retry) {
...
}

Usage:

F assert_resolvable(h:Str, title="Resolve host", times=45, sleep=2) {
	retry(times=times, sleep=sleep, title=title)
		body       => { `dig "+short" $h`.lines() }
		success_cb => { log_test_ok("...") }
		fail_cb    => { throw TestFail("...") }
}

test() mini framework for … tests!

Definition:

F test(name:Str, f:Fun) {
...
}

Usage:

test("Resolving gateway FQDN $gw_fqdn") with {
    assert_resolvable(gw_fqdn, "Resolve gateway FQDN")
}

Have your own ideas?

Please comment with your ideas regarding this, other and proposed NGS features, they can make it into the language. Or fork/improve/pull request.


Have a nice weekend!

Creating a language is easier now

I do look at other languages when designing and implementing a new shell and a language called NGS. As time goes by, there are more languages to look at and learn from. Hence, I think creating a language in recent years is easier than ever before. I would like to thank authors and contributors of all the languages!cubes-677092_640

NGS is heavily based on ideas that already existed before NGS. Current languages provide many of them and work on NGS includes sorting out which of these ideas resonate with my way of thinking and which don’t.

Of course NGS does have it’s own original ideas (read: I’m not aware of any other language that implemented these ideas) and they are critical to NGS, creating NGS is not just filtering others’ ideas 🙂

Some stolen ideas inspired by other languages

  1. Multi-methods & kind-of-object-orientation in NGS – mostly from CLOS
  2. External commands execution and redirection syntax – bash (while changing behaviour of the $ expansion, now known to be source of many bugs).
  3. Anonymous functions and closures – not sure where I got it from, I’d guess Lisp.
  4. Declarative primitives concept – heavily based on basic concept behind configuration management tools
  5. inspect() – Ruby‘s inspect()
  6. code() – the .perl property from Perl. Not fully implemented in NGS. Should return the code that would evaluate to the given value. Probably comes from Lisp originally which usually prints the values in a way it can read them back.
  7. Ordered hash (When you iterate a Hash, the items are in the same order as they were inserted) – PHP, also implemented in Ruby and if I’m correct in V8.
  8. The in and not in operators – Python.
  9. Many higher order functions such as map and filter probably originated in Scheme or Lisp.
  10. Translate anything that possible into function calls. + operator for example is actually a function call – Python, originally probably from Scheme or Lisp.

Ideas trashed before implementation

While working on NGS I think about features I can add. I have to admit that many of these candidate features seem fine for a few seconds to a few minutes… till I realize that something similar is already implemented in language X and it does not resonate with my way of thinking (also known as “This does not look good”).

Square brackets syntax for creating a list

Suppose we have this common piece of code:

list = []
some_kind_of_loop {
    ...
    list.push(my_new_item)
    ...
}

In this case it would advantageous to express the concept of building a list with a special syntax so it would be obvious from a first glance what’s happening. Since array literal already had [1,2,3,...] syntax, the new building-a-list syntax would be [ something here ]. And that’s how we get list comprehensions in Python (also in other languages). The idea is solid but when I see it implemented, it’s very easy for me to realize that I don’t like the syntax. List comprehension syntax is the preferred way to construct lists in Python and seems like it’s being widely used. I’d like to make clear that I do like Python and list comprehensions is one of the very few things I’m not fond of.

So what NGS has for building lists? Let’s see:

# Python
[x*2 for x in range(10) if x > 5]

# NGS straightforward translation
collector
  for(x;10)
    if x > 5
      collect(x * 2)

Doesn’t look like much of an improvement. That’s because straightforward translations are not representative. The collector‘s use above is not a good example of it’s usage. More about collector later in this article.

Here are the NGS-y ways:

# Python
[x*2 for x in range(10) if x > 5]

# NGS alternatives, pick your compactness vs clarity
(0..10).filter(F(elt) elt>5).map(F(elt) elt*2)
10.filter(F(elt) elt>5).map(F(elt) elt*2)
10.filter(X>5).map(X*2)
10?(X>5)/(X*2)

NGS does not have special syntax for building lists. NGS does have special syntax for building anything and it’s called collector.

Syntax for everything

Just say “no”. It is tempting at first to make syntax for all the concepts in the language or at least for most of them. When taken to it’s extreme, we have Perl and APL.

I don’t like Perl’s syntax, especially sigils, especially that there is more than one sigil, (unlike $ in bash). Note that Perl kind of admitted sigils were used inconsistently (or confusingly? not sure) so sigils syntax was partially changed in Perl 6. On the other hand, Perl 6 added sigil modifiers, called “twigil“s. I counted 9 of them.

APL has many mathematical symbols in it’s syntax. Many of them are not on a keyboard. While the language is very terse and expressive, I don’t like the syntax and the fact that typing in a program in APL is not a straightforward task. [Update: Clarification: I’m not familiar with APL as opposed to other languages mentioned in this article. I was pointed out that I was quick to judge APL. It does have mathematical symbols but it actually doesn’t have much syntax. Most of the symbols are just built-in functions and operators. I’ve also found out one symbol for “go to line” and one for “define function”/”end function”. Actually APL syntax rules are interesting “think different” approach as they are implementation of mathematical notation. Thanks for the link, geocar.]

I do agree with Python’s developers reluctant attitude towards adding new syntax, especially new characters. Perl appears to be on the other end of the spectrum.

I’d like to come back to the collector syntax and show that even when adding a syntax, it’s advantageous to add less of it and reuse existing language concepts if possible.

The initial thought was to base the collector syntax on Lisp’s (loop …) macro , specifically the macro’s collect keyword: (loop ... (collect ...) ...). I was about do add two new keywords: collector and collect. The code would look like this:

mylist = collector
  for(x;10)
    if x > 5
      collect x * 2

I’ve started considering implementation strategies and came up with the idea that it would be simpler to have one keyword only, collector. collector wraps single expression to the right (which can be a { code block }) in a function with one argument named collect. The code becomes this:

mylist = collector
  for(x;10)
    if x > 5
      collect(x * 2)

When the collector expression is evaluated, the wrapped code runs with provided collect function. When collect is a function and not a special syntax it allows to use all the facilities that work with functions. Example: somelist.filter(...).each(collect) .

There is more to collector! It has optional initialization argument. Let’s see it in the following example:

collector/0
    arr.each(F(elt) {
        if predicate(elt)
            collect(1)
    })

The code above counts number of elements in arr which satisfy the predicate. That’s roughly how standard library’s count() is implemented.

When collector is not followed by / (slash) and initialization value, the default initialization argument is newly created empty list. The default initial value was chosen to be empty list as it appears that most uses of collector

Advantages of the implementation where collect is a function and not a keyword:

  1. Simpler implementation
  2. Collector behaviour customization – there is a way to provide your own collect function for your types which I will not describe here. Standard library defines such functions for array, hash and integer types.
  3. Functional facilities available for the collect function
  4. collect function can have more than one argument making it possible to collect key-value pairs if constructing a Hash: collector ... collect(k, v) ... (this is from filter() implementation for Hash in standard library).

I consider collector to be a good example of “less syntax is better”.

Easier, not easy

I do enjoy the situation that I have many other languages to look at for ideas, good or bad.

NGS needs original features for it’s domain (systems administration tasks niche). These domain-specific features make NGS stand out. Other languages don’t have them because they are either not domain-specific or have wider domain or are too old to have features for the today’s systems administration domain. Every original feature has a risk attached to it. If you haven’t seen it work before there is practically no way to predict how good or bad a feature will turn out. More about original features in NGS in another post.

While creating a language is now easier than before it’s still not an easy task 🙂 … but you can help. Fork, code, make a pull request.


Have a nice weekend!

Update: discussion on reddit

JSON vs data structure

When you see var j = {"x": 1} in JavaScript, it is plainly wrong to say that j is now JSON or references JSON or holds JSON for that matter.

binary-797263_640

That’s because j now references a data structure in memory. A data structure is not JSON nor YAML nor any other serialization format.

A data structure can sometimes be serialized to JSON or to YAML or to other formats. A data structure can sometimes be deserialized from these formats.

The code on the right side of the assignment looks like JSON. Don’t let this confuse you. It’s a JavaScript code and it evaluates to a data structure as many other JavaScript expressions do. It could easily be var j = {'a': 1, 'f': function() {} } . You wouldn’t say it’s JSON, right? There is not much difference between the two JavaScript expressions var j = {"x": 1} and var j = {'a': 1, 'f': function() {} } for this matter.

Data structure vs it’s serialized form

Data structure is the layout of the data. In our case it is in memory. Data structures can also be on disk, think data file of a database. Data structures are “good for” accessing and modifying the data that they hold. In our case it means one can use the expression j.x to access the field x or j.x = 7 to modify it.

Serialized form of a data is a string of characters that can be saved to a file, read from such file or sent over the network. There is no easy way to manipulate such data directly. Modifying serialized data usually involves deserializing it, modifying and serializing back.

Serialization limitations

Not any data structure can be serialized at all. Example (specific to JSON format):

var j = {}
j['circ'] = j
JSON.stringify(j)
TypeError: Converting circular structure to JSON

Not any data structure can be serialized in a way that would ensure that desirialization would produce similar data structure. Example (specific to JavaScript + JSON):

var j = {'a': 1, 'f': function() {} }
JSON.stringify(j)
'{"a":1}'

Have a nice week!

dnsimple SSL defaults to simple, not secure

padlock-166882_640

When purchasing an SSL certificate at dnsimple I was amazed to discover that by default it’s dnsimple who generates the private key for you. The checkbox that says “I want to provide a custom CSR” is unchecked by default. Not checking it causes dnsimple to generate your private key and a CSR for you. Simple – yes. Secure – no.

Generating a private key for anyone but yourself is a big security no-no. This is common knowledge for people that are selling certificates. Still somehow it’s default at dnsimple.

Please stop doing these stupid things!

Declarative primitives or mkdir -p for the cloud

After some positive feedback regarding the concept of declarative primitives I would like to elaborate about it.

Defining declarative primitives

Declarative primitives is just a description of existing techniques. I gave it a name because I’m not aware of any other term describing these techniques. The idea behind declarative approach is to describe the desired state or result and not particular command or operations to achieve it.

Example: mkdir -p dir1/dir2/dir3

The outcome of the command does not depend on current state (whether the directory exists or not). You describe the desired state: directories dir1, dir2 and dir3 should exist after the command is run. Note that mkdir dir1/dir2/dir3 does not have the same effect: it fails if dir1 does not exist or dir2 does not exist or if dir3 exists.

The phrase declarative primitives emphasizes granularity. Existing declarative tools for the cloud operate on many described resources, build dependency graphs and run in order that they decide. Declarative primitives provide a very flexible way to control a single resource or a group of few resources of the same type. The flexibility comes from granularity. You decide how you combine the resources. You can easily integrate existing resources. You can modify just the properties of your interest on the resources you choose. This approach is ideal of scripting in my opinion.

Where are declarative primitives for the cloud?

sky-414198_640

I believe that when writing a script, using mkdir -p should be similar to using AwsElb(...).converge() for example. I’m working on implementing it (as a library for the Next Generation Shell) and I’m not aware of any other project that does it.

There are many projects for managing the cloud, how are they different?

Here are the solutions that I’m aware of and how familiar I am with each one:

  1. CloudFormation – using frequently (I prefer YAML syntax for it)
  2. Terraform – I’ve read the documentation and bits of source code
  3. Cloudify – familiar with the product, made modules for it
  4. Puppet – was using it intensively on few different projects
  5. Chef – was using it intensively in many projects
  6. Ansible – unfamiliar with this one (only took a look at documentation) so not reviewing it below
  • All take the declarative approach. You describe many resources or the entire system and feed the description to the tool which in turn does all the work. None of these solutions was designed to provide you with the primitives that could be easily used in your scripts. These tools just don’t match my view regarding scripting.
  • These tools can do a rollback on error for example. They can do that precisely because they have the description of the entire system or big parts of it. It will take some additional work to implement rolling back using declarative primitives. The question is whether you need the rollback functionality …
  • Some of these tools can be made to work with different clouds relatively easily. Working with different clouds easily may also possible with declarative primitives but the library I’m currently working on does not have such goal.
  • Except for Chef, the tools in the list above use formats or DSLs not based on real programming languages. This means that except for trivial cases you will be using some additional tool to generate the descriptions of desired states. Limited DSLs do not work. See Puppet and Ansible that started with simple description languages and now they are almost real programming languages … which where never designed as programming languages, which has consequences.
  • I’m not aware of any option in the tools above that lets you view definitions of existing resources, which prevents you from starting managing existing resources with these tools and from cloning existing resources. I have started implementing the functionality that lets you generate the script that would build an existing resource: SomeResource(...).code() . This will allow easy modification or cloning.
  • A feature missing both from these tools and from my library is generating a code to start with for a given resource type (say security group or load balancer). Writing CloudFormation definition for a type with many properties is a nightmare. Nobody should start from scratch. Apache or Nginx configuration files are good example of starting points. Similar should be done for the cloud resources.
  • Note that Chef and Puppet were originally designed to manage servers. I don’t have any experience using them for managing the cloud but I can guess it would be less optimal than dedicated tools (the first three tools).

Scripting the cloud – time to do it right!