What I did not steal from Perl 6

I’m curious about programming languages. Not because I’m creating one right now. I always was. This post is about ideas and features that I have seen in Perl 6 and found interesting. If you are curious about programming languages in general, you should take a look at these.

There are various reasons for not stealing the interesting ideas from Perl 6:

  1. I’m trying to keep number of concepts in NGS as small as possible. If I’m not seeing huge immediate value in a concept – I skip it.
  2. Not taking anything that I think can confuse me or other programmers. I’m not talking here because someone is a beginner. I’m talking about confusing concepts.
  3. Simply because I don’t have enough resources to implement it at the moment.

Here are the interesting Perl 6 features, in no particular order (except the first one). There are also my comments whether I would like the feature in NGS or why not.

  1. Syntax. Very expressive an terse. Perl6 has even more of it than Perl 5. Now that we got rid of the $ and friends in the room:
  2. Grammars. Would actually be nice to have something like that in NGS.
  3. Lots of operators. The most interesting concept is Metaoperators. I’m trying to keep the amount of syntax elements in NGS relatively low. There are already two syntaxes in NGS: commands and expressions. Not taking more syntax without serious need.
  4. How the “pointy block” syntax mixes with “for” syntax: for @list -> @element . NGS already has several syntaxes for Lambdas.
  5. Flow control
    1. when” flow control. The closest NGS has is “cond” and friends, stolen from Lisp.
    2. repeat while / repeat until . It would be nice to have something like that in NGS.
    3. once . Not sure about this one. The functionality might be needed.
  6. Slips. The behaviour is frightening me: if it does expand, how do I pass a Slip if I just want to pass it, say as an item of an array? NGS uses syntax for slips: [1, 2, *myitems, 3, 4] which I think is cleaner. You know you can’t pass it because it’s syntax.
  7. .WHAT method. I stole something similar from Ruby: the inspect method.

As a special note, I have seen a welcome change from $arr[0] to @arr[0] . I think it removes confusion. (That was Perl 5 vs Perl 6).

Please don’t be offended if you are a Perl 6 hacker and you see that there is amazing feature that I have not mentioned. It could be that I’ve seen this in several other languages already or maybe I did not find it interesting or … maybe I just missed it. Don’t hesitate to leave a comment anyway.


Happy coding, in whatever language rocks your boat! Except for bash. Coding in bash will never be happy.

List of JSON tools for command line

I am considering making a JSON parsing and generating command line tool. Started with looking around a bit. Below is a list of existing JSON command line tools. Numbers are [GitHub stars] at the time of writing this post. (… contributed by …) means that this post was updated with the item.

  • jq [11126] – filter, extract, modify and output JSON or text using DSL
  • jid [4426] – “You can drill down JSON interactively by using filtering queries like jq.” (item contributed by /u/Tacticus)
  • gron [4103] – convert JSON or JSON lines (from file/stdin/url) to text (path=value) which can be processed with grep/sed/diff; the tool also supports converting back to JSON after such processing
  • jo [2209] – generate JSON based on command line arguments and stdin; can read data from files and place it as base64 encoded values
  • JSON.sh [1635] – written in shell/gawk; “traverses the JSON objects and prints out the path to the current object (as a JSON array) and then the object, without whitespace”
  • jsawk [1239] – focused primarily on filtering and transforming a list (or an object)
  • json (by trentm) [1218] – “massaging JSON on your Unix command line”; JS-like syntax for extracting values; in-place file editing
  • rq [1007] – awk/sed-like tool for structured data; supports several formats, including JSON
  • TickTick [469] – use JSON syntax directly in bash; “This is just a fun hack”
  • jshon [309] – very CLI-ish way to extract, manipulate and output the data
  • jl [308] – “a tiny functional language for querying and manipulating JSON”; visually reminds Haskell
  • jsonpp [244] – JSON pretty printer (item contributed by /u/ferbass)
  • fx [227] – conveniently run your JS code to manipulate JSON.
  • RecordStream [224] – create, manipulate and output records; supports JSON; Perl-based so grep expressions for example are in Perl.
  • JSON.awk [186] – JSON.sh fork in awk; after fork the projects added different features.
  • jp [184] – “command line interface to JMESPath” (link contributed by Evgeny Zislis)
  • json-command [143] – conveniently manipulate JSON using JS.
  • jsonv.sh [130] – convert JSON to CSV; specify paths in JSON to
  • jgrep (aka “JSON-grep”) [78] – “Command line tool and API for parsing JSON documents” in Ruby (item contributed by /u/tophlammiepie)
  • jsed [48] – manipulate and extract data; somewhat similar to jsawk in mindset
  • jsongrep [9] (by dsc) – extract data at given path using shell globs and output one per line
  • jsongrep [0] (by terrycojones) – easily extract data at given path

Honorable mentions

Update 2018-09-10:

I’ve added related post in which I argue that jq functionality belongs to a shell.


If you feel that some project is missing from the list, please let me know in comments below.

The missing link of Ops tools

It’s like we went from horse to spaceship, skipping everything in between.

Background

Let’s say you are managing your system in AWS. Amazon provides you with API to do that. What are your options for consuming that API?

Option 1: CLI or library for API access

AWS CLI let’s us access the API from the command line and bash scripts. Python/Ruby/Node.js and other languages can access the API using appropriate libraries.

Option 2: Declarative tools

You declare how the system should look like, the tool figures out dependencies and performs any API calls that are needed to achieve the declared state.

Problem with using CLI or API libraries

Accessing API using CLI or libraries is fine for one off tasks. In many cases, automation is needed and we would like to prepare scripts. Ideally, these scripts would be idempotent (can be run multiple times, converging to the desired state and not ruining it). We then quickly discover how clunky these scripts are:

# Script "original"
if resource_a exists then
  if resource_a_property_p != desired_resource_a_property_p then
    set resource_a_property_p to desired_resource_a_property_p
  end
  if resource_a_property_q != desired_resource_a_property_q then
    ...
  end
else
  # resource_a does not exist
  create resource_a
  set resource_a_property_p to desired_resource_a_property_p
  ...
end
# more chunks like the above

It’s easy to see why you wouldn’t want to write and maintain a script such as above.

How the problem was solved

What happened next: jump to “Option 2”, declarative tools such as CloudFormation, Terraform, etc.

rocket-1374248_640

Other possible solution that never happened

If you have developed any code, you probably know what refactoring is: making the code more readable, deduplicate shared code, factoring out common patterns, etc… without changing the meaning of the code. The script above is an obvious candidate for refactoring, which would be improving “Option 1” (CLI or a library for API access) above, but that never happened.

All the ifs should have been moved to a library and the script could be transformed to something like this:

# Script "refactored"
create_or_update(resource_a, {
  property_p = desired_resource_a_property_p
  property_q = desired_resource_a_property_q
})
# more chunks like the above

One might say that the “refactored” script looks pretty much like input file of the declarative tools mentioned above. Yes, it does look similar; there is a huge difference though.

Declarative tools vs declarative primitives library

By “declarative primitives library” I mean a programming language library that provides idempotent functions to create/update/delete resources. In our cases these resource are VPCs, load balancers, security groups, instances, etc…

Differences of declarative tools vs declarative primitives library

  1. Declarative tools (at least some of them) do provide dependency resolution so they can sort out in which order the resources should be created/destroyed.
  2. Complexity. The complexity of mentioned tools can not be ignored; it’s much higher than one of  declarative primitives library. Complexity means bugs and higher maintenance costs. Complexity should be considered a negative factor when picking a tool.
  3. Some declarative tools track created resources so they can easily be destroyed, which is convenient. Note that on the other hand this brings more complexity to the tool as there must be yet another chunk of code to manage the state.
  4. Interacting with existing resources. Between awkward to impossible with declarative tools; easy with correctly built declarative primitives library. Example: delete all unused load balancers (unused means no attached instances): AWS::Elb().reject(X.Instances).delete()
  5. Control. Customizing behaviour of your script that uses declarative primitives library is straightforward. It’s possible but harder with declarative tools. Trivial if in a programming language can look like count = "${length(var.public_subnets) > 0 ? 1 : 0}" (approved Terraform VPC module).
  6. Ease of onboarding has declarative tools as a clear winner – you don’t have to program and don’t even need to know a programming language, but you can get stuck without knowing it:
  7. Getting stuck. If your declarative tool does not support a property or a resource that you need, you might need to learn a new programming language because the DSL used by your tool is not the programming language of the tool itself (Terraform, Puppet, Ansible). When using declarative primitives library on the other hand you can always either extend it when/if you wish (preferable) or make your own easy workaround.
  8. Having one central place where potentially all resources are described as text (Please! don’t call that code, format is not a code!). It should be easier done with declarative tools. In practice, I think it depends more on your processes and how you work.

As you can see, it’s not black and white, so I would expect both solutions be available so that we, Ops, could choose according to our use case and our skills.

My suggestion

I don’t only suggest to have something between a horse and a spaceship; I work on a car. As part of the Next Generation Shell (a shell and a programming language for ops tasks) I work on declarative primitives library. Right now it covers some parts of AWS. Please have a look. Ideally, join the project.

Next Generation Shell – https://github.com/ilyash/ngs

Feedback

Do you agree that the jump between API and declarative tools was too big? Do you think that the middle ground, declarative primitives approach, would be useful in some cases? Comment here or on Reddit.


Have a nice day!

Why I have no favorite programming language

TL;DR – because for me there is no good programming language.

I’m doing mostly systems engineering tasks. I manage resources in Cloud and on Linux machines mostly. I can almost hear your neurons firing half a dozen names of programming languages. I do realize that they are used by many people for systems engineering tasks:

  • Go
  • Python
  • Ruby
  • Perl
  • bash

The purpose of this post is not to diminish the value of these languages; the purpose is to share why I don’t want to use any of the languages above when I write one of my systems-engineering-task scripts. My hope is that if my points resonate with you, the reader, you might want to help spread the word about or even help with my suggested solution described towards the end.

man-2756206_640

So let’s go over and see why I don’t pick one of the languages:

Why not language X?

All languages

  • Missing smart handling of exit codes of external processes. Example in bash: if test -f my_file (file is not there, exit code 1) vs if test --f my_file (syntax error, exit code 2). If you don’t spot the syntax error with your eyes, everything behaves as if the file does not exist.
  • Missing declarative primitives libraries (for Cloud resources and local resources such as files and users). Correction: maybe found one, in Perl – (R)?ex ; unfortunately it’s not clear from the documentation how close it is to my ideas.

All languages except bash

  • Inconvenient/verbose work with files and processes. Yes, there are libraries for that but there is no syntax for that, which would be much more convenient. Never seen something that could compare to my_process > my_file or echo my_flag > my_file .

Go

  • Compiled
  • Error handling is a must. When I write a small script, it’s more important for me for it to be concise than to handle all possible failures; in many cases I prefer an exception over twice-the-size script. I do understand how mandatory and explicit error handling can be a good thing for larger programs or programs with greater stability requirements.
  • Dependencies problem seem to be unresolved issue

Python

  • Functional programming is second level citizen. In particular list/dictionary comprehension is the Pythonic way while I prefer map and filter. Yes, that’s probably one of the features that makes Python easier to learn and suggested first language. Not everything that’s optimized for beginners must be good for more experienced users. It’s OK.
  • Mixed feelings about array[slice:syntax] . It’s helpful but slice:syntax is only applicable inside [ ] , in other places you must use slice(...) function to create the same slice object

Ruby and Perl

  • The Sigils syntax does not resonate with me.

Ruby

I can’t put my finger on something specific but Ruby does not feel right for me.

Perl

  • Contexts and automatic flattening of lists in some cases make the language more complicated than it should.
  • Object orientation is an afterthought.
  • Functions that return success status. I prefer exceptions. Not the default behaviour in Perl but an afterthought: autodie.
  • Overall syntax feeling (strictly matter of personal taste).

bash

Note that bash was created in a world that was vastly different from the world today: different needs, tasks, languages to take inspiration from.

  • Missing data structures (flat arrays and hashes is not nearly enough). jq is a workaround, not a solution in my eyes.
  • Awkward error handling with default of ignoring the errors completely (proved to be bad idea)
  • Expansion of undefined variable to empty string (proved to be bad idea)
  • -e ,  -u and other action at a distance options.
  • Unchecked facts but my feelings:
    • When bash was created, it was not assumed that bash will be used for any complex scripting.
    • bash was never “designed” as a language, started with simple commands execution and other features were just bolted on as time goes by while complete redesign and rewrite were off the table, presumably for compatibility.
  • Syntax
  • No widely used libraries (except few for init scripts) and no central code repository to search for modules (Correct me if I’m wrong here. I haven’t heard of these).

My suggested solution

I would like to fill the gap. We have systems-engineering-tasks oriented language: bash. We have quite a few modern programming languages. What we don’t have is a language that is both modern and systems-engineering-tasks oriented. That’s exactly what I’m working on: Next Generation Shell. NGS is a fully fledged programming language with domain specific syntax and features. NGS tries to avoid the issues listed above.

Expected questions and my answers

People work with existing languages and tools. Why do you need something else?

  • I assume I have lower bullshit tolerance than many others. Some people might consider it to be normal to build more and more workarounds (especially around anemic DSLs) where I say “fuck this tool, I would already finish the task without it (preferably using appropriate scripting language)”. I don’t blame other people for understandable desire to work with “standard” tools. I think it’s not worth it when the solutions become too convoluted.
  • I am technically able to write a new programming language that solves my problems better than other languages.

Another programming language? Really? We have plenty already.

  • I would like to remind you that most of the programming languages were born out of dissatisfaction with existing ones.
  • Do you assume that we are at global maximum regarding the languages that we have and no better language can be made?

Feedback

Would you use NGS? Which features it must have? What’s the best way to ease the adoption? Please comment here, on Reddit (/r/bash , /r/ProgrammingLanguages) or on Hacker News.


Update: following feedback roughly of the form “Yes, I get that but many Ops tasks are done using configuration management tools and tools like CloudFormation and Terraform. How NGS compares to these tools” – there will be a blog post comparing NGS to the mentioned tools. Stay tuned!


Have a nice day!

NGS unique features – Argv command line arguments builder

Background: what is NGS?

NGS LOGO

NGS, the Next Generation Shell is a (work in progress) shell and a programming language built ground up for systems engineering tasks. You can think of it as bash that’s designed today: sane syntax, data structures, functional programming, extensibility, cloud in mind, declarative primitives.

What’s the problem with constructing command line arguments?

The problem affects only more “advanced” cases of constructing command line arguments when some arguments might or might not be present. Let’s consider this example:

# Made-up syntax, resembling NGS
args = []
if 'Subnets' in props {
  args += '--subnets'
  args += props['Subnets']
}
if ... {
  args += ...
}
if ... {
  args += ...
}
...
aws elb create-load-balancer ... $args

Wouldn’t it be cleaner to get rid of all the ifs? … and what happens if props['Subnets'] is an empty array?

How Argv facility in NGS solves the problem?

Argv is a result of factoring out the common code bits involved in constructing command line arguments. The ifs above were also factored out. They are now in Argv.

Let’s look at usage example (real NGS code, from AWS library)

argv = Argv({
  '--load-balancer-name': rd.anchor.name
  '--listeners': props.ListenerDescriptions.encode_json()
  '--subnets': rd.opt_prop('Subnets', props).map(only(ResDef, ids))
})
rd.run('create ELB', %(aws elb create-load-balancer $*argv))

The important points here are:

  1. Argv is a function with a single parameter which must be of type Hash (also called “dictionary” in some languages)
  2. The keys of the Hash are switches’ names (--load-balancer-name, --listeners, --subnets)
  3. The values of the Hash are values for the switches

The “if” that decides whether a switch is present in the resulting argv is inside Argv implementation and your code is clean of it. The values of the Hash are considered when Argv decides whether a switch should be present. null, empty array and instances of type EmptyBox are considered by Argv as missing values and it discards the switch. For convenience, instances of type FullBox are unboxed when constructing the result of Argv.

The Argv facility is yet another point among others that shows why NGS and systems engineering tasks are best fit.


Have a nice weekend!

 

NGS unique features – improving NodeJS require()

Background: what is NGS?

NGS, the Next Generation Shell is a (work in progress) shell and a programming language built ground up for systems engineering tasks. You can think of it as bash that’s designed today: sane syntax, data structures, functional programming, extensibility, cloud in mind, declarative primitives.

What’s good in NodeJS’ require()

I like most of how require() works in JavaScript. I’m not talking in this post about npm, just NodeJS require() function. require() does not pollute your namespace, you just get a reference, it’s simple to use and easy to reason about.

const a = require('cool-aws-wrapper');
// Can not be done easily with AWS SDK:
a.deleteRoute53Record('testing25.example.com');

What’s there to improve in require() ?

NodeJS modules are usually fall into one of the categories:

  1. Class definition / big library that manages it’s own namespace. These usually end with module.exports = MyClass. No problem here.
  2. Group of functions or classes. These usually end with module.exports = { func1, func2, func3, ...} lists (ES6 syntax, otherwise written as module.exports =  { func1: func1, ... } ) which I think are cumbersome.

How require() and modules look in NGS?

Note that require() in NGS is work in progress and it doesn’t have much of the functionality that NodeJS provides. I just started with things that bothered me the most.

Consistent with other places in NGS, require() returns the last evaluated expression. NodeJS for example returns module.exports which you must explicitly set as the result of require().

I think of modules primary as a namespaces. Creating a namespace in NGS has a syntax: ns { ... } .

Combining require() behaviour of returning last evaluated expression and namespace syntax, typical NGS module consists of single top level expression which evaluates to a namespace. The whole module file can look like this:

ns {

  global init

  type Vpc
  type Subnet

  F init(v:Vpc) {
    ...
  }

  F _helper_func(s:Str) { ... }

  MY_CONST = 42

  F ok() {
    echo("OK")
  }

}

Let’s ignore the global for now, it’s about how methods and types’ instances creation are implemented in NGS. Anything defined inside the ns { ... } is exposed as namespace member so usage of the above module could look like this:

{
  m = require('mymodule.ngs')
  vpc = m::Vpc()
  echo(m::MY_CONST)
  m::ok()
}

As you probably guessed, the :: operator is the namespace member access operator.

There is no need to explicitly state what module/namespace exports. That’s the improvement over NodeJS’ require().

How ns works and more options for the curios

ns { … } returns a Hash

As stolen from NodeJS, the namespace syntax (ns { ... }) returns a Hash. In NodeJS, require() typically returns JavaScript Object which is close enough for the purpose of this post.

About :: operator

The namespace member access operator :: is actually a Hash key access operator. It is helpful because the regular syntax for accessing members is not always a good fit for namespaces. The regular member access syntax is dot (.) but the dot syntax is also a function call: myobj.field – is a field/key/attribute access but myobj.func() is equivalent to func(myobj). For example, m::ok() will call the ok function defined in the module, m.ok() will call the function ok in current lexical environment with m as parameter.

As a bonus, since :: is an operator, it is implemented as function call. This means you can define how :: works with types that you define and modify how :: works with existing types.

ns { … } syntax implementation

For simplicity of implementation and absence of obvious reasons against, ns { ... } syntax is just a syntactic sugar for defining anonymous function without parameters and calling it immediately. The though behind this decision was simple: “I need to implement namespaces. Let’s see where I have them already. Oh, namespaces are already implemented in functions. This is so convenient, I can use this mechanism with minimal effort”.

How ns knows what to return?

ns is mostly a syntactic hack:

  1. Inside the ns body, the first statement, before any use-supplied statements is _exports = {} which sets the local variable _exports to an empty Hash.
  2. Any assignment and function definition also set _exports["something"]. MY_CONST = 42 becomes MY_CONST = 42;  _exports["MY_CONST"] = MY_CONST;
  3. Exception to the rule above are variables and functions with names starting with underscore (_). They are not automatically added to _exports. This for example is why _exports itself is not exported.
  4. Last statement, after all user-supplied statements is _exports.

The behavior I just described looks like sane defaults to me. As we all know, the life is usually more complex than hello world examples and customizations are need. Here are two ways to customize the resulting namespace.

  1. return your_expr – since ns is just a function, you can use return at any point to return your own custom namespace.
  2. manipulate _exports however you want towards the end of ns body. For example after _exports .= filterv(Type) only types will be exported. _exports.filterk(/^pub_/) will only export symbols (keys) that have names that start with pub_ .

Improvement suggestions are welcome! Have a nice day!

NGS unique features – Hash methods I wish I had in other languages

NGS is a language and a shell that I am building for systems administration tasks. Enough of the language is implemented to enable writing some useful scripts. The shell is not there yet.

Some of the Hash methods in NGS

The methods for working with Hash I have not seen all at once in other languages are:

  1. filterk – filter Hash by key (produces Hash)
  2. filterv – filter Hash by value (produces Hash)
  3. mapk – map Hash keys (produces Hash)
  4. mapv – map Hash values (produces Hash)
  5. mapkv – map Hash keys and values (produces Hash as opposed to map which produces an array)
  6. without – filters out specific key

How these are actually used? Following is an excerpt from the pollute method (function), which is a part of the AWS module. It uses several of the Hash methods I mentioned, making the method a good example. pollute method (as in “pollute global namespace”) enables using Vpc variable for example instead of AWS::Vpc and so on. I would like to have this behaviour for small quick-and-dirty scripts but not as default so it’s in a method that one can optionally call.

F pollute(do_warn=true) {

    vars =
        _exports.filterk(/^AMI_OWNER/) +
        _exports.filterv(Type).without('Res').without('ResDef') +
        ...

    if do_warn {
        warn("Polluting ...: ${vars.keys().join(', ')}")
    }

    vars.mapk(resolve_global_variable).each(set_global_variable)
}

Let’s go over the code above step by step:

F pollute(do_warn=true) { ... } defines the pollute method with optional parameter do_warn that has default value of true.

_exports is a Hash containing all of the AWS’ module variables and functions, similar to NodeJS module.exports but members are added automatically rather than explicitly. Only the methods and variables that do not start with _ (underscore) are added. One can modify _exports in any way before the end of the module. I will write more in detail about require() and modules in NGS in another post.

filterk(/^AMI_OWNER/) filters all the variables that match the given RegExp

filterv(Type) filters all the variables that are of type Type. These are AWS types’ definitions, such as Vpc, Subnet or Instance.

without('...') filters out the types I don’t like to override.

+ between and after _exports.filterk(...) and _exports.filterv(...) joins the hashes.

mapk translates variables’ names into their index (using resolve_global_variable)

each runs set_global_variable with variable index and the value to set

Hash methods in other languages

I am aware that some of the methods above are present in other languages or libraries. Some examples:

  1. Ruby has mapv (transform_values) method.
  2. Rails has mapk (transform_keys) and mapv.
  3. Perl 6 can modify values in a convenient manner:for %answers.values -> $v is rw { $v += 10 };.

What I have not seen is a language which has all the methods above out of the box. I have a feeling that arrays get all the fame methods while hashes (dictionaries) often get less attention in other languages.

Why NGS has all these methods?

NGS is aiming to be convenient for systems administration tasks. More often than not these tasks include data manipulation. NGS has many functions (methods) for data manipulation, including the ones listed in this post.

Update: reddit discussion


Have a nice day!