How fucked is AWS CLI/API

In the 21st century, we can do significantly better than this crap AWS CLI. We just need to think from the users’ perspective for a change and not just dump on users whatever we were using internally or seen in a nightmare and then implemented.

thinking-3082823_640
Think from the users’ perspective

I’m working on a solution which is described towards the end of the post. It’s not ready yet but the (working btw) examples will convince the reader that “significantly better” is very possible… I hope.

Background

I’m building AWS library and a command line tool. Since I’m doing it for a shell-like language (NGS), the AWS library that I’m building uses AWS CLI. Frustration and anger are prominent feelings when working with AWS CLI. I will just list here some of the reasons behind that feeling.

furious-2514031_640
AWS CLI/API – WTF were they thinking?

Overall impression

Separate teams worked on different services and were not talking to each other. Then the something like this happened: “OK, we have these APIs here, let’s expose them. User experience? No, we don’t have time for that / we don’t care”.

Tags

Representing a map (key-value pairs) as an array ( "Tags": [ { "Key": "some tag name", "Value": "some tag value" }, ... ] ), as AWS CLI does is insane… but only when you think about the user (developer in this case) experience.

shocked-2681488_640
AWS Tags – no, not in this form!

Reservations

When listing EC2 instances using aws ec2 describe-instances, the data is organized as list of Reservations and each Reservation has list of instances. I’m using AWS for quite a few years and I never needed to do anything with Reservations but I did spent some time unwrapping again and again the interesting data (list of instances) from the unwanted layer of madness. It really feels like “OK, Internally we have Reservations and that’s how our API works, let’s just expose it as it is”.

annoyed-3126442_640
Who da fuck ever used Reservations?

Results data structure

AWS CLI usually (of course not always!) returns a map/hash at the top level of the result. The most interesting data would be called for example LoadBalancerDescriptions, or Vpcs, or Subnets, or … ensuring difficulty making generic tooling around AWS CLI. Thanks! Do you even use your own AWS CLI, Amazon?

Inconsistencies

beer-2370783_640
Kind of the same but not really

Security Groups

These are of course special. Think of aws ec2 create-security-group ...

  1. --group-name and --description are mandatory command line arguments. For now, I haven’t seen any other resource creation that requires both name and description.
  2. The command returns something like { "GroupId": "sg-903004f8" } which is unlike many other commands which return a hash with the properties of the newly created resource, not just the ID.

Elastic Load Balancer

Oh, I like this one. It’s unlike any other.

  1. The unique key by which you find a load balance is name, unlike other resources which use ids.
  2. Tags on load balancers work differently. When you list load balancers, you don’t get the tags with the list like you have when listing instances, subnets, vpcs, etc. You need to issue additional command: aws elb describe-tags --load-balancer-names ...
  3. aws elb describe-load-balancers – items in the returned list have field VPCId while other places usually name it VpcId .

Target groups

aws elbv2 deregister-targets --target-group-arn ...

Yes, this particular command and it’s “target group” friends use ARN, not a name (as ELB) and not an ID (like most EC2 resources).

DHCP options (sets)

(Haven’t used but considered so looked at documentation and here is what I found)

Example: aws ec2 create-dhcp-options --dhcp-configuration "Key=domain-name-servers,Values=10.2.5.1,10.2.5.2"

Yes, the create syntax is unlike other commands and looks like filter syntax. Instead of say --name-servers ns1 ns2 ... switch you have "Key=domain-name-servers,Values=" WTF?

Route 53

aws route53 list-hosted-zones does not return the tags. Here is an output of the command:

{
 "HostedZones": [
 {
 "Id": "/hostedzone/Z2V8OM9UJRMOVJ",
 "Name": "test1.com.",
 "CallerReference": "test1.com",
 "Config": {
 "PrivateZone": false
 },
 "ResourceRecordSetCount": 2
 },
 {
 "Id": "/hostedzone/Z3BM21F7GYXS7Y",
 "Name": "test2.com.",
 "CallerReference": "test2.com",
 "Config": {
 "PrivateZone": false
 },
 "ResourceRecordSetCount": 2
 }
 ]
}

Wanna get the tags? F*ck you! Here is the command: aws route53 list-tags-for-resources --resource-type hostedzone --resource-ids Z2V8OM9UJRMOVJ Z3BM21F7GYXS7Y . Get it? You are supposed to get the Id field from the list generated by list-hosted-zones, split it by slash and then use the last part as resource ids. Tagging zones also uses the rightmost part of id: aws route53 change-tags-for-resource --resource-type hostedzone --resource-id Z2V8OM9UJRMOVJ --add-tags Key=t1,Value=v11

… but that apparently was not enough differentiation 🙂 Check this out: aws elb add-tags --load-balancer-names NAME --tags TAGS vs aws route53 change-tags-for-resource --resource-type hostedzone --resource-id ID --add-tags TAGS and aws elb remove-tags --load-balancer-names NAME --tags TAGS vs aws route53 change-tags-for-resource --resource-type hostedzone --resource-id ID --remove-tag-keys TAGS . Trick question: on how many dimensions this is different? So wrong on so many levels 🙂

Here is another one: when you fetch records from a zone you use the full id, /hostedzone/Z2V8OM9UJRMOVJ , not Z2V8OM9UJRMOVJaws route53 list-resource-record-sets --hosted-zone-id /hostedzone/Z2V8OM9UJRMOVJ

 

(many more to come)

Expected comments and answers

feedback-2044700_640
Internet discussions are the best 😉

Just use Terraform

I might some day. For the cases I have in mind, for now I prefer a tool that:

  1. Is conceptually simpler (improved scripting, not something completely different)
  2. Doesn’t require a state file (I don’t want to explain to a client with two servers where the state file is and what it does)
  3. Can be easily used for ad-hoc listing and manipulation of the resources, even when these resources were not created/managed by the tool.
  4. Can stop and start instances (impossible with Terraform last time I checked a few months ago)

Just use CloudFormation

I don’t find it convenient. Also, see Terraform points above.

By the way, the phrases of the form “Just use X” are not appreciated.

You just shit on what other people do to push your tools

Yes, I reserve the right to shit on things. Here I mean specifically AWS CLI. Freedom of speech, you know… especially when:

  1. The product (AWS API) is technically subpar
  2. The creator of the product does not lack resources
  3. The product is used widely, wasting huge amounts of time of the users

If I’m investing my time to write my own tool purely out of frustration, I will definitely tell why I think the product is crap.

Is that just a rant or you have alternatives?

I’m working on it. The command line tool is called na (Ngs Aws). It’s a thin wrapper around declarative primitives library for AWS.

excited-3126450_640
There might be a solution to this madness!!!

The command line tool that I’m working on is not anywhere ready but here is a gist of what you can do with it:

# list vpcs
na vpc

# Find the default VPC
na vpc IsDefault:true

# List security groups of all vpcs which have tag "Name" "v1".
na vpc Name=v1 sg

# Idempotent operations, in this case deletion.
# No error when this group does not exist.
# Delete "test-group-name" in the given vpc(s).
na vpc Name=v1 sg GroupName:test-group-name DEL

# List all volumes which are attached to stopped instances
na i State:stopped vol

# Delete all volumes that are not attached to instances
na vol State:available DEL

# Stop all instances which are tagged as
# "role" "ocsp-proxy" and "env" "dev".
na i role=ocsp-proxy env=dev SET State:stopped
  1. Na never dumps huge amounts of data to your terminal. As a human, you will probably not be able to process it so when number of items (rows in a table actually) is above certain configurable threshold, you will see something like this:
    $ na vol 
    === Digest of 78 rows ===
    ...

    It will show how many unique values there are in each column, min and max value for each column. Thinking about displaying top 5 (or so) top and bottom values for each column.

  2. Na has concept of “related” resources so when you write something like na i State:stopped vol , it knows that the volumes you want to show are related to the instances that you mentioned. In this particular case, it means volumes attached to the instances.
  3. Note the consistency of what you see in output and arguments to CLI. If something is called “State” in the data structure returned by AWS CLI, it will be called “State” in the query, not “state” (“–state”).

I will be updating this post as things come along.

The missing link of Ops tools

It’s like we went from horse to spaceship, skipping everything in between.

Background

Let’s say you are managing your system in AWS. Amazon provides you with API to do that. What are your options for consuming that API?

Option 1: CLI or library for API access

AWS CLI let’s us access the API from the command line and bash scripts. Python/Ruby/Node.js and other languages can access the API using appropriate libraries.

Option 2: Declarative tools

You declare how the system should look like, the tool figures out dependencies and performs any API calls that are needed to achieve the declared state.

Problem with using CLI or API libraries

Accessing API using CLI or libraries is fine for one off tasks. In many cases, automation is needed and we would like to prepare scripts. Ideally, these scripts would be idempotent (can be run multiple times, converging to the desired state and not ruining it). We then quickly discover how clunky these scripts are:

# Script "original"
if resource_a exists then
  if resource_a_property_p != desired_resource_a_property_p then
    set resource_a_property_p to desired_resource_a_property_p
  end
  if resource_a_property_q != desired_resource_a_property_q then
    ...
  end
else
  # resource_a does not exist
  create resource_a
  set resource_a_property_p to desired_resource_a_property_p
  ...
end
# more chunks like the above

It’s easy to see why you wouldn’t want to write and maintain a script such as above.

How the problem was solved

What happened next: jump to “Option 2”, declarative tools such as CloudFormation, Terraform, etc.

rocket-1374248_640

Other possible solution that never happened

If you have developed any code, you probably know what refactoring is: making the code more readable, deduplicate shared code, factoring out common patterns, etc… without changing the meaning of the code. The script above is an obvious candidate for refactoring, which would be improving “Option 1” (CLI or a library for API access) above, but that never happened.

All the ifs should have been moved to a library and the script could be transformed to something like this:

# Script "refactored"
create_or_update(resource_a, {
  property_p = desired_resource_a_property_p
  property_q = desired_resource_a_property_q
})
# more chunks like the above

One might say that the “refactored” script looks pretty much like input file of the declarative tools mentioned above. Yes, it does look similar; there is a huge difference though.

Declarative tools vs declarative primitives library

By “declarative primitives library” I mean a programming language library that provides idempotent functions to create/update/delete resources. In our cases these resource are VPCs, load balancers, security groups, instances, etc…

Differences of declarative tools vs declarative primitives library

  1. Declarative tools (at least some of them) do provide dependency resolution so they can sort out in which order the resources should be created/destroyed.
  2. Complexity. The complexity of mentioned tools can not be ignored; it’s much higher than one of  declarative primitives library. Complexity means bugs and higher maintenance costs. Complexity should be considered a negative factor when picking a tool.
  3. Some declarative tools track created resources so they can easily be destroyed, which is convenient. Note that on the other hand this brings more complexity to the tool as there must be yet another chunk of code to manage the state.
  4. Interacting with existing resources. Between awkward to impossible with declarative tools; easy with correctly built declarative primitives library. Example: delete all unused load balancers (unused means no attached instances): AWS::Elb().reject(X.Instances).delete()
  5. Control. Customizing behaviour of your script that uses declarative primitives library is straightforward. It’s possible but harder with declarative tools. Trivial if in a programming language can look like count = "${length(var.public_subnets) > 0 ? 1 : 0}" (approved Terraform VPC module).
  6. Ease of onboarding has declarative tools as a clear winner – you don’t have to program and don’t even need to know a programming language, but you can get stuck without knowing it:
  7. Getting stuck. If your declarative tool does not support a property or a resource that you need, you might need to learn a new programming language because the DSL used by your tool is not the programming language of the tool itself (Terraform, Puppet, Ansible). When using declarative primitives library on the other hand you can always either extend it when/if you wish (preferable) or make your own easy workaround.
  8. Having one central place where potentially all resources are described as text (Please! don’t call that code, format is not a code!). It should be easier done with declarative tools. In practice, I think it depends more on your processes and how you work.

As you can see, it’s not black and white, so I would expect both solutions be available so that we, Ops, could choose according to our use case and our skills.

My suggestion

I don’t only suggest to have something between a horse and a spaceship; I work on a car. As part of the Next Generation Shell (a shell and a programming language for ops tasks) I work on declarative primitives library. Right now it covers some parts of AWS. Please have a look. Ideally, join the project.

Next Generation Shell – https://github.com/ilyash/ngs

Feedback

Do you agree that the jump between API and declarative tools was too big? Do you think that the middle ground, declarative primitives approach, would be useful in some cases? Comment here or on Reddit.


Have a nice day!

Why I have no favorite programming language

TL;DR – because for me there is no good programming language.

I’m doing mostly systems engineering tasks. I manage resources in Cloud and on Linux machines mostly. I can almost hear your neurons firing half a dozen names of programming languages. I do realize that they are used by many people for systems engineering tasks:

  • Go
  • Python
  • Ruby
  • Perl
  • bash

The purpose of this post is not to diminish the value of these languages; the purpose is to share why I don’t want to use any of the languages above when I write one of my systems-engineering-task scripts. My hope is that if my points resonate with you, the reader, you might want to help spread the word about or even help with my suggested solution described towards the end.

man-2756206_640

So let’s go over and see why I don’t pick one of the languages:

Why not language X?

All languages

  • Missing smart handling of exit codes of external processes. Example in bash: if test -f my_file (file is not there, exit code 1) vs if test --f my_file (syntax error, exit code 2). If you don’t spot the syntax error with your eyes, everything behaves as if the file does not exist.
  • Missing declarative primitives libraries (for Cloud resources and local resources such as files and users). Correction: maybe found one, in Perl – (R)?ex ; unfortunately it’s not clear from the documentation how close it is to my ideas.

All languages except bash

  • Inconvenient/verbose work with files and processes. Yes, there are libraries for that but there is no syntax for that, which would be much more convenient. Never seen something that could compare to my_process > my_file or echo my_flag > my_file .

Go

  • Compiled
  • Error handling is a must. When I write a small script, it’s more important for me for it to be concise than to handle all possible failures; in many cases I prefer an exception over twice-the-size script. I do understand how mandatory and explicit error handling can be a good thing for larger programs or programs with greater stability requirements.
  • Dependencies problem seem to be unresolved issue

Python

  • Functional programming is second level citizen. In particular list/dictionary comprehension is the Pythonic way while I prefer map and filter. Yes, that’s probably one of the features that makes Python easier to learn and suggested first language. Not everything that’s optimized for beginners must be good for more experienced users. It’s OK.
  • Mixed feelings about array[slice:syntax] . It’s helpful but slice:syntax is only applicable inside [ ] , in other places you must use slice(...) function to create the same slice object

Ruby and Perl

  • The Sigils syntax does not resonate with me.

Ruby

I can’t put my finger on something specific but Ruby does not feel right for me.

Perl

  • Contexts and automatic flattening of lists in some cases make the language more complicated than it should.
  • Object orientation is an afterthought.
  • Functions that return success status. I prefer exceptions. Not the default behaviour in Perl but an afterthought: autodie.
  • Overall syntax feeling (strictly matter of personal taste).

bash

Note that bash was created in a world that was vastly different from the world today: different needs, tasks, languages to take inspiration from.

  • Missing data structures (flat arrays and hashes is not nearly enough). jq is a workaround, not a solution in my eyes.
  • Awkward error handling with default of ignoring the errors completely (proved to be bad idea)
  • Expansion of undefined variable to empty string (proved to be bad idea)
  • -e ,  -u and other action at a distance options.
  • Unchecked facts but my feelings:
    • When bash was created, it was not assumed that bash will be used for any complex scripting.
    • bash was never “designed” as a language, started with simple commands execution and other features were just bolted on as time goes by while complete redesign and rewrite were off the table, presumably for compatibility.
  • Syntax
  • No widely used libraries (except few for init scripts) and no central code repository to search for modules (Correct me if I’m wrong here. I haven’t heard of these).

My suggested solution

I would like to fill the gap. We have systems-engineering-tasks oriented language: bash. We have quite a few modern programming languages. What we don’t have is a language that is both modern and systems-engineering-tasks oriented. That’s exactly what I’m working on: Next Generation Shell. NGS is a fully fledged programming language with domain specific syntax and features. NGS tries to avoid the issues listed above.

Expected questions and my answers

People work with existing languages and tools. Why do you need something else?

  • I assume I have lower bullshit tolerance than many others. Some people might consider it to be normal to build more and more workarounds (especially around anemic DSLs) where I say “fuck this tool, I would already finish the task without it (preferably using appropriate scripting language)”. I don’t blame other people for understandable desire to work with “standard” tools. I think it’s not worth it when the solutions become too convoluted.
  • I am technically able to write a new programming language that solves my problems better than other languages.

Another programming language? Really? We have plenty already.

  • I would like to remind you that most of the programming languages were born out of dissatisfaction with existing ones.
  • Do you assume that we are at global maximum regarding the languages that we have and no better language can be made?

Feedback

Would you use NGS? Which features it must have? What’s the best way to ease the adoption? Please comment here, on Reddit (/r/bash , /r/ProgrammingLanguages) or on Hacker News.


Update: following feedback roughly of the form “Yes, I get that but many Ops tasks are done using configuration management tools and tools like CloudFormation and Terraform. How NGS compares to these tools” – there will be a blog post comparing NGS to the mentioned tools. Stay tuned!


Have a nice day!

NGS unique features – Argv command line arguments builder

Background: what is NGS?

NGS LOGO

NGS, the Next Generation Shell is a (work in progress) shell and a programming language built ground up for systems engineering tasks. You can think of it as bash that’s designed today: sane syntax, data structures, functional programming, extensibility, cloud in mind, declarative primitives.

What’s the problem with constructing command line arguments?

The problem affects only more “advanced” cases of constructing command line arguments when some arguments might or might not be present. Let’s consider this example:

# Made-up syntax, resembling NGS
args = []
if 'Subnets' in props {
  args += '--subnets'
  args += props['Subnets']
}
if ... {
  args += ...
}
if ... {
  args += ...
}
...
aws elb create-load-balancer ... $args

Wouldn’t it be cleaner to get rid of all the ifs? … and what happens if props['Subnets'] is an empty array?

How Argv facility in NGS solves the problem?

Argv is a result of factoring out the common code bits involved in constructing command line arguments. The ifs above were also factored out. They are now in Argv.

Let’s look at usage example (real NGS code, from AWS library)

argv = Argv({
  '--load-balancer-name': rd.anchor.name
  '--listeners': props.ListenerDescriptions.encode_json()
  '--subnets': rd.opt_prop('Subnets', props).map(only(ResDef, ids))
})
rd.run('create ELB', %(aws elb create-load-balancer $*argv))

The important points here are:

  1. Argv is a function with a single parameter which must be of type Hash (also called “dictionary” in some languages)
  2. The keys of the Hash are switches’ names (--load-balancer-name, --listeners, --subnets)
  3. The values of the Hash are values for the switches

The “if” that decides whether a switch is present in the resulting argv is inside Argv implementation and your code is clean of it. The values of the Hash are considered when Argv decides whether a switch should be present. null, empty array and instances of type EmptyBox are considered by Argv as missing values and it discards the switch. For convenience, instances of type FullBox are unboxed when constructing the result of Argv.

The Argv facility is yet another point among others that shows why NGS and systems engineering tasks are best fit.


Have a nice weekend!

 

Why Next Generation Shell?

Background

I’m a systems engineer. The job that I’m doing is also called system, SRE, DevOps, production engineer, etc. I define my job as everything between “It works on my machine” of a developer and real life. Common tasks are setting up and maintaining cloud-based infrastructure: networking, compute, databases and other services. Other common tasks are setting up, configuring and maintaining everything inside a VM: disks+mounts, packages, configuration files, users, services. Additional aspects include monitoring, logging and graphing.

The problem

If we take specifically systems engineering tasks such as running a VM instance in a cloud, installing and running programs on a server and modifying configuration files, typical scripting (when no special tools are used) is done in either bash or Python/Ruby/Perl/Go.

Bash

The advantage of bash is that bash is domain specific. It is convenient for running external programs and files manipulation.

# Count lines in all *.c files in this directory and below
wc -l $(find . -name '*.c')

# Make sure my_file has the line my_content
echo my_content >my_file

# Run a process and capture the output
out=$(my_process)

The disadvantage of bash is horrible syntax, pitfalls everywhere, awkward error handling and many features one would expect from a programming language (such as data structures, named functions parameters, etc) are missing.

# Inconsistent, awkward syntax,
# result of keeping backwards compatibility
if something;    then ... fi
while something; do ... done

# Can remove / if MY_DIR is not defined
# unless in "set -u" mode
rm -rf "$MY_DIR/"

# Removes files "a" and "b" instead of "a b"
myfile="a b"
rm $myfile

# Silently ignores the error unless in "set -e" mode
my_script

# Function parameters can't be named, they are
# in $1, $2, ... or in $@ and $*
myfunc() {
  FILE="$1"
  OPTION_TO_ENABLE="$2"
  ...
}

Leave bash alone, it was not intended for programming, don’t do anything in bash, just use external programs for everything.

What do you observe? Is it or is it not used as a programming language in real life?

General-Purpose programming languages

Python/Ruby/Perl/Go are general-purpose programming languages.

The advantage of general-purpose programming languages is in their power, better syntax, ability to handle arbitrary data structures.

orig = [1,2,3]
doubled = [x*2 for x in orig]

The disadvantage of general-purpose programming languages is that they are not and can not be as convenient for systems engineering tasks because they are not focusing on this particular aspect of programming (in contrast to bash and other shells for example).

# Write whole file - too verbose
f = open('myfile', 'w+')
f.write('mycontent')
f.close()

# Run a process and capture the output
# https://docs.python.org/3.5/library/subprocess.html
proc = subprocess.Popen(...)
try:
    outs, errs = proc.communicate(timeout=15)
except TimeoutExpired:
    proc.kill()
    outs, errs = proc.communicate()

Summary

My conclusion is that there is no handy language for systems engineering tasks. On one hand there is bash that is domain specific but is not a good programming language and does not cover today’s needs, on the other hand there are general-purpose programming languages which do not specialize on this kinds of tasks.

You can use Puppet, Chef, Ansible, Terraform, CloudFormation, Capistrano and many other tools for common systems engineering tasks. What if your task is not covered by existing tools? Maybe one-off? Maybe a case where using one of the existing tools is not an optimal solution? You would like to write a script, right? In that case, your life sucks because scripting sucks. That’s because there is no convenient language and libraries to get systems engineering tasks done with minimal friction and effort.

Solution

I suggest, creating a new programming language (with a shell) which is domain specific, as bash, and which incorporates important features of general-purpose programming languages: data structures, exceptions, types, multiple dispatch.

My way of looking at it: imagine that bash was created today, taking into account today’s reality and things that became clear with time. Some of them are:

  • The shell is used as a programming language.
  • A system is usually a set of VMs and APIs, not a single machine.
  • Most APIs return JSON so data structures are needed as multiple jq calls are not convenient.
  • Silently ignoring errors proved to be bad strategy (hence set -e switch which tries to solve the problem).
  • Silently substituting undefined variables with empty strings proved to be bad strategy (hence set -u switch).
  • Expanding $x into multiple arguments proved to be error prone.
  • Syntax matters.
  • History entries without context have limited usefulness (cd $DIR for example: what was the current directory before cd and what was in $DIR ?)
  • UX
    • Spitting lots of text to a terminal is useless as it can not be processed by a human.
    • Feedback is important.
      • Exit code should be displayed by default.
      • An effort should be made to display status and progress of a process.
      • Ideally, something like pv should be integrated into the shell.

I’m not only suggesting the solution I’ve just described. I’m working on it. Please give it a try and/or join to help developing it: NGS – Next Generation Shell.

NGS LOGO

# Make sure my_file has the line my_content
echo my_content >my_file

# Run a process and capture the output
out=`my_process`

# Get process handle (used to access output, exit code, killing)
p=$(my_process)

# Get process output and parse it, getting structured data
amis=``aws ec2 describe-images --owner self``
echo(amis.len()) # number of amis, not lines in output

# Functional programming support
orig = [1,2,3]
doubled = orig.map(X*2)

# Function parameters can be named, have default values, etc
F myfunc(a,b=1,*args,**kwargs) {
  ...
}

# Create AWS VPC and Gateway (idempotent)
NGS_BUILD_CIDR = '192.168.120.0/24'
NGS_BUILD_TAGS = {'Name': 'ngs-build'}
vpc = AWS::Vpc(NGS_BUILD_TAGS).converge(CidrBlock=NGS_BUILD_CIDR, Tags=NGS_BUILD_TAGS)
gw  = AWS::Igw(Attachments=[{'VpcId': vpc}]).converge(Tags=NGS_BUILD_TAGS)

I don’t think scripting is the right approach.

It really depends on the task, constraints, your approach and available alternative solutions. I expect that situations needing scripting will be with us for a while.

Another programming language? Really? Why the world needs yet another programming language?

I agree that creating a new language needs justification because the effort that goes into creating a language and learning a language is considerable. Productivity gains of using the new language must outweigh the effort of learning and switching.

NGS creation is justified in exactly the same way as many other languages were justified: dissatisfaction with all existing programming languages when trying to solve specific problem or a set of similar problems. In case of NGS the dissatisfaction is specifically how existing programming languages address the systems engineering tasks niche. NGS addresses this particular niche with a unique combination of features and trade offs. Productivity of using NGS comes from best match between the tool and the problems being solved.

Yet another shell? We have plenty already but they all have serious adoption problems.

NGS will be implementing ideas which are not present in other shells. Hopefully, the advantages will be worthy enough to justify switching.

I’ll be just fine with bash/Python/Ruby/Perl/Go

You will. The decision to learn and use a new language depends on your circumstances: how many systems engineering tasks you are doing, how much you suffer, how much easier the tasks will become with NGS, how easily this can be done in your company / on your project and whether you are willing to take the risk.

You could just write a shell based on Ruby or Python or whatever, leveraging all the time and effort invested in existing language.

I could and I didn’t. Someone else did it for Python and for Scala (take a look, these are interesting projects).

  • I don’t think it’s the right solution to stretch existing language to become something else.
  • NGS has features that can not be implemented in a straightforward way as a library: special syntaxes for common tasks, multiple dispatch.

One could just write a library for Python or Ruby or whatever happens to be his/her favorite programming language, leveraging all the time and effort already invested in existing language.

In order to be similar to NGS, one would not only have to build a library but also change language syntax. I personally know only two languages that can do that: Lisp (using reader macros) and Perl6 (using grammar facility). These are general-purpose programming languages. Turning them into something NGS-like will be a significant effort, which I don’t think is justified.

PowerShell appears to be similar to what you describe here.

Note that I have very limited experience with PowerShell. The only aspect I definitely like is consistent usage of the $ sigil.

  • It’s probably a matter of taste and what you are accustomed to but I like NGS’ syntax more. PowerShell is pretty verbose.
  • DSC appears to be focused on resources inside a server/VM. NGS plans similar functionality. Meanwhile, NGS uses this approach in the AWS library: vpc = AWS::Vpc(NGS_BUILD_TAGS).converge(CidrBlock=NGS_BUILD_CIDR, Tags=NGS_BUILD_TAGS)

There are libraries for Python that make systems engineering tasks easier.

Right, sh for example. Such solution can’t be used as shell, it just improves the experience of calling external program from Python.


Was this post convincing? Anything is missing to convince you personally? Let me know!

Have a nice day!

NGS unique features – improving NodeJS require()

Background: what is NGS?

NGS, the Next Generation Shell is a (work in progress) shell and a programming language built ground up for systems engineering tasks. You can think of it as bash that’s designed today: sane syntax, data structures, functional programming, extensibility, cloud in mind, declarative primitives.

What’s good in NodeJS’ require()

I like most of how require() works in JavaScript. I’m not talking in this post about npm, just NodeJS require() function. require() does not pollute your namespace, you just get a reference, it’s simple to use and easy to reason about.

const a = require('cool-aws-wrapper');
// Can not be done easily with AWS SDK:
a.deleteRoute53Record('testing25.example.com');

What’s there to improve in require() ?

NodeJS modules are usually fall into one of the categories:

  1. Class definition / big library that manages it’s own namespace. These usually end with module.exports = MyClass. No problem here.
  2. Group of functions or classes. These usually end with module.exports = { func1, func2, func3, ...} lists (ES6 syntax, otherwise written as module.exports =  { func1: func1, ... } ) which I think are cumbersome.

How require() and modules look in NGS?

Note that require() in NGS is work in progress and it doesn’t have much of the functionality that NodeJS provides. I just started with things that bothered me the most.

Consistent with other places in NGS, require() returns the last evaluated expression. NodeJS for example returns module.exports which you must explicitly set as the result of require().

I think of modules primary as a namespaces. Creating a namespace in NGS has a syntax: ns { ... } .

Combining require() behaviour of returning last evaluated expression and namespace syntax, typical NGS module consists of single top level expression which evaluates to a namespace. The whole module file can look like this:

ns {

  global init

  type Vpc
  type Subnet

  F init(v:Vpc) {
    ...
  }

  F _helper_func(s:Str) { ... }

  MY_CONST = 42

  F ok() {
    echo("OK")
  }

}

Let’s ignore the global for now, it’s about how methods and types’ instances creation are implemented in NGS. Anything defined inside the ns { ... } is exposed as namespace member so usage of the above module could look like this:

{
  m = require('mymodule.ngs')
  vpc = m::Vpc()
  echo(m::MY_CONST)
  m::ok()
}

As you probably guessed, the :: operator is the namespace member access operator.

There is no need to explicitly state what module/namespace exports. That’s the improvement over NodeJS’ require().

How ns works and more options for the curios

ns { … } returns a Hash

As stolen from NodeJS, the namespace syntax (ns { ... }) returns a Hash. In NodeJS, require() typically returns JavaScript Object which is close enough for the purpose of this post.

About :: operator

The namespace member access operator :: is actually a Hash key access operator. It is helpful because the regular syntax for accessing members is not always a good fit for namespaces. The regular member access syntax is dot (.) but the dot syntax is also a function call: myobj.field – is a field/key/attribute access but myobj.func() is equivalent to func(myobj). For example, m::ok() will call the ok function defined in the module, m.ok() will call the function ok in current lexical environment with m as parameter.

As a bonus, since :: is an operator, it is implemented as function call. This means you can define how :: works with types that you define and modify how :: works with existing types.

ns { … } syntax implementation

For simplicity of implementation and absence of obvious reasons against, ns { ... } syntax is just a syntactic sugar for defining anonymous function without parameters and calling it immediately. The though behind this decision was simple: “I need to implement namespaces. Let’s see where I have them already. Oh, namespaces are already implemented in functions. This is so convenient, I can use this mechanism with minimal effort”.

How ns knows what to return?

ns is mostly a syntactic hack:

  1. Inside the ns body, the first statement, before any use-supplied statements is _exports = {} which sets the local variable _exports to an empty Hash.
  2. Any assignment and function definition also set _exports["something"]. MY_CONST = 42 becomes MY_CONST = 42;  _exports["MY_CONST"] = MY_CONST;
  3. Exception to the rule above are variables and functions with names starting with underscore (_). They are not automatically added to _exports. This for example is why _exports itself is not exported.
  4. Last statement, after all user-supplied statements is _exports.

The behavior I just described looks like sane defaults to me. As we all know, the life is usually more complex than hello world examples and customizations are need. Here are two ways to customize the resulting namespace.

  1. return your_expr – since ns is just a function, you can use return at any point to return your own custom namespace.
  2. manipulate _exports however you want towards the end of ns body. For example after _exports .= filterv(Type) only types will be exported. _exports.filterk(/^pub_/) will only export symbols (keys) that have names that start with pub_ .

Improvement suggestions are welcome! Have a nice day!

NGS unique features – Hash methods I wish I had in other languages

NGS is a language and a shell that I am building for systems administration tasks. Enough of the language is implemented to enable writing some useful scripts. The shell is not there yet.

Some of the Hash methods in NGS

The methods for working with Hash I have not seen all at once in other languages are:

  1. filterk – filter Hash by key (produces Hash)
  2. filterv – filter Hash by value (produces Hash)
  3. mapk – map Hash keys (produces Hash)
  4. mapv – map Hash values (produces Hash)
  5. mapkv – map Hash keys and values (produces Hash as opposed to map which produces an array)
  6. without – filters out specific key

How these are actually used? Following is an excerpt from the pollute method (function), which is a part of the AWS module. It uses several of the Hash methods I mentioned, making the method a good example. pollute method (as in “pollute global namespace”) enables using Vpc variable for example instead of AWS::Vpc and so on. I would like to have this behaviour for small quick-and-dirty scripts but not as default so it’s in a method that one can optionally call.

F pollute(do_warn=true) {

    vars =
        _exports.filterk(/^AMI_OWNER/) +
        _exports.filterv(Type).without('Res').without('ResDef') +
        ...

    if do_warn {
        warn("Polluting ...: ${vars.keys().join(', ')}")
    }

    vars.mapk(resolve_global_variable).each(set_global_variable)
}

Let’s go over the code above step by step:

F pollute(do_warn=true) { ... } defines the pollute method with optional parameter do_warn that has default value of true.

_exports is a Hash containing all of the AWS’ module variables and functions, similar to NodeJS module.exports but members are added automatically rather than explicitly. Only the methods and variables that do not start with _ (underscore) are added. One can modify _exports in any way before the end of the module. I will write more in detail about require() and modules in NGS in another post.

filterk(/^AMI_OWNER/) filters all the variables that match the given RegExp

filterv(Type) filters all the variables that are of type Type. These are AWS types’ definitions, such as Vpc, Subnet or Instance.

without('...') filters out the types I don’t like to override.

+ between and after _exports.filterk(...) and _exports.filterv(...) joins the hashes.

mapk translates variables’ names into their index (using resolve_global_variable)

each runs set_global_variable with variable index and the value to set

Hash methods in other languages

I am aware that some of the methods above are present in other languages or libraries. Some examples:

  1. Ruby has mapv (transform_values) method.
  2. Rails has mapk (transform_keys) and mapv.
  3. Perl 6 can modify values in a convenient manner:for %answers.values -> $v is rw { $v += 10 };.

What I have not seen is a language which has all the methods above out of the box. I have a feeling that arrays get all the fame methods while hashes (dictionaries) often get less attention in other languages.

Why NGS has all these methods?

NGS is aiming to be convenient for systems administration tasks. More often than not these tasks include data manipulation. NGS has many functions (methods) for data manipulation, including the ones listed in this post.

Update: reddit discussion


Have a nice day!