The new Life of tap()


I’m designing and implementing Next Generation Shell, a programming language (and a shell) for “DevOps” tasks (read: running external commands and data manipulation are frequent).

I came across a programming pattern (let’s call it P) as follows:

  1. An object is created
  2. Some operations are performed on the object
  3. The object is returned from a function (less frequently – stored in a variable)

P Using Plain Approach

The typical code for P looks in NGS like the following:

F my_func() {
  my_obj = MyType() = "blah"
  my_obj  # last expression is evaluated and returned from my_func()

The above looks repetitive and not very elegant. Given the frequency of the pattern, I think it deserves some attention.

Attempt 1 – set()

In simpler but pretty common case when only assignment to fields is required after creating the object, one could use set() in NGS:

F my_func() {
  MyType().set(name = "blah")

or, for multiple fields:

F my_func() {
    name = "blah"
    field2 = 100
    field3 = "you get the idea"

Side note: parameters to methods can be separated by commas or new lines, like in the example above.

I feel quite OK with the above but the cons are:

  1. Calling a method is not supported (unless that method returns the original object, in which case one could MyType().set(...).my_method())
  2. Setting of fields can not be interleaved in a straightforward manner with arbitrary code (for example to calculate the fields’ values)

Attempt 2 – tap()

I’m familiar with tap() from Ruby. It looked quite useful so NGS also had tap() for quite a while. Here is how P would look like in NGS when implemented with tap():

F my_func() {
  MyType().tap({ = "blah"

Tap takes an arbitrary value, runs the given callback (passing that value as the only argument) and returns the original value. It is pretty flexible.

Can’t put my finger on what’s exactly is bothering me here but the fact is that I was not using tap() to implement P.

Attempt 3 – expr::{ … }

New Life of tap()

This one is very similar to tap() but it is syntactically distinct from tap.

F my_func() {
  MyType()::{ = "blah"
    # arbitrary code here

I think the main advantage is that P is easily visually distinguishable. For example, if you only want to know the type of the expression returned, you can relatively easy skip everything between ::{ and } . Secondary advantage is that it’s a slightly less cluttered than tap().

Let’s get into the details of how the above works.


  1. MyType() in our case is an expression. Happens to be a method call which returns a new object.
  2. :: – namespace field access operator. Typical use case is my_namespace::my_field.
  3. { ... } – anonymous function syntax. Equivalent to a function with three optional parameters (A, B, and C, all default to null).

Note that all three syntax elements above are not unique to this combination. Each one of them is being used in other circumstances too.

Up until recently, the :: syntax was not allowing anonymous function as the second argument. That went against NGS design: all methods should be able to handle as many types of arguments as possible. Certainly limiting arguments’ types syntactically was wrong for NGS.


In NGS, any operator is transformed to a method call. :: is no exception. When e1::e2 is encountered, it is translated into a call to method :: with two arguments: e1 and e2.

NGS relies heavily on multiple dispatch. Let’s look at the appropriate definition of the :: method from the standard library:

F '::'(x, f:Fun) {

Not surprisingly, the definition above is exactly like the definition of F tap() ... (sans method and parameters naming).

Examples of expr::{ … } from the Standard Library

# 1. Data is an array. Each element is augmented with _Region field.
data = cb(r)::{
  A._Region = ConstIter(r)

# 2. push() returns the original object, which is modified in { ... }
F push(s:Set, v) s::{ A.val[v] = true }

# 3. each() returns the original object.
# Since each() in { ... } would return the keys() and not the Set,
# we are working around that with s::{...}
F each(s:Set, cb:Fun) s::{ A.val.keys().each(cb) }

# 4. Return what c_kill() returns unless it's an error
F kill(pid:Int, sig:Int=SIGNALS.TERM) {
  c_kill(pid, sig)::{
    A == -1 throws KillFail("Failed to kill pid $pid with signal $sig")
    A != 0 throws Error("c_kill() did not return 0 or -1")

Side note: the comments are for this post, standard library has more meaningful, higher level comments.

A Brother Looking for Use Cases

While changing syntax to allow anonymous function after ::, another change was also made: allow anonymous function after . so that one could write expr.{ my arbitrary code } . The whole expression returns what the arbitrary code returns. Unfortunately, I did not come across (or maybe haven’t noticed) real use cases. The appropriate . method in the standard library is defined as follows:

F .(x, f:Fun) f(x)

# Allows
echo(5.{ A * 2 })  # 10

Have any use cases which look less stupid than the above? Let me know.

bash or Python? The Square Pegs and a Round Hole Situation

The question “should I do it in bash or in Python?” is both frustrating and common. Why one even needs to choose between two alternatives which are both inadequate for the task at hand? Why try to pick one of the square pegs for the round hole? I believe that you should not be in this annoying situation when just trying to write a script and get back to your endless stream of other todos.

Best illustration that I managed to find in 2 minutes

Both are Inadequate for Ops

bash does not meet any modern expectations for syntax, error handling nor has ability to work with structured data (beyond arrays and associative arrays which can not be nested). Let it go. You are not usually coding in assembly, FORTRAN, C, or C++, do you? They just don’t match the typical Ops tasks. Don’t make your life harder than it should be. Let it go. (Let’s not make it a blanket statement. Use your own judgement when to make an exception).

Python along with many other languages are general purpose programming languages which were not intended to solve specifically Ops problems. The consequence is longer and less readable scripts when dealing with files or running external programs, which are both pretty common for Ops. For example, try to check every status code of every program you run, see how your code looks like. Sure you can import 3rd party library for that. Is that as convenient as having automatic checking by default + list of known programs which don’t return zero + convenient syntax for specifying/overriding expected exit code? I guess not.

Your disorientation and frustration is completely legitimate.


Multitude of attempts to provide viable alternatives by different people are in progress. As others authors, I would like to help my Ops colleagues to avoid frustration and be productive. It just feels good.

Informative section is over. Shameless plug about an alternative that I am developing and how it is special follows.

Next Generation Shell as an Alternative

How Next Generation Shell differs from the alternatives? Reasonable question that I would ask too before investing any more time if I was the reader.


Current shells as well as proposed alternatives treat UI as nothing has happened since the 70-s: mostly typing commands and getting some text back.

How about real interaction with the objects on the screen? Oops. In a typical shell there are no objects on the screen, it’s just a dumpster of combined text (if you are lucky; could be binary) from stdout and from stderr from one or more processes (unless steps were taken). WTF? It doesn’t help you to win. It helps you to lose time. NGS does what is intended to make you productive. I have organized my thoughts about how the UI should look and behave on the wiki page.

Programming language

It looks like alternative solutions have the “let’s make the shell better” approach and therefore are heavily based on the shell syntax and paradigms.

There is also “let’s make a library for existing language” approach which doesn’t hit the target either. Can’t have a syntax for common ops tasks for example.

And finally there is “let’s make a library and a syntax on top of existing language”. Not sure about this one. Sounds good in theory. Looked some time ago at something like this and the overall impression was … awkward (for the lack of better word).

Approach in NGS: let’s make a good programming language for Ops, which fits the use cases and has syntax and facilities for the most common tasks such as running external programs.

The language follows the principle that the most common tasks should have their own syntax or a library function (depending on usage frequency). Examples:

  1. ``external program`` – runs external program and parses the output (JSON is auto detected, easily extensible for anything else).
  2. status(), log(), debug(), retry() – standard library functions. How many times an Ops person should write his/her own retry()? It’s insane.
  3. Argv() facility for constructing command line parameters (for calling external program).
  4. p=$(my_prog my_args &); ....; p.wait().

Small number of “big” core concepts in the language (types with inheritance, multiple dispatch and exceptions).


Most of the standard library is in NGS.

The UI (only recently started working on it) is in NGS. It doesn’t make sense that when a user of the shell wants to fix a bug in the UI and suddenly he/she needs to learn Go, Rust, C or whatever other language.

We use NGS at work.

Most of the demo scripts come from either current or previous work.

How to proceed?

  1. Install NGS.
  2. Consult documentation and look at sample scripts
  3. Write scripts for non-production-critical tasks.
  4. I am here to help. Do not hesitate to contact me with questions, suggestions, or feedback. If there is anything Ops-y you are trying to do seems to be easier in bash or Python – open an issue, because that’s a bug from my perspective. Something is inconvenient? Yep, also a bug.

If you are like me, you will find at least some satisfaction in using the most appropriate tool before continue to the myriad of other tasks that are in your todo queue.

Or Just Learn More

  1. Is NGS for you? Take a look at intended use cases.
  2. Take a look at how NGS compares to other programming languages.
  3. Browse sample scripts to get some impression about the language.