Arguments and Parameters

These two words are used interchangeably. Please don’t. They mean different things. Here is my concise explanation.

Argument

A value passed into a function/method during invocation.

my_func(arg1, arg2)

Additional names for “argument” are “actual argument” and “actual parameter”.

Parameter

A name of a variable in the function/method definition. During invocation, the variable is used in the function/method body to refer to the value of the passed argument.

F my_func(param1, param2) {
  ...
  # Using param1 and param2 for a computation
  ...
}

Additional name for “parameter” is “formal argument”.

Tip – Parametrize

If you struggle to remember which one is which, this might help: when you “parameterize” a piece of code, you add parameters to the code. Then you have the code with the parameter used in it, with the first occurrence in the function/method definition.

# Initial version

echo("Hello, Joe")

# Parametrized version. "name" is a parameter.

F hello(name) {
  echo("Hello, ${name}")
}

See Also


Hope this helps! Have a nice day!


Updates after Reddit discussion:

  • I never asked the difference as an interview question. If I would:
    • Getting this wrong – tiny negative point
    • Not understanding why using correct terminology matters – big negative point
    • Understanding the difference and using these words interchangeably (knowingly incorrectly) – huge negative point
    • Providing fake facts to support your opinion that these words are interchangeable – huge negative point
  • Explaining why using correct terminology matters is out of scope of this post

Telegraph and the Unix Shell

Following is my opinion of the interactive mode of the Unix Shell. The interactive mode of the Unix shell is not much different from using Telegraph. I’ll substantiate the claim and point out what went wrong.

History

Short history of relevant inventions and events follows.

Telegraph – 1840s

Telegraph is a “point-to-point text messaging system”

Teleprinter – 1887

Teleprinter improved over telegraph’s user interface by adding a keyboard. Essentially it’s still a text messaging system.

“Computers used teleprinters for input and output from the early days of computing.”

Computer Terminal with VDU – 1950s

Computer Terminal with a “video display unit (VDU)” improved over teleprinter by replacing the printer with a screen.

( Summary Till this Point )

We see incremental progress in transmitting text. First between humans and later between a human and computer. The core concept did not change over time. Send text, receive text in response.

VT 52 – 1974/1975

VT 52 is released. It’s a pretty big deal because it supports cursor movement. Text on the screen can therefore be updated. In more general terms, it allowed interaction with what’s on the screen.

Bill Joy – vi – 1976

Bill Joy released vi in 1976. He figured that since cursor movement is supported he can make an editor that actually interactively uses the whole screen. That’s in contrast to the ex editor which didn’t.

What Went Wrong with the Shell?

The shell never caught up with the idea of interactivity on the screen. It’s still a “point-to-point text messaging system”. The paradigm shift never happened. Treating the received text as if it was printed on paper puts the idea of interacting with the output completely out of the realm of possibilities.

The “interactive” shell is not that much interactive. It is only “interactive” compared to “batch”. If you take a 24 lines terminal, considering that all interaction happens on one line (the “command line” I guess), that’s about 4% interactivity by line count.

Practical Consequences

Copy + Paste

Does the following sequence of operations happen to you when working in a shell?

  1. Type and run a command
  2. Start typing a second command
  3. Copy something from the output of the first command
  4. Paste that as an argument to the second command

I know that it does happen to me quite often. Why can’t the shell (tab) complete from the output of the first command? Because the shell has no idea (1) what’s in that output (2) what’s the semantic meaning of what’s in that output and how to use it for completion.

Imagine for a moment a website where you see a list of items but if you want more information about one of them, you need to copy its name and paste somewhere else, adding a word or two. Sounds weird? How is this OK in a shell then?

For the “shell is not supposed to do that” camp: you are welcome to continue to use Nano, Pico and Notepad for programming because “text editor is not supposed to do that”; I’ll use an IDE because it’s more productive.

Each Command on its own

Typically a shell user tries to achieve a goal. For that, typically a series of related commands. The Unix shell doesn’t know or care about that.

Let’s say you issued ls *.txt and you see the file that you were interested in in the output. You can’t interact with it despite being on the screen right in front of you so you proceed. Now you typed cp and pressed tab for completion. The completion will happily try to complete the names of all the files in the directory, not only *.txt which are way more likely to be meant by the user. There isn’t be any relation between sequentially issued commands in the Unix shell. Want composition? OK, use pipe or start scripting.


I might expand this article in the future.

The new Life of tap()

Background

I’m designing and implementing Next Generation Shell, a programming language (and a shell) for “DevOps” tasks (read: running external commands and data manipulation are frequent).

I came across a programming pattern (let’s call it P) as follows:

  1. An object is created
  2. Some operations are performed on the object
  3. The object is returned from a function (less frequently – stored in a variable)

P Using Plain Approach

The typical code for P looks in NGS like the following:

F my_func() {
  my_obj = MyType()
  my_obj.name = "blah"
  my_obj.my_method(...)
  my_obj  # last expression is evaluated and returned from my_func()
}

The above looks repetitive and not very elegant. Given the frequency of the pattern, I think it deserves some attention.

Attempt 1 – set()

In simpler but pretty common case when only assignment to fields is required after creating the object, one could use set() in NGS:

F my_func() {
  MyType().set(name = "blah")
}

or, for multiple fields:

F my_func() {
  MyType().set(
    name = "blah"
    field2 = 100
    field3 = "you get the idea"
  )
}

Side note: parameters to methods can be separated by commas or new lines, like in the example above.

I feel quite OK with the above but the cons are:

  1. Calling a method is not supported (unless that method returns the original object, in which case one could MyType().set(...).my_method())
  2. Setting of fields can not be interleaved in a straightforward manner with arbitrary code (for example to calculate the fields’ values)

Attempt 2 – tap()

I’m familiar with tap() from Ruby. It looked quite useful so NGS also had tap() for quite a while. Here is how P would look like in NGS when implemented with tap():


F my_func() {
  MyType().tap({
    A.name = "blah"
    A.my_method()
  })
}

Tap takes an arbitrary value, runs the given callback (passing that value as the only argument) and returns the original value. It is pretty flexible.

Can’t put my finger on what’s exactly is bothering me here but the fact is that I was not using tap() to implement P.

Attempt 3 – expr::{ … }

New Life of tap()

This one is very similar to tap() but it is syntactically distinct from tap.

F my_func() {
  MyType()::{
    A.name = "blah"
    # arbitrary code here
    A.my_method()
  }
}

I think the main advantage is that P is easily visually distinguishable. For example, if you only want to know the type of the expression returned, you can relatively easy skip everything between ::{ and } . Secondary advantage is that it’s a slightly less cluttered than tap().

Let’s get into the details of how the above works.

Syntax

  1. MyType() in our case is an expression. Happens to be a method call which returns a new object.
  2. :: – namespace field access operator. Typical use case is my_namespace::my_field.
  3. { ... } – anonymous function syntax. Equivalent to a function with three optional parameters (A, B, and C, all default to null).

Note that all three syntax elements above are not unique to this combination. Each one of them is being used in other circumstances too.

Up until recently, the :: syntax was not allowing anonymous function as the second argument. That went against NGS design: all methods should be able to handle as many types of arguments as possible. Certainly limiting arguments’ types syntactically was wrong for NGS.

Semantics

In NGS, any operator is transformed to a method call. :: is no exception. When e1::e2 is encountered, it is translated into a call to method :: with two arguments: e1 and e2.

NGS relies heavily on multiple dispatch. Let’s look at the appropriate definition of the :: method from the standard library:

F '::'(x, f:Fun) {
  f(x)
  x
}

Not surprisingly, the definition above is exactly like the definition of F tap() ... (sans method and parameters naming).

Examples of expr::{ … } from the Standard Library

# 1. Data is an array. Each element is augmented with _Region field.
data = cb(r)::{
  A._Region = ConstIter(r)
}


# 2. push() returns the original object, which is modified in { ... }
F push(s:Set, v) s::{ A.val[v] = true }


# 3. each() returns the original object.
# Since each() in { ... } would return the keys() and not the Set,
# we are working around that with s::{...}
F each(s:Set, cb:Fun) s::{ A.val.keys().each(cb) }


# 4. Return what c_kill() returns unless it's an error
F kill(pid:Int, sig:Int=SIGNALS.TERM) {
  c_kill(pid, sig)::{
    A == -1 throws KillFail("Failed to kill pid $pid with signal $sig")
    A != 0 throws Error("c_kill() did not return 0 or -1")
  }
}

Side note: the comments are for this post, standard library has more meaningful, higher level comments.

A Brother Looking for Use Cases

While changing syntax to allow anonymous function after ::, another change was also made: allow anonymous function after . so that one could write expr.{ my arbitrary code } . The whole expression returns what the arbitrary code returns. Unfortunately, I did not come across (or maybe haven’t noticed) real use cases. The appropriate . method in the standard library is defined as follows:

F .(x, f:Fun) f(x)

# Allows
echo(5.{ A * 2 })  # 10

Have any use cases which look less stupid than the above? Let me know.

The Original Sin in IT

You have a program with human readable output. Now a second program needs that information. There are two choices:

  1. Do the technically right and challenging thing – create a protocol and rewrite the first program and then write the second program (utilizing something like libxo in the first program maybe).
  2. Fuck everybody for the decades to come by deciding that parsing text is the way to go.

I would like to thank everybody involved in choosing number 2!

jc is an effort to fix (more work around) that decision. Thanks to the author and I hope it will make your DevOps life at least a bit more tolerable.

The Pseudo Narrow Waist in Unix

Background

This is a pain-driven response to post about Narrow Waist of Unix Architecture. If you have the time, please read that post.

The (very simplified and rough) TL;DR of the above link:

  1. The Internet has “Narrow Waist”, the IP protocol. Anything that is above that layer (TCP, HTTP, etc), does not need to be concerned with lower level protocols. Each piece of software therefore does not need to concern itself with any specifics of how the data is transferred.
  2. Unix has “Narrow Waist” which is text-based formats. You have a plethora of tools that work with text. On one side of of Narrow Waist we have different formats, on another side text manipulating tools.

I agree with both points. I disagree with implied greatness of the Unix “design” in this regard. I got the impression that my thoughts in this post are likely to be addressed by next oilshell blog posts but nevertheless…

Formats

Like hierarchy of types, we have hierarchy formats. Bytes is the lowest level.

Bytes

Everything in Unix is Bytes. Like in programming languages, if you know the base type, you have a certain set of operations available to you. In case of Bytes in Unix, that would be cp, zip, rsync, dd, xxd and quite a few others.

Text

A sub-type (a more specific type) of Bytes would be Text. Again, like in a programming language, if you know that your are dealing with data of a more specific type, you have more operations available to you. In case of Text in Unix it would be: wc, tr, sed, grep, diff, patch, text editors, etc.

X

For the purposes of this discussion X is a sub-type of Text. CSV or JSON or a program text, etc.

Is JSON a sub-type of Text? Yes, in the same sense that a cell phone is a communication device, a cow is an animal, and a car is a transportation device. Exercise to the reader: are this useful abstractions?

Cow is an animal

The Text Hell

The typical Unix shell approach for working with X are the following steps:

  1. Use Text tools (because they are there and you are proficient wielder)
  2. One of:
    1. Add a bunch of fragile code to bring Text tools to level where they understand enough of X (in some cases despite existing command line tools that deal specifically with X)
    2. Write your own set of tools to deal with the relevant subset of X that you have.
  3. Optional but likely: suffer fixing and extending number 2 for each new “corner case”.

The exception here are tools like jq and jc which continue gaining in popularity (for a good reason in my opinion). Yes, I am happy to see declining number of “use sed” recommendations when dealing with JSON or XML.

Interestingly enough, if a programmer would perform the above mentioned atrocities in almost any programming language today, that person would be pointed out that it’s not the way and libraries should be used and “stop using square peg for round hole”. After few times of unjustified repetition of the same offense, that person should be fired.

Square peg / round hole

Somehow this archaic “Unix is great, we love POSIX, we love Text” approach is still acceptable…

Pipes Text Hell

  1. Create a pipe between different programs (text output becomes text input of the next program)
  2. Use a bunch of fragile code to transform between what first program produces and the second one consumes.

Where Text Abstraction is not Useful

Everywhere almost. In order to do some of the most meaningful/high-level operations on the data, you can’t ignore it’s X and just work like it is Text.

Editing

The original post says that since the format is Text, you can use vim to edit it. Yes you can… but did you notice that any self respecting text editor comes with plugins for various X’s? Why is that? Because even the amount of useful “text editing” is limited when all you know you are dealing with Text. You need plugins for semantic understanding of X in order to be more productive.

Wanna edit CSV in a text editor without CSV plugin? OK. I prefer spreadsheet software though.

Have you noticed that most developers use IDEs that “understand” the code and not Notepad?

Lines Count

Simple, right? wc -l my.csv. Do you know the embedded text in quotes does not have newlines? Oops. Does it have header line? Oops.

Text Replacement

Want to try to rename a method in a Java program? sed -i 's/my_method/our_method/g' *.java, right? Well, depends on your luck. I would highly recommend to do such kind of refactoring using an IDE that actually understands Java so that you rename: only specific method in a specific class as opposed to unfortunately named methods and variables, not to mention arbitrary strings.

Search / Indexing

Yep… except that understanding of the semantics helps here quite a bit. That’s why you have utilities which understand specific programming languages that do the indexing.

Conclusion

I do not understand the fascination with text. Still waiting for any convincing arguments why is it so “great” and why the interoperability that it provides is not largely a myth. Having a set of tools enabling one to do subpar job each time is better than not having them but is it the best we can?

My previous dream of eradicating text where it does not make sense (my blog post from 2009) came true with HTTP/2. Apparently I’m not alone in this regard.

Sorry if anything here was harsh. It’s years of pain.

Clarification – Layering

Added: 2022-02-07 (answering, I hope, https://www.reddit.com/r/ProgrammingLanguages/comments/t2bmf2/comment/hzm7n44/)

Layering in case of IP protocol works just fine. Implementer of HTTP server really does not care about the low level transport details such as Ethernet. Also the low level drivers don’t care which exactly data they deliver. Both sides of the Waist don’t care about each other. This works great!

My claim is that in case of the Text Narrow Waist, where X is on one hand of and the Text tools are on the other, there are two options:

  1. Tools ignore X and you have very limited functionality you get out of the tools.
  2. Tools know about X but then it’s “leaky abstraction” and not exactly a Narrow Waist.

That’s why I think that in case of Text, the Narrow Waist is more of an illusion.


Have a nice week!

On Accidental Serialization Formats

Let’s talk about the “just separate with comma and stick it into one field” type of serialization.

You had two strings (abc and def) and you joined them with a separator. What do you have now? One string with two elements, right? Right, abc,def. Well… two or more actually, depending on how many times the chosen separator occurred in the original strings: if they were a,bc and def, you’ve got a,bc,def, which is 3 elements according to our format. Oops. Leaving out the question whether leading and trailing spaces are significant.

Wanna add escaping for the separator then? a,bc and def are now serialized as a\,bc,def. Now the parsing became more complex. You can’t just split the string by the separator (you would get 3 elements: a\ and bc and def. You need to scan the serialized data, considering escaping when splitting. You also need to remove the escaping character from the result. How about escaping the escape character? If original data is a\bc, it is serialized as a\\bc). Yet another something not to forget.

Don’t like escaping then? How about encoding like in URL? a,bc becomes a%2Cbc. You can now once again split the string by the separator character… assuming it was encoded. Which characters you encode anyway? If you encode all ASCII characters, the result is 3 times the original and is completely unreadable. It least you are “safe” with regards to separator now, it is encoded for sure so no split problems. You have to add a decoding routine now though.

If your serialized thing goes into a database, consider how indexing would work. It probably won’t. Maybe you should model your domain properly in the database and not serialize at all. Hint: if the values ever need to be treated differently/separately by the database, they go into different cells/rows/columns/fields, not one. There are very rare exceptions. Notable exception is the ability of databases to handle JSON fields (examples: MySQL, PostgreSQL). Note that this capability can fit or not fit your use case.

Want to satisfy your artistic needs and do something clever about the serialization? Do it at home then please. Don’t waste time that your colleagues could use on something more productive than dealing with your custom format.

Strong advice: don’t do custom serialization format, use existing serialization formats and libraries.


Seen something to add to the above? Leave a comment!

API-less

I was always complaining about how AWS made APIs that are inconsistent in so many dimensions. AWS teams can’t even (read “don’t care”) to agree on how to name pagination related fields.

Because I am such who was that again? an important person, I hope at least someone everybody in AWS have read my blog and they are checking by mistake it at least twice a day for new stupid rants extreme wisdom.

One of these people said “Fook you, Ilya” and made a service without API. Not kidding. Meet AWS Control Tower. No API, no CLI, no CloudFormation. “Take that, Ilya!”

For some balance I will mention that other people in AWS are trying to do some uniform API. Not stellar at the moment but hey, at least they are trying.

Have a nice $(curl https://api.timeofday.example.com/) !

AWS Cloud Control is in for Review

Since 2016 I have worked on AWS library. On 2021-09-30, AWS released something comparable, AWS Cloud Control API. If it’s comparable, let’s compare. I hope the reader will find this unique perspective interesting.

Purpose

AWS Cloud Control provides uniform API to Create, Read, Update, Delete and List different resources. In short, CRUDL.

My library aims to provide uniform simplified declarative interface where the main functionality is to converge to desired state.

Supported Resources

Cloud Control supports much more resources. My library supports very few, the ones that I needed the most for my work. It’s just how experimental library by one person in spare time compares with a product from AWS.

Mode of Operation

Cloud Control

Operates on one resource at a time.

Cloud Control provides relatively low level API, CRUDL. The advantage of Cloud Control over existing APIs is the uniformity, not achieving a desired state. You either create or update, and you need to know which one you are doing. (Deletion is a different story).

I am amazed that somebody at AWS was able to pull this off… where different services don’t agree on naming conventions for tagging resources, how and what can be tagged, and don’t have a convention for naming pagination fields. (Which by the way is nightmare for the users but typically abstracted by Terraform or CloudFormation). Sorry for the rant.

The operations are performed asynchronously. You have an API to query the status: GetResourceRequestStatus. This approach is better for advanced use cases, where multiple resources are being created simultaneously. (Like CloudFormation?)

The operations are idempotent. This is achieved using client token, as in other AWS services.

My Library

Operates either on one resource or a set of resources of the same type at a time. Edit: … from default, specified , or all regions.

The library looks up the resources and either creates or updates them, depending on whether they exist or not.

The operations are performed synchronously. The library throws exception if anything goes wrong. I find this approach much more user friendly for basic use cases.

Idempotency is out of the picture. Should it be in? Probably. Looks like omission on my side. How to best shield the user of the library from this issue? Don’t know yet. Need to think.

Describing Desired State

Cloud Control

The “create” API has desired state parameter: DesiredState.

The “update” API doesn’t, which I find very strange. I thought that desired state is something to be achieved no matter what the current state is. Desired state is only used in the “create” API so from my perspective it could have been called “properties” or whatever.

The “update” API has PatchDocument parameter, which I find not user friendly for the following reasons:

  1. Who actually works with JSON patch format? It’s the first time I see that I need to use it.
  2. I think it is less mentally challenging to describe … desired state and not the delta. This is typically done by IaC tools, including CloudFormation: calculate the diff between current and desired state for the user, so that the user would not need to get into this.
  3. It makes the update inconsistent with create.

My Library

There is no separate create and update. The user specifies the desired state as a parameter to the “converge” function. Converge then either creates or updates the resource / resources. The (non existent) “create” and “update” are therefore completely uniform in the library.

Search / Filtering

Cloud Control

Search is not supported. In practice it means listing all the resources and filtering the result on the client side. Typically that can be done in AWS CLI with the --query flag which is supported globally, for any AWS CLI command. Unfortunately I don’t see a way to make it work in this situation. The returned result has ResourceDescriptions field, an array where each item has Properties field, the Properties field is a string (JSON). Apparently JMESPath does not support parsing JSON in this situation. This means that the output of Cloud Control AWS CLI will be piped to jq or maybe a programming language for filtering and/or further processing.

My Library

While the number of supported resources and filters is low, the library supports filtering. The filtering is done on the server or on the client, completely transparent to the users of the library. What’s done on the server and what’s on the client? Simple – when filtering a given property is supported on the server side, it’s done there.

Desired State Format

Cloud Control

Cloud Control uses CloudFormation syntax. This makes sense.

My Library

My library uses the same format as you would see when using AWS CLI to describe the resource. It allows access to properties that CloudFormation does not have, and is unlikely to have, so for example this works:

  1. instance = AWS::Instance(...).converge(State = 'running') — which creates or turns on the specified EC2 instance / instances. Turning on or off EC2 instances is not supported in CloudFormation.
  2. vpc = AWS::Vpc(IsDefault=true).expect(1) — get a reference to default VPC (to use in further operations).

Closing Thoughts

  1. Cloud Control looks like a step in the right direction.
  2. Having Properties as a string is a major ergonomic issue.
  3. JSON patch for update is a huge ergonomic issue.
  4. Search (filtering in List) functionality is missing.
  5. “Desired state” naming is unjustified.

Cloud Control is a big effort, therefore let’s give the team some slack and see how the API is improved with time? Hopefully soon 🙂


Have a nice day, evening, or night! Mornings are just statistically less nice…

Failed Stealing from Python

I made a mistake. Hope you will learn something from it.

Mental Shortcuts

Heuristic, tl;dr for the purposes of this article – mental shortcut. The brain chooses to do less thinking to save energy. It relies on simple rules to get the result. The result is correct… some times.

I took a mental shortcut when working on my own programming language, Next Generation Shell. It was a mistake.

Additionally, I have ignored the uneasy but very vague feeling that what I’m doing is not 100% correct. From my experience I knew I shouldn’t ignore it but I still did it. Another mistake.

I “thought”

Below are heuristics that led to the wrong decision.

Copying features from popular languages is pretty “safe”. After all, “everybody” is using the language and it’s OK. Social proof. Wrong. Everybody does mistakes. Popular languages have design issues too.

It’s OK to copy a feature because it’s very basic aspect of a language. Nope. Python messed up arguments passing. And I copied that mess.

The Fix

Python 3.8 has the fix. I have mixed feelings about it. Still not sure how I should fix it in NGS.

Takeaway

Beware of mental shortcuts. There are situations where these are not acceptable. The tricky part is to detect that you are using a mental shortcut in a situation where it’s not appropriate. I hope that with awareness and practice we can do it.

Also note that your $job is most likely paying you to not take mental shortcuts.

The Web vs Unix

I would like to share my perspective on what’s wrong with Unix. This time by comparing and contrasting it to the Web.

User Agent

With some amount of stretch, one could say that the equivalent of the browser (user agent) would be a shell with a terminal. You interact with it and it does things for you, like the browser.

User Interface

The web browser is capable of rendering textual and graphical content. Unix shell relies on the terminal (usually emulator mimicking decades old hardware) for user interface and in most cases is limited to fixed width font text.

The main difference is how you can interact with the content. In the shell – you can’t. The shell is out of the game. The content that you see on your screen goes from a program and straight into the terminal, bypassing the shell. Compared to a browser, that sounds insane: you are just unable to interact with the content that you see. Ironically enough, this is happening in what’s called “interactive shell“. Some terminals match the text with predefined set of patterns and allow some minimal interaction such as ability to click on a link to open it in a browser.

The browser is a strict superset when we look at interaction capabilities: you can type in and you can interact with the objects on the screen by clicking them. What an amazing concept! Maybe some day shells will be able to that too! Meanwhile, in the shell, you type your commands and get a dump of text back, with rare exceptions.

I would summarize that the shell is a shitty user agent. 💩

I already can hear the coming “shell is not supposed to do it” argument. My opinion: shell is supposed to do whatever is needed for me to be productive. If it’s your “Unix Philosophy” vs me being productive then you can continue to use Notepad (or ed, the standard text editor, for that matter) and I would be using an IDE, OK?

Layout Engine

They also call it browser engine. That’s because on the web it’s in the browser. But where is it on Unix? Everywhere. Yes, the “Make each program do one thing well” is out of the window long ago. Each program does (hopefully) one thing and then it also does the layout of the output.

Each program has the following main options for handling the input/output:

  1. Primitive output – the program dumps some text on standard output. Let’s include colored text here. It’s just some additional color codes. This is equivalent to not having a layout engine. Sample program in this category: grep.
  2. Interactive UI – the program uses ncurses or similar library. It’s relatively small number of programs.
  3. Layout engine – the program contains some form of a layout engine. This is pretty common. Sample programs in this category: ls, ps, top, diff (columns output), wc, …

Common issues with the “layout engines” causing unpleasant broken view in Unix include:

  1. Improper handling of data which is wider than some hard coded fixed value
  2. Improper handling of Unicode
  3. Failure to accommodate for “unexpected” terminal escape codes in the input (which after processing find their way to the output in utilities like sed)

TCP/IP

Pipe

Let’s talk about pipes. Before everybody gets offended and says pipes is the sacred cow best feature of Unix. Yes, it probably is.

Pipes would roughly correspond to the TCP/IP protocols.  Pipes deliver data. For now, let’s leave alone the fact that they are unidirectional as opposed to TCP, which is bidirectional.

Since the web is a stack of protocols, the obvious question would be how other parts of the stack correspond? Read on.

HTTP

HTTP would correspond to text. Well, mostly text. Sometimes null character separated records. Sometimes something else. That’s the standard “format” to communicate between Unix applications.

“Write programs to handle text streams, because that is a universal interface.” – Basics of the Unix Philosophy.

The original claim is that text is the best for interoperability: large number of utilities have text as input and also output text, manipulating it in many different ways in between. Sounds like a dream. Except in reality this dream turned out to be a nightmare.

Incompatible, ad-hoc formats

Text on Unix is not a single format. It’s a bunch of ad-hoc formats, typically incompatible between different programs. That’s why we have a variety of tools such as sed, cut, awk and alikes. Here is my hot take: these tools are not solutions, they are workarounds. When you don’t have a protocol to communicate between applications, you need a bunch of adapters. Like Sisyphus, one need to write these adapters. All the time. Forever. Text parsing and manipulation feels like core part of Unix. From my perspective it’s an accidental complexity.

On a philosophical note: the “universal interface” should have been a stream of bytes or maybe even bits. I guess it was not found to be very useful. Apparently if you add a line separator character it is good enough to become a recommendation. But why stop there? Maybe add more structure? Maybe accommodate the fact that most of the data is either records with named fields or tables with named columns? Are you sure you counted the columns right for your awk '{print $8}' ?

This is in contrast to HTTP which is spoken by everyone on the Web.

Some Hope

Newer CLIs do usually have an option to output JSON or (less prevalent) YAML. They are forming a new ecosystem with different set of tools. From my perspective, it is proving the point that the “universal interface” might not be that universal and not as productive as envisioned (should I dare and say “unacceptable”?) .

Should it be the half-way structured, aka semi-structured JSON? Is it the sweet spot? I mean why stop here? Maybe we need something with schema? Let me know what you think.

One of the notable projects, jc, is an adapter between the “universal interface” and something that you would actually like to work with.

Shameless Plug

If only we could have a shell that could play a role in this new ecosystem… or maybe even push it a bit in the direction of having semantically meaningful objects on the screen so that interaction would be possible…

Yes, I am aware of other projects solving the same issue. While we mostly agree on the problem, I haven’t yet seen a project which sees the solution the same way as I do.


Happy, DevOps-ing!