I’m curious about programming languages. Not because I’m creating one right now. I always was. This post is about ideas and features that I have seen in Perl 6 and found interesting. If you are curious about programming languages in general, you should take a look at these.
There are various reasons for not stealing the interesting ideas from Perl 6:
I’m trying to keep number of concepts in NGS as small as possible. If I’m not seeing huge immediate value in a concept – I skip it.
Not taking anything that I think can confuse me or other programmers. I’m not talking here because someone is a beginner. I’m talking about confusing concepts.
Simply because I don’t have enough resources to implement it at the moment.
Here are the interesting Perl 6 features, in no particular order (except the first one). There are also my comments whether I would like the feature in NGS or why not.
Syntax. Very expressive an terse. Perl6 has even more of it than Perl 5. Now that we got rid of the $ and friends in the room:
Grammars. Would actually be nice to have something like that in NGS.
Lots of operators. The most interesting concept is Metaoperators. I’m trying to keep the amount of syntax elements in NGS relatively low. There are already two syntaxes in NGS: commands and expressions. Not taking more syntax without serious need.
How the “pointy block” syntax mixes with “for” syntax: for @list -> @element . NGS already has several syntaxes for Lambdas.
“when” flow control. The closest NGS has is “cond” and friends, stolen from Lisp.
once . Not sure about this one. The functionality might be needed.
Slips. The behaviour is frightening me: if it does expand, how do I pass a Slip if I just want to pass it, say as an item of an array? NGS uses syntax for slips: [1, 2, *myitems, 3, 4] which I think is cleaner. You know you can’t pass it because it’s syntax.
.WHAT method. I stole something similar from Ruby: the inspect method.
As a special note, I have seen a welcome change from $arr to @arr . I think it removes confusion. (That was Perl 5 vs Perl 6).
Please don’t be offended if you are a Perl 6 hacker and you see that there is amazing feature that I have not mentioned. It could be that I’ve seen this in several other languages already or maybe I did not find it interesting or … maybe I just missed it. Don’t hesitate to leave a comment anyway.
Happy coding, in whatever language rocks your boat! Except for bash. Coding in bash will never be happy.
jq is a great tool. It does what bash can not – work with structured data. I use it. I would like not to use it.
In my opinion, working with structured data is such a basic thing that it makes much more sense to be handled by the language itself. I want my shell to be capable and I strongly disagree with the view that a shell “is not supposed to do that”. Shell is supposed to do whatever is needed to make my life easier. Handling structured data is one of these things.
If “shell is not supposed to do that”, by that logic, bash is not supposed to do anything except for running external commands and routing the data between them. Doesn’t it seem odd that bash does have builtin string manipulation then? Maybe bash shouldn’t have added associative arrays in version 4? … or arrays in version 2? How about if and while ? Maybe bash shouldn’t have them either?
jq is a symptom that bash can’t handle today’s reality: structured data. The world is increasingly more about APIs. APIs consume and return structured data. I do work with APIs from shell. Don’t you guys use AWS CLI or any other API that returns JSON?
The reality has changed. bash hasn’t. I’m working on bash alternative. Please help me with it. Or at least spread the word.
If you don’t like my project, join Elvish . Elvish is another shell that supports structured data.
I bet you’ve seen a lot of scripts with seemingly innocent if [ -e blah ];then ...; else ...; fi or something similar . What’s the problem? if has at most two branches while test , [ and [[ have three different exit codes. Oops.
If you make a syntax error (or any other error occurs) in the test , [ or [[ expression, it will return the exit code 2 (or above, according to man test). if will take the else branch. If you are lucky, you will notice the error message from the test, [ or [[ commands. If not, the else branch will always be executed.
I don’t want to use bash. The pitfall above is one of the many reasons. Unfortunately, I do use bash because it’s still best tool for some tasks. I’m working on alternative to bash. It’s called NGS, the Next Generation Shell. In NGS, the situation above is solved as one would expect from a modern programming language: exit codes 2 and above throw exception.
If you also think that there should be a viable alternative to bash, you are welcome to help me working on it.
I’ve got “protection” feature by default (and I didn’t notice I even had it up until now) from my internet provider, Bezeq International. In the last few days I was experiencing selective reachability. Some IPs were just blocked by the “protection”.
More than 20 minutes with support that wanted to install their binaries on my laptop (I couldn’t do it for many reasons) and then about 5 minutes with some more senior guy that after hearing the symptoms just turned that thing off. Everything works fine now.
Hope this helps other people so they could recognize the situation and immediately know what’s happening.
One of GitHub web IPs was blocked.
;; ANSWER SECTION:
static.xx.fbcdn.net. 3599 IN CNAME scontent.xx.fbcdn.net.
scontent.xx.fbcdn.net. 59 IN A 22.214.171.124
Broken AWS. Manifested in timeouts talking to various services endpoints.
I’m not saying that people that made Puppet and Ansible are not smart. It’s that we could learn from the mistakes they made… unless we don’t consider those being mistakes.
Puppet and Ansible went through very similar difficult situation. They have limited themselves to a declarative format and then they tried to accommodate the real life. Terraform has this situation right now.
The situation is:
Declarative format being used
People need something more powerful, like a programming language because … real life where conditionals, loops and data transformations make much more sense than working around declarative languages limitations.
Interestingly enough, they all did not switch to a proper programming language. Maybe because that would be at least partially admitting that the product should have been a library in the first place?
Terraform is actually in very crappy situation because even if they decide to expose everything as a library as the main interface, I don’t see people start using Go for “infrastructure as code”. Not as smooth as Ruby or Python anyway.
HCL to JSON one-to-one mapping. When I read “having a clean 1:1 mapping between HCL and JSON, and ensuring every feature of HCL is supported in JSON” I immediately thought that there must be converting tools then… and was not disappointed 🙂 “In future versions of Terraform, we will also support native tooling to convert HCL to JSON and JSON to HCL cleanly (including comments)”
Approach that in my eyes failed, again and again, is to start with your own declarative language and then with time grow the language. (SQL being among notable exceptions)
Puppet is the best example. map and each, added in Puppet 4.0.0 are, in my opinion, just two in a sea of evidence that the envisioned simple format has failed to handle the needs of the real world.
Ansible’s loop looks bad as the whole idea of making top levels of programs in YAML based syntax (and the rest in Python).
In my opinion, it makes more sense to create a language first and then libraries for it, not a library and then a language around it.
My hope for Terraform
I think Terraform guys are smart. Among other things, it manifests in implementing data sources. Data sources make Terraform much more flexible. I think it’s very clever.
Terraform, which started declarative, are now inventing their own programming language. They are going the way of Puppet and Ansible. I hope they can do better, in this awkward situation: there are quite a lot of constraints on the programming language because of the existing syntax and semantics.
I am considering making a JSON parsing and generating command line tool. Started with looking around a bit. Below is a list of existing JSON command line tools. Numbers are [GitHub stars] at the time of adding the entry.
jq  – filter, extract, modify and output JSON or text using DSL
jid  – “You can drill down JSON interactively by using filtering queries like jq.” (item contributed by /u/Tacticus)
gron  – convert JSON or JSON lines (from file/stdin/url) to text (path=value) which can be processed with grep/sed/diff; the tool also supports converting back to JSON after such processing
jo  – generate JSON based on command line arguments and stdin; can read data from files and place it as base64 encoded values
JSON.sh  – written in shell/gawk; “traverses the JSON objects and prints out the path to the current object (as a JSON array) and then the object, without whitespace”
jsawk  – focused primarily on filtering and transforming a list (or an object). Update 2019-09-30: as @joeytwiddle suggested in comment, the project appears to be unmaintained and doesn’t work with recent Node.js versions. Latest commit and latest closed issue are from 2015.
json (by trentm)  – “massaging JSON on your Unix command line”; JS-like syntax for extracting values; in-place file editing
jj  – “JJ is a command line utility that provides a fast and simple way to retrieve or update values from JSON documents. It’s powered by GJSON and SJSON under the hood.” (item contributed by /u/RomanaOswin, 2021-03-28) (also noticed it was mentioned earlier in comments by Harald Hanche-Olsen , sorry did not see that)
rq  – awk/sed-like tool for structured data; supports several formats, including JSON
dasel  – “Dasel (short for data-selector) allows you to query and modify data structures using selector strings. Comparable to jq / yq, but supports JSON, YAML, TOML, XML and CSV with zero runtime dependencies.” (item contributed by /u/AndydeCleyre, 2021-03-27)
TickTick  – use JSON syntax directly in bash; “This is just a fun hack”
jtc  “cli tool to extract, manipulate and transform source JSON” (item contributed by Dmitry in the comments, 2019, added 2021-03-28)
jshon  – very CLI-ish way to extract, manipulate and output the data
jl  – “a tiny functional language for querying and manipulating JSON”; visually reminds Haskell
faq . “faq is a tool intended to be a more flexible jq, supporting additional formats. The additional formats are converted into JSON and processed with libjq”. Supports: BSON, Bencode, JSON, TOML, XML, YAML. Added on 2020-10-11.
jsonv.sh  – convert JSON to CSV; specify paths in JSON to
jgrep (aka “JSON-grep”)  – “Command line tool and API for parsing JSON documents” in Ruby (item contributed by /u/tophlammiepie)
jello . “Filter JSON and JSON Lines data with Python syntax”. Added on 2020-10-11.
jsed  – manipulate and extract data; somewhat similar to jsawk in mindset
nvim-jqx  “easily browse and preview json files in neovim” using Quickfix buffer (item contributed by /u/evergreengt, 2021-03-28)
jtbl  “A simple cli tool to print JSON data as a table in the terminal.”. Added on 2020-10-11.
yamlpath  “Along with providing a standard for defining YAML Paths, this project aims to provide generally-useful command-line tools which implement YAML Paths.” (item contributed by /u/AndydeCleyre, 2021-03-27)
jayin  “Piping with js at terminal”. Added on 2019-09-30 following comment from @joeytwiddle.
jsongrep  (by dsc) – extract data at given path using shell globs and output one per line
jc  – “jc is used to JSONify the output of many standard linux cli tools”. Added on 2019-10-29 following comment from Kelly Brazil.
jsongrep  (by terrycojones) – easily extract data at given path
About to remove an empty directory? Use rmdir, not rm -rf. rmdir will not remove the directory if it’s not empty. Make it a habit. Thank me once every 2 to 20 years when you don’t remove your stuff by accident.
Marketing people know that. So when you get the impression that “everybody uses X”, please be aware that it can be intentional and maybe does not match the reality. It can be just a trick. Bloggers for example, have incentives to write about X (consultants that can make money if you adopt X).
I don’t want to accuse any specific firm or product in this post but I suspect the “coolest”, the most advertised and the most pushed down our throats products. There is no guarantee makers of X are interested in your success. They are surely interested in their own growth and success.
Hope this post increases your chances to survive next marketing attack just by being aware of yet another deceitful marketing technique.