According to DiSC, about a quarter of all people should be communicating like me. We want information, not fluff or stories. We are here to get the answer to our question: “what’s X?” (a technology, a format, a piece of software, etc). Yet, the number of blog posts which answer “What’s X?” concisely is roughly zero. I am going to fix this with my future posts as time allows. Stay tuned.
Everything below is implementation detail. You can stop reading here and save a few minutes.
Here is my plan. Feel free to use it as a guideline for your blog posts too.
Discretion
We are looking for the author’s discretion about what’s important (hint: typically concepts and architecture), not a dump of everything that the author knows about the topic. We are here for “I would have written a shorter letter, but I did not have the time“. Yes, that 5 minutes read should take hours if not days to write. Otherwise, what’s the value?
Can’t answer the question concisely? We doubt your understanding of the topic.
Context
“Context is important, the blog post must provide context, blah blah …”.
The context was already established by searching “what’s X” or following a link to the post. It’s annoying when the text starts with fluff, keeping us wondering when and whether my question will be answered.
If there is some *really* important context, that’s not 5 paragraphs. Sorry, storytellers.
Show why the Topic is Important
“You need to show why X is important and where it’s used”.
This information is available in every other blog post and/or on Wikipedia and/or is one search away.
Underlying Concepts
Don’t explain the underlying concepts, link to them (like DiSC above). Don’t waste our time if we already know that.
The aim of the post is to give a simple, concise, and high level description of the event loop. I’m intentionally leaving out many details and being somewhat imprecise.
If you need detailed picture, follow the links in this post. In this case, I recommend reading this short post first.
Why Event Loop?
To deal with concurrency (multiple “things” need to be happening at what looks like the same time), there are few models. To discuss the models, we need to know what a thread is.
Roughly, a thread is a sequence of computing instructions that runs on a single CPU core.
Roughly, Event Loop manages on which tasks the thread works and in which order.
Queue
The queue contains what’s called in different sources messages, events, and tasks.
Message/event/task is a reference to a piece of code that needs to be scheduled to run. Example: “the code responsible for handling the click of button B + full details of the click event to pass to the code”.
Event Loop
The queue is checked for new tasks. If there is none, we wait for one to appear.
The first task in the queue is scheduled and starts running. The code runs till completion. In many languages, await keyword in the code counts as completion and everything after await is scheduled to run later – new task in the queue.
Repeat from number 1. That’s the loop. It’s called Event Loop because it processes events from the queue in a loop.
Adding Events to the Queue
Tasks are added to the queue for two reasons:
Something happened (user clicked on a button, network packet arrived, etc).
The code that was running in step 2 of the event loop scheduled something to run later.
You use AWS CDK. It’s great. It does a lot for you. Then one day something goes wrong. OK, it didn’t happen yet. But you want to be prepared for that (at least to some extent). The following information is what I have found when I was preparing. Sharing to hopefully save the reader some time.
Basics
Before we dive in, let’s just make sure we’ve got the basics covered
cdk ls
cdk ls lists all the stacks in the app, including the pipeline.
Example from my test project:
$ cdk ls
Proj1Stack
Proj1Stack/Deploy1/LambdaStack1
Proj1Stack/Deploy2/LambdaStack1
Proj1Stack is the pipeline.
Deploy1 and Deploy2 are “stages”
cdk synth
cdk synth $STACK_NAME >1.yaml is your friend, a debugging tool. It shows the generated CloudFormation.
cdk.out directory
cdk.out is the directory where cdk synth outputs everything that’s need for deploying (CloudFormation templates, related assets, metadata). They call it Cloud Assembly.
All assets are named based on the hash of their content so they are unique and immutable.
Build with name Synth (a CodeBuild project that runs cdk synth)
Build with name SelfMutate (a CodeBuild project that runs cdk deploy to update the pipeline)
Build with name FileAsset1 (a CodeBuild project that runs cdk-assets publish). From reading sources: there might be several cdk-assets publish commands configured in the CodeBuild project.
Then two CloudFormation deploy actions per each “stage” you want to deploy to (usage of change sets is the default but can be disabled as per documentation, see useChangeSets):
CHANGE_SET_REPLACE
CHANGE_SET_EXECUTE
cdk-assets
“It will take the assets listed in the manifest, prepare them as required and upload them to the locations indicated in the manifest.”
Note that cdk-assets is not making any decisions; metadata in the cdk.out directory has the information about assets, how to build/transform them and where they go.
cdk-assets can only handle two types of assets:
files (including directories). cdk-assets knows how to zip directories and how to upload files and directories to S3.
(From reading source code) Didn’t see in use but apparently cdk-assets can also run an executable to package a file (or directory?). In this case the content-type of the output is assumed to be application/zip. 🤷♂️
Docker images. cdk-assets knows how to build Docker images and push them into registry.
Sample command to see the list of assets: npx cdk-assets ls -p cdk.out/Proj1Stack.assets.json
What is Built When?
Files – unprocessed
If the files/directories don’t need any processing, they are just copied over to cdk.out during cdk synth and given a name which is a hash of the contents.
cdk-assets builds a docker image and pushes it into the specified repository. The input for the build of the image is a directory in cdk.out which has the Dockerfile and related files.
Deploy
After everything was built and uploaded during cdk synth and cdk-assets, the deploy uses CloudFormation template (templates?) from the cdk.out directory. At this point the assets (which the template references) are in ECR and S3.
I tried to condense the information that deemed important.
Let me know if something is missing or if you see mistakes.
The plan is to update this post as I discover new information.
My mistakes so far
Started taking notes quite a few hours into the process instead of from the start. Especially it would save me the jumping between the pipeline and the build projects to re-check what each action does.
These two words are used interchangeably. Please don’t. They mean different things. Here is my concise explanation.
Argument
A value passed into a function/method during invocation.
my_func(arg1, arg2)
Additional names for “argument” are “actual argument” and “actual parameter”.
Parameter
A name of a variable in the function/method definition. During invocation, the variable is used in the function/method body to refer to the value of the passed argument.
F my_func(param1, param2) {
...
# Using param1 and param2 for a computation
...
}
Additional name for “parameter” is “formal argument”.
Tip – Parametrize
If you struggle to remember which one is which, this might help: when you “parameterize” a piece of code, you add parameters to the code. Then you have the code with the parameter used in it, with the first occurrence in the function/method definition.
# Initial version
echo("Hello, Joe")
# Parametrized version. "name" is a parameter.
F hello(name) {
echo("Hello, ${name}")
}
Following is my opinion of the interactive mode of the Unix Shell. The interactive mode of the Unix shell is not much different from using Telegraph. I’ll substantiate the claim and point out what went wrong.
History
Short history of relevant inventions and events follows.
Telegraph – 1840s
Telegraph is a “point-to-point text messaging system”
Teleprinter – 1887
Teleprinter improved over telegraph’s user interface by adding a keyboard. Essentially it’s still a text messaging system.
“Computers used teleprinters for input and output from the early days of computing.”
Computer Terminal with VDU – 1950s
Computer Terminal with a “video display unit (VDU)” improved over teleprinter by replacing the printer with a screen.
( Summary Till this Point )
We see incremental progress in transmitting text. First between humans and later between a human and computer. The core concept did not change over time. Send text, receive text in response.
VT 52 – 1974/1975
VT 52 is released. It’s a pretty big deal because it supports cursor movement. Text on the screen can therefore be updated. In more general terms, it allowed interaction with what’s on the screen.
Bill Joy – vi – 1976
Bill Joy released vi in 1976. He figured that since cursor movement is supported he can make an editor that actually interactively uses the whole screen. That’s in contrast to the ex editor which didn’t.
What Went Wrong with the Shell?
The shell never caught up with the idea of interactivity on the screen. It’s still a “point-to-point text messaging system”. The paradigm shift never happened. Treating the received text as if it was printed on paper puts the idea of interacting with the output completely out of the realm of possibilities.
The “interactive” shell is not that much interactive. It is only “interactive” compared to “batch”. If you take a 24 lines terminal, considering that all interaction happens on one line (the “command line” I guess), that’s about 4% interactivity by line count.
Practical Consequences
Copy + Paste
Does the following sequence of operations happen to you when working in a shell?
Type and run a command
Start typing a second command
Copy something from the output of the first command
Paste that as an argument to the second command
I know that it does happen to me quite often. Why can’t the shell (tab) complete from the output of the first command? Because the shell has no idea (1) what’s in that output (2) what’s the semantic meaning of what’s in that output and how to use it for completion.
Imagine for a moment a website where you see a list of items but if you want more information about one of them, you need to copy its name and paste somewhere else, adding a word or two. Sounds weird? How is this OK in a shell then?
For the “shell is not supposed to do that” camp: you are welcome to continue to use Nano, Pico and Notepad for programming because “text editor is not supposed to do that”; I’ll use an IDE because it’s more productive.
Each Command on its own
Typically a shell user tries to achieve a goal. For that, typically a series of related commands. The Unix shell doesn’t know or care about that.
Let’s say you issued ls *.txt and you see the file that you were interested in in the output. You can’t interact with it despite being on the screen right in front of you so you proceed. Now you typed cp and pressed tab for completion. The completion will happily try to complete the names of all the files in the directory, not only *.txt which are way more likely to be meant by the user. There isn’t be any relation between sequentially issued commands in the Unix shell. Want composition? OK, use pipe or start scripting.
I’m designing and implementing Next Generation Shell, a programming language (and a shell) for “DevOps” tasks (read: running external commands and data manipulation are frequent).
I came across a programming pattern (let’s call it P) as follows:
An object is created
Some operations are performed on the object
The object is returned from a function (less frequently – stored in a variable)
P Using Plain Approach
The typical code for P looks in NGS like the following:
F my_func() {
my_obj = MyType()
my_obj.name = "blah"
my_obj.my_method(...)
my_obj # last expression is evaluated and returned from my_func()
}
The above looks repetitive and not very elegant. Given the frequency of the pattern, I think it deserves some attention.
Attempt 1 – set()
In simpler but pretty common case when only assignment to fields is required after creating the object, one could use set() in NGS:
F my_func() {
MyType().set(name = "blah")
}
or, for multiple fields:
F my_func() {
MyType().set(
name = "blah"
field2 = 100
field3 = "you get the idea"
)
}
Side note: parameters to methods can be separated by commas or new lines, like in the example above.
I feel quite OK with the above but the cons are:
Calling a method is not supported (unless that method returns the original object, in which case one could MyType().set(...).my_method())
Setting of fields can not be interleaved in a straightforward manner with arbitrary code (for example to calculate the fields’ values)
Attempt 2 – tap()
I’m familiar with tap() from Ruby. It looked quite useful so NGS also had tap() for quite a while. Here is how P would look like in NGS when implemented with tap():
F my_func() {
MyType().tap({
A.name = "blah"
A.my_method()
})
}
Tap takes an arbitrary value, runs the given callback (passing that value as the only argument) and returns the original value. It is pretty flexible.
Can’t put my finger on what’s exactly is bothering me here but the fact is that I was not using tap() to implement P.
Attempt 3 – expr::{ … }
New Life of tap()
This one is very similar to tap() but it is syntactically distinct from tap.
F my_func() {
MyType()::{
A.name = "blah"
# arbitrary code here
A.my_method()
}
}
I think the main advantage is that P is easily visually distinguishable. For example, if you only want to know the type of the expression returned, you can relatively easy skip everything between ::{ and } . Secondary advantage is that it’s a slightly less cluttered than tap().
Let’s get into the details of how the above works.
Syntax
MyType() in our case is an expression. Happens to be a method call which returns a new object.
:: – namespace field access operator. Typical use case is my_namespace::my_field.
{ ... } – anonymous function syntax. Equivalent to a function with three optional parameters (A, B, and C, all default to null).
Note that all three syntax elements above are not unique to this combination. Each one of them is being used in other circumstances too.
Up until recently, the :: syntax was not allowing anonymous function as the second argument. That went against NGS design: all methods should be able to handle as many types of arguments as possible. Certainly limiting arguments’ types syntactically was wrong for NGS.
Semantics
In NGS, any operator is transformed to a method call. :: is no exception. When e1::e2 is encountered, it is translated into a call to method :: with two arguments: e1 and e2.
NGS relies heavily on multiple dispatch. Let’s look at the appropriate definition of the :: method from the standard library:
F '::'(x, f:Fun) {
f(x)
x
}
Not surprisingly, the definition above is exactly like the definition of F tap() ... (sans method and parameters naming).
Examples of expr::{ … } from the Standard Library
# 1. Data is an array. Each element is augmented with _Region field.
data = cb(r)::{
A._Region = ConstIter(r)
}
# 2. push() returns the original object, which is modified in { ... }
F push(s:Set, v) s::{ A.val[v] = true }
# 3. each() returns the original object.
# Since each() in { ... } would return the keys() and not the Set,
# we are working around that with s::{...}
F each(s:Set, cb:Fun) s::{ A.val.keys().each(cb) }
# 4. Return what c_kill() returns unless it's an error
F kill(pid:Int, sig:Int=SIGNALS.TERM) {
c_kill(pid, sig)::{
A == -1 throws KillFail("Failed to kill pid $pid with signal $sig")
A != 0 throws Error("c_kill() did not return 0 or -1")
}
}
Side note: the comments are for this post, standard library has more meaningful, higher level comments.
A Brother Looking for Use Cases
While changing syntax to allow anonymous function after ::, another change was also made: allow anonymous function after . so that one could write expr.{ my arbitrary code } . The whole expression returns what the arbitrary code returns. Unfortunately, I did not come across (or maybe haven’t noticed) real use cases. The appropriate . method in the standard library is defined as follows:
F .(x, f:Fun) f(x)
# Allows
echo(5.{ A * 2 }) # 10
Have any use cases which look less stupid than the above? Let me know.
You have a program with human readable output. Now a second program needs that information. There are two choices:
Do the technically right and challenging thing – create a protocol and rewrite the first program and then write the second program (utilizing something like libxo in the first program maybe).
Fuck everybody for the decades to come by deciding that parsing text is the way to go.
I would like to thank everybody involved in choosing number 2!
jc is an effort to fix (more work around) that decision. Thanks to the author and I hope it will make your DevOps life at least a bit more tolerable.
The (very simplified and rough) TL;DR of the above link:
The Internet has “Narrow Waist”, the IP protocol. Anything that is above that layer (TCP, HTTP, etc), does not need to be concerned with lower level protocols. Each piece of software therefore does not need to concern itself with any specifics of how the data is transferred.
Unix has “Narrow Waist” which is text-based formats. You have a plethora of tools that work with text. On one side of of Narrow Waist we have different formats, on another side text manipulating tools.
I agree with both points. I disagree with implied greatness of the Unix “design” in this regard. I got the impression that my thoughts in this post are likely to be addressed by next oilshell blog posts but nevertheless…
Formats
Like hierarchy of types, we have hierarchy formats. Bytes is the lowest level.
Bytes
Everything in Unix is Bytes. Like in programming languages, if you know the base type, you have a certain set of operations available to you. In case of Bytes in Unix, that would be cp, zip, rsync, dd, xxd and quite a few others.
Text
A sub-type (a more specific type) of Bytes would be Text. Again, like in a programming language, if you know that your are dealing with data of a more specific type, you have more operations available to you. In case of Text in Unix it would be: wc, tr, sed, grep, diff, patch, text editors, etc.
X
For the purposes of this discussion X is a sub-type of Text. CSV or JSON or a program text, etc.
Is JSON a sub-type of Text? Yes, in the same sense that a cell phone is a communication device, a cow is an animal, and a car is a transportation device. Exercise to the reader: are this useful abstractions?
Cow is an animal
The Text Hell
The typical Unix shell approach for working with X are the following steps:
Use Text tools (because they are there and you are proficient wielder)
One of:
Add a bunch of fragile code to bring Text tools to level where they understand enough of X (in some cases despite existing command line tools that deal specifically with X)
Write your own set of tools to deal with the relevant subset of X that you have.
Optional but likely: suffer fixing and extending number 2 for each new “corner case”.
The exception here are tools like jq and jc which continue gaining in popularity (for a good reason in my opinion). Yes, I am happy to see declining number of “use sed” recommendations when dealing with JSON or XML.
Interestingly enough, if a programmer would perform the above mentioned atrocities in almost any programming language today, that person would be pointed out that it’s not the way and libraries should be used and “stop using square peg for round hole”. After few times of unjustified repetition of the same offense, that person should be fired.
Square peg / round hole
Somehow this archaic “Unix is great, we love POSIX, we love Text” approach is still acceptable…
Pipes Text Hell
Create a pipe between different programs (text output becomes text input of the next program)
Use a bunch of fragile code to transform between what first program produces and the second one consumes.
Where Text Abstraction is not Useful
Everywhere almost. In order to do some of the most meaningful/high-level operations on the data, you can’t ignore it’s X and just work like it is Text.
Editing
The original post says that since the format is Text, you can use vim to edit it. Yes you can… but did you notice that any self respecting text editor comes with plugins for various X’s? Why is that? Because even the amount of useful “text editing” is limited when all you know you are dealing with Text. You need plugins for semantic understanding of X in order to be more productive.
Wanna edit CSV in a text editor without CSV plugin? OK. I prefer spreadsheet software though.
Have you noticed that most developers use IDEs that “understand” the code and not Notepad?
Lines Count
Simple, right? wc -l my.csv. Do you know the embedded text in quotes does not have newlines? Oops. Does it have header line? Oops.
Text Replacement
Want to try to rename a method in a Java program? sed -i 's/my_method/our_method/g' *.java, right? Well, depends on your luck. I would highly recommend to do such kind of refactoring using an IDE that actually understands Java so that you rename: only specific method in a specific class as opposed to unfortunately named methods and variables, not to mention arbitrary strings.
Search / Indexing
Yep… except that understanding of the semantics helps here quite a bit. That’s why you have utilities which understand specific programming languages that do the indexing.
Conclusion
I do not understand the fascination with text. Still waiting for any convincing arguments why is it so “great” and why the interoperability that it provides is not largely a myth. Having a set of tools enabling one to do subpar job each time is better than not having them but is it the best we can?
Layering in case of IP protocol works just fine. Implementer of HTTP server really does not care about the low level transport details such as Ethernet. Also the low level drivers don’t care which exactly data they deliver. Both sides of the Waist don’t care about each other. This works great!
My claim is that in case of the Text Narrow Waist, where X is on one hand of and the Text tools are on the other, there are two options:
Tools ignore X and you have very limited functionality you get out of the tools.
Tools know about X but then it’s “leaky abstraction” and not exactly a Narrow Waist.
That’s why I think that in case of Text, the Narrow Waist is more of an illusion.
Let’s talk about the “just separate with comma and stick it into one field” type of serialization.
You had two strings (abc and def) and you joined them with a separator. What do you have now? One string with two elements, right? Right, abc,def. Well… two or more actually, depending on how many times the chosen separator occurred in the original strings: if they were a,bc and def, you’ve got a,bc,def, which is 3 elements according to our format. Oops. Leaving out the question whether leading and trailing spaces are significant.
Wanna add escaping for the separator then? a,bc and def are now serialized as a\,bc,def. Now the parsing became more complex. You can’t just split the string by the separator (you would get 3 elements: a\ and bc and def. You need to scan the serialized data, considering escaping when splitting. You also need to remove the escaping character from the result. How about escaping the escape character? If original data is a\bc, it is serialized as a\\bc). Yet another something not to forget.
Don’t like escaping then? How about encoding like in URL? a,bc becomes a%2Cbc. You can now once again split the string by the separator character… assuming it was encoded. Which characters you encode anyway? If you encode all ASCII characters, the result is 3 times the original and is completely unreadable. It least you are “safe” with regards to separator now, it is encoded for sure so no split problems. You have to add a decoding routine now though.
If your serialized thing goes into a database, consider how indexing would work. It probably won’t. Maybe you should model your domain properly in the database and not serialize at all. Hint: if the values ever need to be treated differently/separately by the database, they go into different cells/rows/columns/fields, not one. There are very rare exceptions. Notable exception is the ability of databases to handle JSON fields (examples: MySQL, PostgreSQL). Note that this capability can fit or not fit your use case.
Want to satisfy your artistic needs and do something clever about the serialization? Do it at home then please. Don’t waste time that your colleagues could use on something more productive than dealing with your custom format.
Strong advice: don’t do custom serialization format, use existing serialization formats and libraries.
Seen something to add to the above? Leave a comment!
Ilya's blog 