Common,,Mistakes

As a programmer / code reviewer, any string manipulation is a red flag and should put you on alert. It’s error prone and mistakes are common. That goes for constructing, parsing, and searching within strings. Avoid string manipulation whenever possible.

Rationales, pitfalls, and alternative solutions follow.

Concatenation

Concatenation usually means having two (or more) separate data items mashed into one. The problems with that are:

  • Syntactically – depending on how this is done, it might be easy, hard, or impossible (guaranteed correctly) to parse the string back into original items. One can think of concatenation as (likely) incorrectly implemented serialization and the corresponding parsing as incorrectly implemented deserialization.
  • Semantically – in many cases string concatenation is semantically incorrect and shouldn’t have been done at all, the items should have been kept separately.

Examples follow.

Emojis in Log Messages

Assume that we have added several methods to our logger:

  • logSecurityIssue(message)
  • logPerformanceIssue(message)
  • logPossibleBug(message)
  • logInvalidInput(message)
  • logInvalidConfiguration(message)

Each one of them results in a new log message of the form: “${category_emoji} ${message}”. The category of the message should have been in its own field instead. The consequences of broken semantics (two-in-one message field) are:

Broken user interface: Now imagine a user interface for this log where the user tries to filter by category. If you mixed the category into the message, there is no “category” from the UI perspective. You would need to search by emoji which is harder to type and can potentially occur inside unrelated messages. Leaving aside what your indexer might decide to do with these emojis.

Harder automated parsing for analytics of these logs: Parsing (for analytics for example) might or might not be an issue. We know that the emoji is at the beginning of the message and is separated by space. That works. Unless… you have generic log(message) method which does not prepend any emoji. Let alone special code in the parser for no-emoji case, how about messages passed to log(message) that can contain emojis in the beginning of the string? Note that even when parsing is “not an issue” it is already more of an issue compared to not having parsing at all, which would be the case if category went into its own field.

Constructing Code

When concatenation is used to construct code with embedded data, the data needs to be properly sanitized and/or escaped /quoted. Failing to do so, a common mistake, enables:

  • Failed execution due to resulting syntax error
  • Incorrect execution due to unintended parsing of syntactically correct result of concatenation
  • An attacker can pass specially crafted data that contains code that will be executed. This is known as Code Injection. Common examples are SQL injection and XSS.

For constructing FHIRPath expressions, for example, use the provided method of passing variables, envVars argument, do not concatenate code and data.

For SQL, almost any framework (ORMs for example) provides a way to pass values without concatenation of SQL “code” and data together.

As to code in general, I’m against generating/constructing code but if you have to, you can build AST and then convert it into code in one place, where you would need to deal with escaping/quoting once.

As a tangent / side note, the more general issue of mixing code and data, or instructions and data is a long known security issue that has known solutions except for LLMs. This issue is unsolved in AI tools and there is no solution in sight.

Constructing URLs

Another common occurrence is using concatenation for constructing URLs. Stop for a minute and think what’s wrong with that…

As with other cases, escaping (or encoding) is often forgotten. I recommend using new URL() in combination with modifying searchParams instead of being on your toes trying to remember escaping/quoting/encoding all the time.

Constructing HTML

Almost any framework here provides escaping/quoting. Use it. Just don’t concatenate HTML and dynamic data. If you do, again, you have to remember to escape/quote all the time.

Constructing Output for Humans only

Here we get into my long-standing issue (aka pet peeve) with Unix. Programs are primarily written to display results to users as text.

This “design” is a nightmare of forgotten escaping/quoting and incorrect parsing, costing everyone in case the output should be actually consumed by the next program… which does happen quite often.

I recommend preparing a data structure and only concatenating it to strings at the latest possible moment for human consumption. This way you can have a switch to (properly) serialize this data instead. Use this switch if the output needs to be used by another program (typically, indicated by non tty stdout). Serialization avoids concatenation (in the bad sense), which is very hard to undo in the next program.

Alternatively, use an existing solution, libxo, to output in configurable formats. Configuration is through command line arguments.

Side note about parsing outputs of programs. A very nice project that does exactly that is jc, which in my opinion would not be needed if we had the correct architecture to begin with.

Parsing

Text parsing is common… and error prone. In general, every parser is more complicated than what seems initially.

Splitting CSV file lines by comma? This is wrong. Commas can be in quoted fields.

Parsing your own format? Unsecaped delimiters – a bug, escaped delimiters – extra code to parse. You can’t win.

If you can’t avoid using your own format (must be very rare occurrence when all existing formats do not match the use case), use a parser library. I like peg parsers. Raku has its own parsing feature. For binary parsers there are also libraries, use them. Whatever works for you, just not hand made. This advice is applicable to close to 100% of cases.

Searching

You have items “foo”, “bar”, and “baz”. They should be an array/list/collection/set (however it is called in your language). If they are mashed into a string, for example “foo:bar:baz”, looking for a substring “bar” to check for element presence is likely to be incorrect. In “real life” the items would be an input which we don’t control. Possible problems are:

Substring match

If you have “foo bar” as an input, your checks for both “foo” and “bar” would succeed, which is not intended of course.

The next obvious step is to look for “:bar:” instead. For that to work in the general case you need to handle the edge cases for the start and the end of the string. It means either have the string “:foo:bar:baz:” or have three separate lookups – “bar:” in the beginning, “:bar” in the end, and “:bar:” in the middle.

If you are going with the “:foo:bar:baz:”, another edge case surfaces. Will you ever be checking if the empty string is an element? With this format, the empty string is always (well, almost) an element here.

Unescaped Separator

Let’s assume you went with “:foo:bar:baz” and checking for “:bar:” presence now. Possible element “foo bar” won’t match. But how about “foo:bar” input element? If your input element has the separator character in it, the search is broken. You need to encode the separator character before adding an element to the string… and decode when you need the value back… and you need to remember to encode the substring that you are looking for too. How this can look like for example?

Let’s say our escape character would be “@”. Input “:” would be encoded as “@c”. Then leaving “@c” in the input as is would be incorrectly decoded to “:” later. You have to escape “@” in the input to. So “@” would be let’s say “@@”.

“foo:bar” encodes as “foo@cbar”, “foo@bar” as “foo@@bar”. That’s true when adding new elements and when preparing substring search.

Such “:foo:bar:baz:foo@cbar:” string represents a serialized array/list/etc and you have just invented your own serialization format. See my earlier post – On Accidental Serialization Formats.

Conclusion

I hope that after reading this post, whenever possible, when presented with a choice, you will go for the simpler and the more semantically correct option which does not involve string manipulation. Left without a choice, you will be prepared for the uphill battle and aware of the pitfalls.

Hope this helps. Good luck!

Common Mistakes[0]

Using the first element of an array without additional checks is a common mistake.

function getMyThing() {
  const things = getMyThings(...);
  return things[0];
}

The code above assumes that there is at least one element in things. Unfortunately, it doesn’t do anything about the situation when this assumption is not true. The following fix (or something similar) is common:

function getMyThing() {
  const things = getMyThings();
  if (things.length == 0) {
    return null;
  }
  return things[0];
}

Despite the (most likely) intent of getting the only element in the array, the example above fails to handle multiple elements in things. Correct handling of having more than one element is rarely seen. If things are sorted and it’s intentional then there is no problem. This is rarely the case though. Choosing random first element is not the correct or intended behavior in most cases.

I do recommend that if you assume having exactly one element in things and you either think it won’t happen or just not ready to handle this situation – add assertion that would both signal the intent and prevent incorrect execution when this situation happens.

import assert from "node:assert";

function getMyThings() {
  return [2,3];
}

function getMyThing() {
  const things = getMyThings();
  assert.equal(things.length, 1, "Expected to have exactly one thing"); // Throws here
  return things[0];
}

console.log(getMyThing());

You can of course to opt for separate checks for length being zero and length being more than one and have more precise error messages… but this is not supposed to happen, right? 🙂

I recommended using assertion above to convey a programming error. Don’t use assertions if:

  • If things are input to the program then use if(...) { throw } or consider some input validation framework (like zod) that would eliminate the need for code like above.
  • If no other part of the code uses assertions then stay consistent with that.

Some languages provide specialized facilities for dealing with exactly one element instead of the more generic assert(). Examples: Enumerable.Single in .NET, the_one in NGS.

This is the first in series of posts about common mistakes in programming. Stay tuned.


Hope this helps. Have a nice $TIME_OF_DAY!

To assert() or not to assert()?

Background

I’ve noticed that I’m using and recommending to use assert() in our TypeScript projects. Particularly, to be used in runtime, including input validation sometimes, as opposed to the common use case which is in tests. While one colleague was fully onboard, another one noted that use of assert() in runtime is non-idiomatic and is therefore confusing and we should use differently-named function(s). While partially agreed, I didn’t think these statements were so clear cut and started digging.

My Opinion

… on use of assertions by use case, after some research (see research results below).

Tests

That’s the natural habitat of assertions and where they should be used freely. I don’t think this idea is a surprise to anybody.

Runtime – Programming Errors

In context of languages that keep assertions in production (ex: JavaScript/TypeScript)…

Yes, I do recommend using assertions for catching programming errors at runtime. Assertions here express the intent clearly and concisely. They are explicit assumptions. Failed assertion means a programming error caught at runtime. In context of an HTTP(S) server, these failed assertions should return HTTP status 500 Internal Server Error.

This includes deployment/configuration errors. For example, if there is an environment variable that is required for your code to run – you should assert() that it was passed. Side note: do it as soon as possible and fail the initialization if it’s missing.

I tend to include invalid data fetched from database in this category too.

Runtime – Input Validation

Assertions are not a good match for input validation. A more elegant and appropriate solution should be used. In TypeScript, for example, it could involve zod or some other validation libraries. For HTTP(S) server side code, input validation failure should return HTTP status 400 Bad Request.


If you are short on time, you can stop here. Below goes some interesting information about assertions though.


Research Results

I have looked into sources of Node.js and undici (deps/undici in Node.js), and vscode, and others for some “authoritative” answers. I have found:

  • Extensive use of assert() at runtime for asserting invariants. This runtime usage is still aligned with the semantics of assert() in tests: failed assertion means programming error.
  • Specifically, these projects, do not use assert() for arguments validation (with the exception of one borderline case).
  • Node.js
    • Uses a bunch of validateFoo functions for argument validation: validateNumber, validateString, validateOneOf, etc that throw appropriate errors. What makes them special and different from assert() is that they produce messages meaningful for argument validation. For that, validateFoo functions have name parameter. And of course, validateFoo functions they convey different intent.
    • States “AssertionErrors are a special class of error that can be triggered when Node.js detects an exceptional logic violation that should never occur. These are raised typically by the node:assert module.”
  • vscode defines and uses functions such as assert(), assertFn() and assertType().
    • Interestingly enough, assertMarkdownFiles() in extensions/github/src/pushErrorHandler.ts doesn’t actually assert anything but filters (I opened GitHub issue).
  • NestJS
    • Defines and uses assertFoo methods NestApplicationContext#assertNotInPreviewMode() and RouterResponseController#assertObservable()at runtime.
    • Uses Channel#assertExchange()method from amqplib. Here, “assert” is used in a sense of “make sure something exists”, which is named “ensure” in some other contexts. TODO: google “ensure”. use of “ensure” in programming
  • assert-plus, a transitive dependency of NestJS, is a wrapper around assert() but specifically for arguments validation. Given around 10M weekly downloads as of writing, it can’t be dismissed.

It looks like some are using “assert” in the broader sense of “this situation was not supposed to happen”, including argument validation while others use “assert” in the stricter sense of catching programming errors.

What is assert()?

I’m intentionally mixing different sources here to provide broader view.

Node.js documentation says:

The node:assert module provides a set of assertion functions for verifying invariants.

C# documentation (for Microsoft.VisualStudio.TestTools.UnitTesting Namespace, Assert class) says:

A collection of helper classes to test various conditions within unit tests. If the condition being tested is not met, an exception is thrown.

C# documentation (for Debug.Assert Method) says:

Checks for a condition; if the condition is false, outputs messages and displays a message box that shows the call stack.

By default, the Debug.Assert method works only in debug builds. Use the Trace.Assert method if you want to do assertions in release builds. For more information, see Assertions in Managed Code.

C++ online reference says:

The definition of the macro assert depends on another macro, NDEBUG, which is not defined by the standard library.

If NDEBUG is defined as a macro name at the point in the source code where <cassert> or <assert.h> is included, the assertion is disabled: assert does nothing.

Java spec says:

An assertion is an assert statement containing a boolean expression. An assertion is either enabled or disabled. If an assertion is enabled, execution of the assertion causes evaluation of the boolean expression and an error is reported if the expression evaluates to false. If the assertion is disabled, execution of the assertion has no effect whatsoever.

… assertions should not be used for argument checking in public methods. Argument checking is typically part of the contract of a method, and this contract must be upheld whether assertions are enabled or disabled.

JUnit documentation says:

Assertions is a collection of utility methods that support asserting conditions in tests.

Lisp documentation says:

assert assures that test-form evaluates to true. If test-form evaluates to false, assert signals a correctable error

… and goes on to provide (assert ...) example for arguments validation

C2 wiki says:

An assertion is a boolean expression at a specific point in a program which will be true unless there is a bug in the program.

How Assertions are Used?

The use cases from above range widely:

  • Tests only (C#, JUnit)
  • Runtime during development only (C#, C++, Java), no-op in production code
  • Runtime (C#, Lisp)

Yes, C# has all three variations as separate facilities.

Some examples from Node.js follow.

Tests

Node.js, node/test/abort/test-abort-backtrace.js

Note that this form gets both of the actual result and the expected result. It can therefore print both when there is discrepancy.

const child = cp.spawnSync(`${process.execPath}`, [`${__filename}`, 'child']);

const stderr = child.stderr.toString();
assert.strictEqual(child.stdout.toString(), '');

...
assert(jsStack.some((frame) => frame.includes(__filename)));

Catching Programming Error at Runtime

Node.js, node/lib/_http_agent.js, expressing intent using assert.fail() rather than just throwing an error.

function createConnection(...args) {

  ...

  // This should be unreachable because proxy config should be null for other protocols.

  assert.fail(`Unexpected proxy protocol ${proxyProtocol}`);

};

Asserting Invariant at Runtime

Node.js, node/lib/_http_client.js

function socketOnData(d) {

  const socket = this;

  const req = this._httpMessage;

  const parser = this.parser;


  assert(parser && parser.socket === socket);

  ...

}

“It’s complicated”

Node.js, node/lib/https.js

In this case, the phrasing suggests catching internal programming error but createConnection() – the caller – “custom agents may override this method to provide greater flexibility,” so assert() here can be seen as (borderline) argument validation.

function getTunnelConfigForProxiedHttps(agent, reqOptions) {

  ...
  
const requestHost = ipType === 6 ? `[${reqOptions.host}]` : reqOptions.host;
const requestPort = reqOptions.port || agent.defaultPort;
const endpoint = `${requestHost}:${requestPort}`;
  
// The ClientRequest constructor should already have validated the host and the port.
  
// When the request options come from a string invalid characters would be stripped away,
  
// when it's an object ERR_INVALID_CHAR would be thrown. Here we just assert in case

  // agent.createConnection() is called with invalid options.
  
assert(!endpoint.includes('\r'));

  assert(!endpoint.includes('\n'));
  ...
}

Interesting Findings Along the Way

Type Checking

Some of these assert()s in JavaScript code check for correct types at runtime and wouldn’t be needed in TypeScript which would check these statically, ahead of time.

Custom Exceptions

In Node.js, the message parameter can be an instance of Error and in this case, it’s thrown instead of the usual AssertionError with the string message message. It was introduced in 2017 to “support a way to provide a custom Error type for assertions. This will help make assert more useful for validating types and ranges.”

??= for caching

Node.js, lib/internal/assert.js

Elegant use of ??= for caching.

let error;

function lazyError() {

  return error ??= require('internal/errors').codes.ERR_INTERNAL_ASSERTION;

}

ThrowIfNull

C# has ArgumentNullException.ThrowIfNull, which I find interesting language design choice – static method on the exception that throws conditionally.

 


Hope it was interesting and enjoyable. What’s your opinion on the topic? Leave your comment below. Have a nice day!

Technologies I’m Impressed by

Following is the list of technologies I’m Impressed by, a completely subjective metric. Since I’m not impressed often, these deserve to be listed.

Impressed

In arbitrary order:

  1. JetBrains IDEs. The thorough understanding of the code is top notch.
  2. Vim. Amazing productivity when editing text.
  3. Linux. No explanation required 🙂
  4. FHIR. Very well thought through data exchange standard for healthcare. Extensible everywhere. Profiles to mandate constraints.
  5. AWS CDK. Amazing productivity gain over CloudFormation due to semantic involvement of the tool. It’s about expressing what you want to achieve, not assembling the solution from what you have.

Hope you have found something new to you in this list. In this case, I recommend taking a look at it.

Subjective Python or why I “hate” Python

Different people look at different facts and also interpret them differently. Following are my chosen facts about Python programming language and my subjective opinion about these facts, the Python language, and its use for DevOps.

Background

As I do from time to time, I post comparisons between Next Generation Shell (NGS), a shell I’m working on, and other languages, Python included. The comparisons are typically involving some small task that I consider a “typical DevOps task” (defined later). As you can imagine, in these comparisons NGS always wins. First, because NGS was specifically designed for and is actually better for many DevOps tasks. Second, because there is no reason for me to post a comparison where NGS is not better. Anyhow…

After such post on LinkedIn (“Still using #Python for #DevOps? You like to suffer?”), where I compared sample NGS and Python code, a friend asked me why I “hate” Python. My answer was that I’m feeling neutral about Python.

I found the topic of my perception of Python interesting and deserving elaboration.

Python

From my perspective, Python is okayish language, relative to other programming languages.

While I understand people that are annoyed by Python 2 to Python 3 migration and breakage, I also realize that it’s very hard to get things right from the start (or even later).

Few things in Python from the top of my head that are annoying for me:

  • Python does not encourage nor supports well basic functional programming: map() and filter(). Due to Python’s syntax, the callback lambda can only be single expression.
  • Instead, it encourages list comprehensions which act as both map() and filter(). I don’t like list comprehensions, they don’t align with how I think. Dict comprehensions too.
  • Typing is afterthought and therefore “The Python runtime does not enforce function and variable type annotations. They can be used by third party tools such as type checkers, IDEs, linters, etc.”
  • Conditional expressions (A if C else B, where C is the condition) are breaking my head.
  • if __name__ == "__main__" hack idiom is ugly as fuck.

There are annoying things in any language though.

My strongest negative feeling about Python, for which I should mostly blame myself, is parameters handling. While working on NGS, early on, I copied was inspired by parameters handling in Python. The thought was that parameter handling is such a basic thing that (A) I shouldn’t spend much time on it and (B) such established language like Python would get it right. Python didn’t get it right. And I copied that atrocity. They ugly-fixed it later. I’m still thinking how to do it right. The “right” for NGS is likely to differ from the “right” for Python. Also, I would like to avoid making the same mistake for the second time.

What do I actually “hate” about Python?

What I “hate” about Python is surprisingly not Python-specific. It’s the use of square pegs for round holes:

Using Python, Ruby, Go, and other languages that were not intended for “typical DevOps tasks” (looking at resulting code for that judgement) when you have clearly better matching alternatives, including NGS and some other modern shells.

“Typical DevOps tasks” include running external programs and small scale data manipulation (files, structured data, mapping, filtering, pattern matching).

Justifying the Square Pegs

This one is annoying, especially hearing the same arguments for 1000th time.

“But I can do it with libraries”. You can. Also you can do it in Assembler, C or FORTRAN with or without libraries. I do not except the argument. The solutions are not equal.

“It’s not that much of a difference”. Depends on the use case. Run external program in Python and handle exit codes properly and let’s talk then.

More legitimate arguments include:

  • “Our team uses Python and so we do DevOps in Python too”
  • “We are afraid we will not find Stack Overflow answers if we are stuck”
  • “We are afraid we won’t be able to hire developers in this new language”
  • “Where is the tooling?”

In my mind, the legitimacy of the arguments above is in reverse proportion with DevOps work volume. The more “DevOps” tasks you do as percentage of your total work, the more advantages of better suited programming languages shine through.

While sometimes the arguments are justified, other times it’s inertia which is holding us back. Like switching away from JavaScript which is not going according to the plan.

Programming Languages in General

Subpar. All of them. At least the ones that I have seen (dozens). Python included. NGS included. We as humanity didn’t figure that out yet. I mean programming languages. We are stuck in paradigms which hold us back.

NGS aims to be less bad for DevOps than the rest because it was designed for the niche. Almost round peg for round hole, way better than the square pegs.

Conclusion

While it might look like I “hate” Python,

  • it’s actually all programming languages
  • and even more so their inappropriate use and the worst is
  • justifying and perpetuating inappropriate use of programming languages

Thanks for reading, have a nice ${TIME_OF_DAY} !


For the curious ones, code comparison follows.

The Code Comparison

During my quest to understand CloudFormation better, I stumbled upon this code:

"Gets the name of the folder with the handler code"

import json
import sys

def main(rpdk_path):
    "Get the source folder from rpdk config"

    with open(rpdk_path) as f:
        obj = json.load(f)
        entrypoint = obj["entrypoint"]
        print(entrypoint.split(".")[0])

if __name__ == "__main__":
    main(sys.argv[1])

To which I responded with NGS version (renaming mostly meaningless obj to conf on the way):


#!/usr/bin/env ngs

# Gets the name of the folder with the handler code

F main(rpdk_path) {
	conf = read(rpdk_path).decode_json()
	echo(conf.entrypoint.split(".")[0])
}

To which my friend responded with another Python version, which was shorter and did not have main().

Command Line Arguments

Both the original and the shorter version in Python require import sys which shows that dealing with command line arguments is not as “important” in Python as in NGS.

The gap is between additional import in Python and out-of-the-box automatic parsing of command line arguments and passing them to main() in NGS.

JSON

Both Python versions require import json, showing again that it’s easier to deal with JSON in NGS than in Python.

The example would be even shorter if the use case was slightly more typical – the JSON file would have .json extension. read(rpdk_path).decode_json() would become fetch(rpdk_path), leaving Python far behind.


Amazing! You stayed here till the end. Congrats and thanks!

Readable TypeScript debug() Output for Protobuf Messages

For search engine indexing and for understanding: Protobuf is frequently used with GRPC. Protobuf is how the requests and responses are serialized.

The Problem

Look at the output below. Printing protobuf messages using debug(‘%j’) produces unreadable output (the first one below).

$ DEBUG=myapp npx ts-node tojson.ts 

  myapp My object without toJSON {"wrappers_":null,"arrayIndexOffset_":-1,"array":[null,null,null,null,null,"Session name 01"],"pivot_":1.7976931348623157e+308,"convertedPrimitiveFields_":{}} +0ms

  myapp My object with toJSON {"id":"","CENSORED4":"","CENSORED3":"","version":"","name":"Session name 01","CENSORED1":"","status":0,"CENSORED2":""} +1ms

Solution

Here is all you need to make it work. SessionResponse below is a sample message (export class SessionResponse extends jspb.Message in another file, after import * as jspb from "google-protobuf";)

// --- Setup ---
import {Message} from "google-protobuf";

import {
    SessionResponse
} from "./PATH/TO/YOUR_GENERATED_PROTOBUF_FILE_pb";

import debug0 from 'debug';
const debug = debug0('myapp');

const req = new SessionResponse().setName("Session name 01");

// --- Test before ---
debug('My object without toJSON %j', req)

// --- Add this to your code --- start ---

declare module "google-protobuf" {
    interface Message {
        toJSON(): any;
    }
}
Message.prototype.toJSON = function () {
    return this.toObject();
}

// --- Add this to your code --- end ---

// --- Test after ---
debug('My object with toJSON %j', req)

Hope this helps. Let me know if you have questions.