Plurrrr

home

Taming Go’s Memory Usage

A couple months ago, we faced a question many young startups face. Should we rewrite our system in Rust?

At the time of the decision, we were a Go and Python shop. The tool we’re building passively watches API traffic to provide “one-click,” API-centric visibility, by analyzing the API traffic. Our users run an agent that sends API traffic data to our cloud for analysis. Our users were using us to watch more and more traffic in staging and production—and they were starting to complain about the memory usage.

This led me to spend 25 days in the depths of despair and the details of Go memory management, trying to get our memory footprint to an acceptable level. This was no easy feat, as Go is a memory-managed language with limited ability to tune garbage collection.

Source: Taming Go’s Memory Usage, or How We Avoided Rewriting Our Client in Rust, an article by Mark Gritter.

The Actor Reentrancy Problem in Swift

When the first time I saw the WWDC presentation about actors, I was thrilled with what it is capable of and how it will change the way we write asynchronous code in the near future. By using actors, writing asynchronous code that is free from data races and deadlocks has never been easier.

All that aside, that doesn’t mean that actors are free from threading issues. If we are not careful enough, we might accidentally introduce a reentrancy problem when using actors.

Source: The Actor Reentrancy Problem in Swift, an article by Lee Kah Seng.

Everyone’s a (Perl) critic, and you can be too!

The perlcritic tool is often your first defense against “awkward, hard to read, error-prone, or unconventional constructs in your code,” per its description. It’s part of a class of programs historically known as linters, so-called because like a clothes dryer machine’s lint trap, they “detect small errors with big effects.” (Another such linter is perltidy, which I’ve referenced in the past.)

Source: Everyone’s a (Perl) critic, and you can be too!, an article by Mark Gardner.

Inspecting coredumps like it's 2021

A coredump is a snapshot of a process’s memory that is usually created by the kernel when a crash happens. These can be fairly helpful to find out which part of the code broke by looking at the backtrace or finding any kind of corruption by introspecting the memory itself. Unfortunately it can be a bit tedious to work with these. This article aims to give an overview over helpful tools & tricks to leverage the full power of coredumps on Nix-based systems.

Source: Inspecting coredumps like it's 2021, an article by Maximilian Bosch.

Python Plotting for Exploratory Data Analysis

Plotting is an essential component of data analysis. As a data scientist, I spend a significant amount of my time making simple plots to understand complex data sets (exploratory data analysis) and help others understand them (presentations).

In particular, I make a lot of bar charts (including histograms), line plots (including time series), scatter plots, and density plots from data in Pandas data frames. I often want to facet these on various categorical variables and layer them on a common grid.

Source: Python Plotting for Exploratory Analysis, an article by Tim Hopper.

Filtering With PiHole and Podman

’ve long been a fan of filtering at the DNS level. The approach that I’ll outline here is similar to what our devices did at Luma. It is not perfect by any means, but if it is setup well then it provides a chokepoint that has a solid return on investment for the amount of time and effort that it takes to standup. It isn’t going to keep any sophisticated actors at bay, but it does demonstrably improve performance and has a good chance of reducing the headaches of administrating a network and the systems on it for a connected family.

Source: It's Always DNS. Filtering With PiHole and Podman, an article by Daniel Peck.

Currying

Currying is an advanced technique of working with functions. It’s used not only in JavaScript, but in other languages as well.

Currying is a transformation of functions that translates a function from callable as f(a, b, c) into callable as f(a)(b)(c).

Currying doesn’t call a function. It just transforms it.

Source: Currying, an article by Ilya Kantor.

Data Compression With Arithmetic Coding

Arithmetic coding is a common algorithm used in both lossless and lossy data compression algorithms.

It is an entropy encoding technique, in which the frequently seen symbols are encoded with fewer bits than rarely seen symbols. It has some advantages over well-known techniques such as Huffman coding. This article will describe the CACM87 implementation of arithmetic coding in detail, giving you a good understanding of all the details needed to implement it.

Source: Data Compression With Arithmetic Coding, an article by Mark Nelson.

Edge

Behind the well-known U.S. security organizations—the FBI and CIA among them—lies a heavily guarded, anonymous government agency dedicated to intelligence surveillance and to a highly specialized brand of citizen protection.

Shock waves of alarm ripple through the clandestine agency when Washington, D.C., police detective Ryan Kessler inexplicably becomes the target of Henry Loving, a seasoned, ruthless “lifter” hired to obtain information using whatever means necessary. While Loving is deft at torture, his expertise lies in getting an “edge” on his victim—leverage—usually by kidnapping or threatening family until the “primary” caves under pressure

In the evening I started in Edge by Jeffery Deaver.

The elements of git

This is not a tutorial. If you're looking for a quick, easy, "how to use git" kind of post, look elsewhere.

The goal of this post is to give you just enough understanding of the git internals that you can build up a correct intuition of what various git commands actually do under the hood.

Source: The elements of git, an article by Gary Verhaegen.

How to Make a Custom Screensaver for Mac OS X

Apple’s default screensavers for Mac OS X are nice, but they get boring after a while. Any true nerd would take matters into their own hands when that happens!

If you want to make your own custom screensaver for Mac OS X, this article is for you. We’ll walk through a tutorial for making a screensaver which emulates the game Pong. It’s going to be super simple — this is just to get you started with how to make a screensaver.

Source: How to Make a Custom Screensaver for Mac OS X, an article by Trevor Phillips.

The Ultimate Question of Programming, Refactoring, and Everything

In this article you will find 42 recommendations about coding in C++ that can help a programmer avoid a lot of errors, save time and effort. The author is Andrey Karpov - technical director of "Program Verification Systems", a team of developers, working on PVS-Studio static code analyzer. Having checked a large number of open source projects, we have seen a large variety of ways to shoot yourself in the foot; there is definitely much to share with the readers. Every recommendation is given with a practical example, which proves the currentness of this question. These tips are intended for C/C++ programmers, but usually they are universal, and may be of interest for developers using other languages.

Source: The Ultimate Question of Programming, Refactoring, and Everything, an article by Andrey Karpov.

Using Podman with BuildKit, the better Docker image builder

BuildKit is a new and improved tool for building Docker images: it’s faster, has critical features missing from traditional Dockerfiles like build secrets, plus additionally useful features like cache mounting. So if you’re building Docker images, using BuildKit is in general a good idea.

And then there’s Podman: Podman is a reimplemented, compatible version of the Docker CLI and API. It does not however implement all the BuildKit Dockerfile extensions. On its own, then, Podman isn’t as good as Docker at building images.

There is another option, however: BuildKit has its own build tool, which is distinct from the traditional docker build, and this build tool can work with Podman.

Let’s see where Podman currently is as far as BuildKit features, and how to use BuildKit with Podman if that is not sufficient.

Source: Using Podman with BuildKit, the better Docker image builder, an article by Itamar Turner-Trauring.

An Introduction to AWK

awk is a powerful tool. It is actually a Turing-complete language, meaning that you can technically write any kind of program with it. You could implement the classic sorting algorithms or more complex things such as a parser or an interpreter. Examples of this kind can be found in the “AWK Programming Language” book written by awk‘s authors. The reason awk is still popular today, though, has nothing to do with its generality and more with its usefulness working in the command line.

Source: An Introduction to AWK, an article by Francesc Vendrell.

Debugging by starting a REPL at a breakpoint is fun

Hello! I was talking to a Python programmer friend yesterday about debugging, and I mentioned that I really like debugging using a REPL. He said he’d never tried it and that it sounded fun, so I thought I’d write a quick post about it.

This debugging method doesn’t work in a lot of languages, but it does work in Python and Ruby and kiiiiiind of in C (via gdb).

Source: Debugging by starting a REPL at a breakpoint is fun, an article by Julia Evans.

Go'ing Insane Part One: Endless Error Handling

I’ve been using Go for a few years now, mostly in my open source project Lazygit. In my day job I use Ruby and Typescript, and I’ve also spent some time with Rust. Each of those languages have design quirks that can grind a developer’s gears, and although my own precious gears have been ground by every language I’ve used, Go is the only language that has made me feel indignant.

Source: Go'ing Insane Part One: Endless Error Handling, an article by Jesse Duffield.

OpenBSD's pledge and unveil from Python

Years ago, OpenBSD gained two new security system calls, pledge(2) (originally tame(2)) and unveil. In both, an application surrenders capabilities at run-time. The idea is to perform initialization like usual, then drop capabilities before handling untrusted input, limiting unwanted side effects. This feature is applicable even where type safety isn’t an issue, such as Python, where a program might still get tricked into accessing sensitive files or making network connections when it shouldn’t. So how can a Python program access these system calls?

Source: OpenBSD's pledge and unveil from Python, an article by Chris Wellons.

Unravelling async for loops

When I decided the next post in my series on Python's syntactic sugar would be on async for, I figured it would be straightforward. I have already done for loops, so I have something to build off of. The language reference also specifies the pseudo-code for async for, so I really didn't have to think too much about what the unravelled form should be.

But then I decided I would break with my practice of using Python 3.8 as the reference version of Python I work from and instead pull from Python 3.10 due to two additions in that version: aiter() and anext(). That decision turned out to complicate my life a bit. 😉

Source: Unravelling async for loops, an article by Brett Cannon.

How to Crawl the Web with Scrapy

Web scraping is the process of downloading data from a public website. For example, you could scrape ESPN for stats of baseball players and build a model to predict a team’s odds of winning based on their players stats and win rates. Below are a few use-cases for web scraping.

  • Monitoring the prices of your competitors for price matching (competitive pricing).
  • Collecting statistics from various websites to create a dashboard e.g. COVID-19 dashboards.
  • Monitoring financial forums and twitter to calculate sentiment for specific assets.

One use-case I will demonstrate is scraping the website indeed.com for job postings. Let’s say you are looking for a job but you are overwhelmed with the number of listings. You could set up a process to scrape indeed every day. Then you can write a script to automatically apply to the postings that meet certain criteria.

Source: How to Crawl the Web with Scrapy, an article by Matt Bass.

A good old-fashioned Perl log analyzer

A recent Lobsters post lauding the virtues of AWK reminded me that although the language is powerful and lightning fast, I usually find myself exceeding its capabilities and reaching for Perl instead. One such application is analyzing voluminous log files such as the ones generated by this blog. Yes, WordPress has stats, but I’ve never let reinvention of the wheel get in the way of a good programming exercise.

Source: A good old-fashioned Perl log analyzer, an article by Mark Gardner.

How to write about web performance

I’ve been writing about performance for a long time. I like to think I’ve gotten pretty good at it, but sometimes I look back on my older blog posts and cringe at the mistakes I made.

This post is an attempt to distill some of what I’ve learned over the years to offer as advice to other aspiring tinkerers, benchmarkers, and anyone curious about how browsers actually work when you put them to the test.

Source: How to write about web performance, an article by Nolan Lawson.

Designing Beautiful Shadows in CSS

In my humble opinion, the best websites and web applications have a tangible “real” quality to them. There are lots of factors involved to achieve this quality, but shadows are a critical ingredient.

When I look around the web, though, it's clear that most shadows aren't as rich as they could be. The web is covered in fuzzy grey boxes that don't really look much like shadows.

In this tutorial, we'll learn how to transform typical box-shadows into beautiful, life-like ones.

Source: Designing Beautiful Shadows in CSS, an article by Joshua Comeau.

The hidden performance overhead of Python C extensions

Python is slow, and compiled languages like Rust, C, or C++ are fast. So when your application is too slow, rewriting some of your code in a compiled extension can seem like the natural approach to speeding things up.

Unfortunately, compiled extensions are sometimes actually slower than the equivalent Python code. And even when they’re faster, the performance improvement might be far less than you’d imagine, due to hidden overhead caused by two factors:

  1. Function call overhead.
  2. Serialization/deserialization overhead.

Let’s see where these hidden performance overheads comes from, and then see some solutions to get around them.

Source: The hidden performance overhead of Python C extensions, an article by Itamar Turner-Trauring.

I18n in Go: Managing Translations

Recently I've been building a fully internationalized (i18n) and localized (l10n) web application for the first time with Go's golang.org/x/text packages. I've found that the packages and tools that live under golang.org/x/text are really effective and well designed, although it's been a bit of a challenge to figure out how to put it all together in a real application.

In this tutorial I want to explain how you can use golang.org/x/text packages to manage translations in your application. Specifically:

  • How to use the golang.org/x/text/language and golang.org/x/text/message packages to print translated messages from your Go code.
  • How to use the gotext tool to automatically extract messages for translation from your code into JSON files.
  • How to use gotext to parse translated JSON files and create a catalog containing translated messages.
  • How to manage variables in messages and provided pluralized versions of translations.

Source: I18n in Go: Managing Translations, an article by Alex Edwards.

Quadratic algorithms are slow (and hashmaps are fast)

Hello! I was talking to a friend yesterday who was studying for a programming interview and trying to learn some algorithms basics.

The topic of quadratic-time vs linear-time algorithms came up, I thought this would be fun to write about here because avoiding quadratic-time algorithms isn’t just important in interviews – it’s sometimes good to know about in real life too! I’ll explain what a “quadratic-time algorithm is” in a minute :)

here are the 3 things we’ll talk about:

  1. quadratic time functions are WAY WAY WAY slower than linear time functions
  2. sometimes you can make a quadratic algorithm into a linear algorithm by using a hashmap
  3. this is because hashmaps lookups are very fast (instant!)

I’m going to try to keep the math jargon to a minimum and focus on real code examples and how fast/slow they are.

Source: Quadratic algorithms are slow (and hashmaps are fast), an article by Julia Evans.

Are Dockerfiles good enough?

Containers have quickly become the favorite way to deploy software, for a lot of good reasons. They have allowed, for the first time, developers to test "as close to production" as possible. Unlike say, VMs, containers have a minimal performance hit and overhead. Almost all of the new orchestration technology like Kubernetes relies on them and they are an open standard, with a diverse range of corporate rulers overseeing them. In terms of the sky-high view, containers have never been in a better place.

I would argue though that in our haste to adopt this new workflow, we missed some steps. To be clear, this is not to say containers are bad (they aren't) or that they aren't working correctly (they are working mostly as advertised). However many of the benefits to containers aren't being used by organizations correctly, resulting in a worse situation than before. While it is possible to use containers in a stable and easy-to-replicate workflow across a fleet of servers, most businesses don't.

Source: Are Dockerfiles good enough?, an article by Mathew Duggan.

An Introduction to Type Level Programming

In this article you’ll learn how to build programs that make heavy use of type-level programming by working through building a theming system. I originally developed the ideas behind this talk and article when trying to write something to unify the various themes and configurations for my own xmonad desktop setup, but the theming system you’ll build as you work through this article can be equally applied to theming web content, desktop or command line applications, or really anything that needs configurable theming.

Source: An Introduction to Type Level Programming, an article by Rebecca Skinner.

Ten Things I Look For In a Code Review

Feedback is critical in any engineering organization – and that feedback often comes through code reviews. Junior engineers learn how to manage complexity, simplify the logic, and to develop the codebase from senior engineers. But, on the other hand, even the most senior engineers benefit from having a second pair of eyes on their code.

Yet, very few organizations set standards around their code reviews. By using a checklist, you can increase code quality across the entire organization. Better yet, it serves as an excellent onboarding document to train new reviewers, expanding the pool of reviewers and expediting the review pipeline.

I've compiled a starting point of 10 questions to ask when reviewing code.

Source: Ten Things I Look For In a Code Review, an article by Matt Rickard.