week 11, 2021

Profiling Python code with py-spy

If you have a Python program that is currently running you may want to understand what the real-world performance profile of the code is. This program could be in a production environment or just on your local machine. You will want to understand where the running program spends its time and if any “hot spots” exist that should be investigated further for improvement. You may be dealing with a production system that is misbehaving and you may want to profile it in an unobtrusive way that doesn’t further impact production performance or require code modifications. What’s a good way to do this? This article will talk about py-spy, a tool that allows you to profile Python programs that are already running.

Source: Profiling Python code with py-spy, an article by Matt Wright.

Writing a Modern HTTP(S) Tunnel in Rust

This post is for anyone interested in writing performant and safe applications in Rust quickly. It walks the reader through designing and implementing an HTTP Tunnel and basic, language-agnostic, principles of creating robust, scalable, observable, and evolvable network applications.

Source: Writing a Modern HTTP(S) Tunnel in Rust, an article by Eugene Retunsky.

Heterometrus silenus eating

In the evening I noticed that Adam's scorpion, a Heterometrus silenus, was out of hiding and wandering around in its enclosure clearly looking for food. So I carefully removed the two enclosures resting atop the lid and then removed the lid and gave the scorpion a mealworm with a pair of tweezers. It grabbed the tweezers instead of the mealworm and the mealworm dropped on the substrate.

Heterometrus silenus eating a mealworm (Tenebrio molitor)
Heterometrus silenus eating a mealworm (Tenebrio molitor).

But because the mealworm moved slightly against the first right leg of the scorpion it started to feel around with its claws and finally found the nice snack.

Speeding up SQL queries by orders of magnitude using UNION

One of the most common cases where SQL query performance can degrade significantly is in a diamond shaped schema, where there are multiple ways of joining two tables together. In such a schema, a query is likely to use OR to join tables in more than one way, which eliminates the optimizer’s ability to create an efficient query plan.

Source: Speeding up SQL queries by orders of magnitude using UNION, an article by Ben Levy and Christian Charukiewicz.

Arenas in Rust

An arena is essentially a way to group up allocations that are expected to have the same lifetime. Sometimes you need to allocate a bunch of objects for the lifetime of an event, after which they can all be thrown away wholesale. It’s inefficient to call into the system allocator each time, and far more preferable to preallocate a bunch of memory for your objects, cleaning it all up at once once you’re done with them.

Source: Arenas in Rust, an article by Manish Goregaokar.

Deep Dive into Docker Internals - Union Filesystem

Working with Docker CLI is very straightforward - you just build, run, inspect, pull and push containers and images, but have you ever wondered how do the internals behind this Docker interface actually work? Behind this simple interface hides a lot of cool technologies and in this article we will explore one of them - the union filesystem - the underlying filesystem behind all the container and image layers...

Source: Deep Dive into Docker Internals - Union Filesystem, an article by Martin Heinz

Cleaning a late 2014 Mac mini

Just before six p.m. I decided to pop open my late 2014 Mac mini to see if I could clean it inside with compressed air. Its fan had been spinning for quite some time non-stop. And even though I had use compressed air on the outside I had the feeling there was still a lot of dust inside and the reason for the fan still spinning noisily.

Dusty inside of a Mac mini late 2014
Dusty inside of a Mac mini late 2014.

I used a plastic gift card to pry open the bottom. A task that was much easier than I had thought; the plastic lid popped off very easy. And then I noticed how much dust bunnies had gathered inside the Mac mini... I could pick up most of it with my hand and used compressed air to blow away the rest.

When I reconnected the Mac mini and turned it on... silence. The dust was really causing the Mac to overheat, and maybe unstable as well, and kept the fan spinning.

Exploiting machine learning pickle files

Many machine learning (ML) models are Python pickle files under the hood, and it makes sense. The use of pickling conserves memory, enables start-and-stop model training, and makes trained models portable (and, thereby, shareable). Pickling is easy to implement, is built into Python without requiring additional dependencies, and supports serialization of custom objects. There’s little doubt about why choosing pickling for persistence is a popular practice among Python programmers and ML practitioners.

Source: Never a dill moment: Exploiting machine learning pickle files, an article by Evan Sultanik.

Reverse Engineering a Docker Image

This started with a consulting snafu: Government organisation A got government organisation B to develop a web application. Government organisation B subcontracted part of the work to somebody. Hosting and maintenance of the project was later contracted out to a private-sector company C. Company C discovered that the subcontracted somebody (who was long gone) had built a custom Docker image and made it a dependency of the build system, but without committing the original Dockerfile. That left company C with a contractual obligation to manage a Docker image they had no source code for. Company C calls me in once in a while to do various things, so doing something about this mystery meat Docker image became my job.

Source: Reverse Engineering a Docker Image, an article by Simon Arneaud.

Do Not Use '>' in Your Command Prompt

Over the years of troubleshooting performance problems in the Unix/Linux world, I have seen multiple cases where a regularly used command line tool in a customer server just stops working for some reason. The tool just returns immediately, doing absolutely nothing. No output printed, no coredumps and the exit code is zero (success!).

This article walks you through a couple of such incidents and in the end I explain how I avoid accidentally doing bad stuff in production in general.

Source: Do Not Use '>' in Your Command Prompt (and How to Stay Safe in Shell), an article by Tanel Põder.

Making lunch faster with Python concurrency

I'm going to tell a story to explain the different concurrency and parallelism options in Python.

In this story we will see how a single person multitasking is like concurrency, and multiple people doing their own tasks is like parallelism. We'll watch these scenarios in action at some well known lunchtime restaurants as they quickly and efficiently serve their customers. Then I'll implement these restaurants in Python, and finally I'll compare the different concurrency options and explain when to use each one.

I'll explain:

  • What the difference is between concurrency and parallelism?
  • The different concurrency options and how they compare, including threading, asyncio, multiprocessing and cloud functions
  • The pro and cons of each concurrency option
  • How to select your concurrency option using a single flowchart

Source: Making lunch faster with Python concurrency, an article by Brendan Maginnis.

Getting started with … Rust

In this series, we look at the most loved languages according to the Stack Overflow developer survey, the spread and use cases for each of them and collect some essential links on how to get into them. First up: Rust.

Source: Getting started with ... Rust, an article by Medi Madelen Gwosdz.

Rust vs. Go: Why They’re Better Together

While others may see Rust and Go as competitive programming languages, neither the Rust nor the Go teams do. Quite the contrary, our teams have deep respect for what the others are doing, and see the languages as complimentary with a shared vision of modernizing the state of software development industry-wide.

In this article, we will discuss the pros and cons of Rust and Go and how they supplement and support each other, and our recommendations for when each language is most appropriate.

Source: Rust vs. Go: Why They’re Better Together, an article by Jonathan Turner and Steve Francia.

Ready Player One

A world at stake. A quest for the ultimate prize. Are you ready?

In the year 2045, reality is an ugly place. The only time Wade Watts really feels alive is when he’s jacked into the OASIS, a vast virtual world where most of humanity spends their days.

When the eccentric creator of the OASIS dies, he leaves behind a series of fiendish puzzles, based on his obsession with the pop culture of decades past. Whoever is first to solve them will inherit his vast fortune—and control of the OASIS itself.

Then Wade cracks the first clue. Suddenly he’s beset by rivals who’ll kill to take this prize. The race is on—and the only way to survive is to win.

In the evening I started in Ready Player One by Ernest Cline.

How to Read ARM64 Assembly Language

ARM64 is a computer architecture that competes with the popular Intel x86-64 architecture used for the CPUs in desktops, laptops, and so on. ARM64 is common in mobile phones, as well as Graviton-based Amazon EC2 instances and the much ballyhooed Apple M1 chips, so knowing about it might be useful! In fact, I have almost certainly spent more time with ARM64 than x86-64 because of the iPhone.

Source: How to Read ARM64 Assembly Language, an article by Scott Wolchok.