A couple months ago, we faced a question many young startups
face. Should we rewrite our system in Rust?
At the time of the decision, we were a Go and Python shop. The tool
we’re building passively watches API traffic to provide “one-click,”
API-centric visibility, by analyzing the API traffic. Our users run
an agent that sends API traffic data to our cloud for analysis. Our
users were using us to watch more and more traffic in staging and
production—and they were starting to complain about the memory
usage.
This led me to spend 25 days in the depths of despair and the
details of Go memory management, trying to get our memory footprint
to an acceptable level. This was no easy feat, as Go is a
memory-managed language with limited ability to tune garbage
collection.
When the first time I saw the WWDC presentation about actors, I was
thrilled with what it is capable of and how it will change the way
we write asynchronous code in the near future. By using actors,
writing asynchronous code that is free from data
races and
deadlocks has never been easier.
All that aside, that doesn’t mean that actors are free from
threading issues. If we are not careful enough, we might
accidentally introduce a reentrancy problem when using actors.
The perlcritic tool is often your first
defense against “awkward, hard to read, error-prone, or
unconventional constructs in your code,” per its
description. It’s
part of a class of programs historically known as
linters, so-called
because like a clothes dryer machine’s lint trap, they “detect small
errors with big effects.” (Another such linter is
perltidy, which I’ve
referenced in the past.)
A coredump is a snapshot of a process’s memory that is usually
created by the kernel when a crash happens. These can be fairly
helpful to find out which part of the code broke by looking at the
backtrace or finding any kind of corruption by introspecting the
memory itself. Unfortunately it can be a bit tedious to work with
these. This article aims to give an overview over helpful tools &
tricks to leverage the full power of coredumps on Nix-based systems.
Plotting is an essential component of data analysis. As a data
scientist, I spend a significant amount of my time making simple
plots to understand complex data sets (exploratory data analysis)
and help others understand them (presentations).
In particular, I make a lot of bar charts (including histograms),
line plots (including time series), scatter plots, and density plots
from data in Pandas data
frames. I
often want to facet these on various categorical variables and layer
them on a common grid.
’ve long been a fan of filtering at the DNS level. The approach that
I’ll outline here is similar to what our devices did at Luma. It is
not perfect by any means, but if it is setup well then it provides a
chokepoint that has a solid return on investment for the amount of
time and effort that it takes to standup. It isn’t going to keep any
sophisticated actors at bay, but it does demonstrably improve
performance and has a good chance of reducing the headaches of
administrating a network and the systems on it for a connected
family.
Python 3.10, which is due out in early October 2021, will include a
large new language feature called structural pattern
matching. This article is a critical but (hopefully) informative
presentation of the feature, with examples based on real-world code.
Arithmetic coding is a common algorithm used in both lossless and
lossy data compression algorithms.
It is an entropy encoding technique, in which the frequently seen
symbols are encoded with fewer bits than rarely seen symbols. It has
some advantages over well-known techniques such as Huffman
coding. This article will describe the
CACM87
implementation of arithmetic coding in detail, giving you a good
understanding of all the details needed to implement it.
In the evening I finished The Broken
Eye,
Lightbringer Book 3 by Brent Weeks. While not as good as the previous
book in the Lightbringer series I liked The Broken Eye a lot. Recommended.
Behind the well-known U.S. security organizations—the FBI and CIA
among them—lies a heavily guarded, anonymous government agency
dedicated to intelligence surveillance and to a highly specialized
brand of citizen protection.
Shock waves of alarm ripple through the clandestine agency when
Washington, D.C., police detective Ryan Kessler inexplicably becomes
the target of Henry Loving, a seasoned, ruthless “lifter” hired to
obtain information using whatever means necessary. While Loving is
deft at torture, his expertise lies in getting an “edge” on his
victim—leverage—usually by kidnapping or threatening family until
the “primary” caves under pressure
In the evening I started in
Edge
by Jeffery Deaver.
This is not a tutorial. If you're looking for a quick, easy, "how to
use git" kind of post, look elsewhere.
The goal of this post is to give you just enough understanding of
the git internals that you can build up a correct intuition of what
various git commands actually do under the hood.
After several attempts and tweets at Google I/O, I started wanting
to make my own notes into something more visually appealing and
presentable so that I can share my notes with everybody else who
want to learn the same subjects.
Apple’s default screensavers for Mac OS X are nice, but they get
boring after a while. Any true nerd would take matters into their
own hands when that happens!
If you want to make your own custom screensaver for Mac OS X, this
article is for you. We’ll walk through a tutorial for making a
screensaver which emulates the game
Pong. It’s going to be super
simple — this is just to get you started with how to make a
screensaver.
In this article you’ll find some tips to reduce the amount of RAM
used when working with pandas, the
fundamental Python library for data analysis and data manipulation.
In this article you will find 42 recommendations about coding in C++
that can help a programmer avoid a lot of errors, save time and
effort. The author is Andrey Karpov - technical director of "Program
Verification Systems", a team of developers, working on PVS-Studio
static code analyzer. Having checked a large number of open source
projects, we have seen a large variety of ways to shoot yourself in
the foot; there is definitely much to share with the readers. Every
recommendation is given with a practical example, which proves the
currentness of this question. These tips are intended for C/C++
programmers, but usually they are universal, and may be of interest
for developers using other languages.
And then there’s Podman: Podman is a reimplemented, compatible
version of the Docker CLI and API. It does not however implement all
the BuildKit Dockerfile extensions. On its own, then, Podman isn’t
as good as Docker at building images.
There is another option, however: BuildKit has its own build tool,
which is distinct from the traditional docker build, and this build
tool can work with Podman.
Let’s see where Podman currently is as far as BuildKit features, and
how to use BuildKit with Podman if that is not sufficient.
DM is a modern Python package manager with PEP
582 support. It installs
and manages packages in a similar way to npm that doesn't need to
create a virtualenv at all!
awk is a powerful tool. It is actually a Turing-complete
language,
meaning that you can technically write any kind of program with
it. You could implement the classic sorting algorithms or more
complex things such as a parser or an interpreter. Examples of this
kind can be found in the “AWK Programming
Language” book
written by awk‘s authors. The reason awk is still popular today,
though, has nothing to do with its generality and more with its
usefulness working in the command line.
Hello! I was talking to a Python programmer friend yesterday about
debugging, and I mentioned that I really like debugging using a
REPL. He said he’d never tried it and that it sounded fun, so I
thought I’d write a quick post about it.
This debugging method doesn’t work in a lot of languages, but it
does work in Python and Ruby and kiiiiiind of in C (via gdb).
Images are a massive part of the Internet. On the median web page,
images account for 51% of the bytes
loaded, so any
improvement made to their speed or their size has a significant
impact on performance.
I’ve been using Go for a few years now, mostly in my open source
project Lazygit. In my day job I use Ruby and Typescript, and I’ve
also spent some time with Rust. Each of those languages have design
quirks that can grind a developer’s gears, and although my own
precious gears have been ground by every language I’ve used, Go is
the only language that has made me feel indignant.
Years ago, OpenBSD gained two new security system calls,
pledge(2) (originally
tame(2))
and unveil. In both, an
application surrenders capabilities at run-time. The idea is to
perform initialization like usual, then drop capabilities before
handling untrusted input, limiting unwanted side effects. This
feature is applicable even where type safety isn’t an issue, such as
Python, where a program might still get tricked into accessing
sensitive files or making network connections when it shouldn’t. So
how can a Python program access these system calls?
In this post, I want to look at the biggest strengths and weaknesses
of Python, with more emphasis on the weaknesses, just because these
problems have been there for years now and some of the rough edges
bleed a lot.
When I decided the next post in my series on Python's syntactic
sugar would be on async for, I figured it would be straightforward. I have already done
for loops, so I
have something to build off of. The language reference also
specifies the pseudo-code for async for,
so I really didn't have to think too much about what the unravelled
form should be.
But then I decided I would break with my practice of using Python
3.8 as the reference version of Python I work from and instead pull
from Python 3.10 due to two additions in that version:
aiter()
and
anext(). That
decision turned out to complicate my life a bit. 😉
Web scraping is the process of downloading data from a public
website. For example, you could scrape ESPN for stats of baseball
players and build a model to predict a team’s odds of winning based
on their players stats and win rates. Below are a few use-cases for
web scraping.
Monitoring the prices of your competitors for price matching
(competitive pricing).
Collecting statistics from various websites to create a dashboard
e.g. COVID-19 dashboards.
Monitoring financial forums and twitter to calculate sentiment for
specific assets.
One use-case I will demonstrate is scraping the website indeed.com
for job postings. Let’s say you are looking for a job but you are
overwhelmed with the number of listings. You could set up a process
to scrape indeed every day. Then you can write a script to
automatically apply to the postings that meet certain criteria.
Over the weekend, a reader of my blog contacted me basically asking
about firewalls. Firewalls themselves are boring in my opinion, so
let’s talk about something Alpine can do that, as far as I know, no
other distribution can easily do out of the box yet: service
isolation using the base networking stack itself instead of
netfilter.
A recent Lobsters
post
lauding the virtues of AWK
reminded me that although the language is powerful and lightning
fast, I usually find myself exceeding its capabilities and reaching
for Perl instead. One such application is analyzing voluminous log
files such as the ones generated by this
blog. Yes, WordPress has
stats, but I’ve never let
reinvention of the wheel get in the way of a good programming
exercise.
I’ve been writing about performance for a long time. I like to think
I’ve gotten pretty good at it, but sometimes I look back on my older
blog posts and cringe at the mistakes I made.
This post is an attempt to distill some of what I’ve learned over
the years to offer as advice to other aspiring tinkerers,
benchmarkers, and anyone curious about how browsers actually work
when you put them to the test.
Writing functional code is often backwards and can feel more like
solving puzzles than like explaining a process to the computer. In
functional languages I often know what I want to say, but it feels
like I have to solve a puzzle in order to express it to the
language. Functional programming is just a bit too weird.
In my humble opinion, the best websites and web applications have a
tangible “real” quality to them. There are lots of factors involved
to achieve this quality, but shadows are a critical ingredient.
When I look around the web, though, it's clear that most shadows
aren't as rich as they could be. The web is covered in fuzzy grey
boxes that don't really look much like shadows.
In this tutorial, we'll learn how to transform typical box-shadows
into beautiful, life-like ones.
This blogpost describes a DoS vulnerability in Haskell's aeson
package. We have followed appropriate procedure for responsible
disclosure but the problem was not fixed, so now we are releasing
this to the public in the hope that it may still be fixed afterall.
Python is slow, and compiled languages like Rust, C, or C++ are
fast. So when your application is too slow, rewriting some of your
code in a compiled extension can seem like the natural approach to
speeding things up.
Unfortunately, compiled extensions are sometimes actually slower
than the equivalent Python code. And even when they’re faster, the
performance improvement might be far less than you’d imagine, due to
hidden overhead caused by two factors:
Function call overhead.
Serialization/deserialization overhead.
Let’s see where these hidden performance overheads comes from, and
then see some solutions to get around them.
There are a few things I think would be good to see implemented in
and around Emacs, but I don’t know if I will find any time soon to
do so properly. There is no pride in hiding ideas one thinks might
be worthwhile, so I’d like to share them here.
Recently I've been building a fully internationalized (i18n) and localized (l10n) web application for the first time with Go's golang.org/x/text packages. I've found that the packages and tools that live under golang.org/x/text are really effective and well designed, although it's been a bit of a challenge to figure out how to put it all together in a real application.
In this tutorial I want to explain how you can use golang.org/x/text packages to manage translations in your application. Specifically:
How to use the golang.org/x/text/language and golang.org/x/text/message packages to print translated messages from your Go code.
How to use the gotext tool to automatically extract messages for
translation from your code into JSON files.
How to use gotext to parse translated JSON files and create a
catalog containing translated messages.
How to manage variables in messages and provided pluralized
versions of translations.
Hello! I was talking to a friend yesterday who was studying for a
programming interview and trying to learn some algorithms basics.
The topic of quadratic-time vs linear-time algorithms came up, I
thought this would be fun to write about here because avoiding
quadratic-time algorithms isn’t just important in interviews – it’s
sometimes good to know about in real life too! I’ll explain what a
“quadratic-time algorithm is” in a minute :)
here are the 3 things we’ll talk about:
quadratic time functions are WAY WAY WAY slower than linear time
functions
sometimes you can make a quadratic algorithm into a linear
algorithm by using a hashmap
this is because hashmaps lookups are very fast (instant!)
I’m going to try to keep the math jargon to a minimum and focus on
real code examples and how fast/slow they are.
Luke Skywalker joins forces with a Jedi Knight, a cocky pilot, a
Wookiee and two droids to save the galaxy from the Empire's
world-destroying battle station, while also attempting to rescue
Princess Leia from the mysterious Darth Vader.
Containers have quickly become the favorite way to deploy software,
for a lot of good reasons. They have allowed, for the first time,
developers to test "as close to production" as possible. Unlike say,
VMs, containers have a minimal performance hit and overhead. Almost
all of the new orchestration technology like Kubernetes relies on
them and they are an open standard, with a diverse range of
corporate rulers overseeing them. In terms of the sky-high view,
containers have never been in a better place.
I would argue though that in our haste to adopt this new workflow,
we missed some steps. To be clear, this is not to say containers are
bad (they aren't) or that they aren't working correctly (they are
working mostly as advertised). However many of the benefits to
containers aren't being used by organizations correctly, resulting
in a worse situation than before. While it is possible to use
containers in a stable and easy-to-replicate workflow across a fleet
of servers, most businesses don't.
Our industry has made incredible strides in the past decade, thanks
in part to technologies like Docker, Docker Compose, and
Kubernetes. However, we are still figuring out how to do development
in the heterogeneous environments in which we live.
In this article you’ll learn how to build programs that make heavy
use of type-level programming by working through building a theming
system. I originally developed the ideas behind this talk and
article when trying to write something to unify the various themes
and configurations for my own xmonad desktop setup, but the theming
system you’ll build as you work through this article can be equally
applied to theming web content, desktop or command line
applications, or really anything that needs configurable theming.
As I noted in my last
blog,
I have been working on a set of tools which enable the building of
so-called “distroless” images based on Alpine. These tools have now
evolved to a point where they are usable for testing in lab
environments, thus I am happy to announce the witchery
project.
Feedback is critical in any engineering organization – and that
feedback often comes through code reviews. Junior engineers learn
how to manage complexity, simplify the logic, and to develop the
codebase from senior engineers. But, on the other hand, even the
most senior engineers benefit from having a second pair of eyes on
their code.
Yet, very few organizations set standards around their code
reviews. By using a checklist, you can increase code quality across
the entire organization. Better yet, it serves as an excellent
onboarding document to train new reviewers, expanding the pool of
reviewers and expediting the review pipeline.
I've compiled a starting point of 10 questions to ask when reviewing
code.