a tumblelog

Overview of python dependency management tools

Totally confused by all the tools for managing dependencies? Pip, venv, Docker, conda, virtualenvwrapper, pipenv, … Which one should you use? Why do we even have all these different tools? Can they work together?

No wonder. The world of Python dependency management is a mess, but once you understand the tools and why they exist, it’s going to be easier to choose the one you want and deal with the others in environments where you can’t choose your favorite ones.

Source: Overview of python dependency management tools, an article by Mario Kostelac.

Taking images in Docker container

Have you ever tried to click photos with your camera in the laptop, of course, you did, but here is a twist, we will click the photo in a Docker container, many of you may be wondering how you can do that? Well, let me get started with this interesting thing.

Source: Taking images in Docker container, an article by Seema Saharan.

mDNS, avahi and docker non-root containers

mDNS is a protocol that resolves hostname to IP addresses within a small network that lacks the name server. By default, mDNS exclusively resolves hostnames ending with .local. But there will be a problem with the hosts that implement .local doesn't support mDNS protocol and can be found via a conventional unicast DNS sever. In those cases, necessary network configuration should be changed.

Source: mDNS, avahi and docker non-root containers, an article by Gnanesh Kunal.

Principal Component Analysis

Principal components analysis (PCA) is one of a family of techniques for taking high-dimensional data, and using the dependencies between the variables to represent it in a more tractable, lower-dimensional form, without losing too much information. It has been widely used for data compression and de-noising. However, its entire mathematical process is sometimes ambiguous to the user.

In this article, I would like to discuss the entire process of PCA mathematically, including PCA projection and reconstruction, with most of the derivations and proofs provided. At the end of the article, I implemented PCA projection and reconstruction from scratch. After reading this article, there should be more black box in PCA anymore.

Source: Principal Component Analysis, an article by Lei Mao.

WordPress passwords, explained and cracked

True or not, a strong password hashing is crucial for a large ecosystem like the WordPress one, which has always been a juicy target for hackers. So, I decided to take a closer look at the hashing system and try to crack WordPress hashes from scratch!

Source: WordPress passwords, explained and cracked, an article by Francesco Carlucci.

Getting started with Kubernetes

In this multi-part series of articles about Kubernetes, I'll try and capture what I think everyone who wants to learn and work with Kubernetes should know about.

Source: Getting started with Kubernetes, an article by Peter Jausovec.

On Marketing Haskell

In the last year, the Haskell language and associated technology have been seen developing into the most mature ecosystem we’ve seen to date, with innovation happening left and right across a variety of fronts. Editor tooling, for instance, is reaching levels of maturity we only dreamed about years ago. Simultaneously, there has been a fair bit of discussion about the economics of the Haskell ecosystem and the confounding factors that have led to its potential stagnation. Most recently, for instance, there have been discussions about “Simple Haskell” as a set of best practices to spur more successful industry projects.

Source: On Marketing Haskell, an article by Stephen Diehl.

Global CSS options with custom properties

I’ve been toying around with some ideas for how to use custom properties (aka CSS variables) for global settings in a project. The idea is to provide control to designers/developers over consistent styles across multiple components.

Source: Global CSS options with custom properties, an article by Mark Otto.

What's in a parser combinator?

One of the interesting facts about writing your own parser combinators library, is that you will learn (or consolidate) other knowledges in the process, like: Functors, Applicatives and, of course, Monads, and more generaly, how to design DSL in Haskell.

Source: What's in a parser combinator?

How Not to Mock Web Requests for Unit Testing

So, how do you test the functionality of how your app responds to a web request without making an actual request and returning a response. One approach is to mock the requests and responses.

Source: How Not to Mock Web Requests for Unit Testing.

Nix Flakes, Part 1: An introduction and tutorial

This is the first in a series of blog posts intended to provide a gentle introduction to flakes, a new Nix feature that improves reproducibility, composability and usability in the Nix ecosystem. This blog post describes why flakes were introduced, and give a short tutorial on how to use them.

Source: Tweag - Nix Flakes, Part 1: An introduction and tutorial, an article by Eelco Dolstra.

Matrix Calculus for DeepLearning (Part1)

Deep Learning is all about linear algebra and calculus. If you try to read any deep learning paper, matrics calculus is a needed component to understanding the concept.

Source: Matrix Calculus for DeepLearning (Part1), an article by Kiran U Kamath.

Hypermodern Python

This article series is a guide to modern Python tooling with a focus on simplicity and minimalism. It walks you through the creation of a complete and up-to-date Python project structure, with unit tests, static analysis, type-checking, documentation, and continuous integration and delivery.

Source: Hypermodern Python, an article by Claudio Jolowicz.

The PEPs of Python 3.9

With the release of Python 3.9.0b1, the first of four planned betas for the development cycle, Python 3.9 is now feature-complete. There is still plenty to do in terms of testing and stabilization before the October final release. The release announcement lists a half-dozen Python Enhancement Proposals (PEPs) that were accepted for 3.9. We have looked at some of those PEPs along the way; there are some updates on those. It seems like a good time to fill in some of the gaps on what will be coming in Python 3.9

Source: The PEPs of Python 3.9, an article by Jake Edge.

Feeding a Caribena versicolor sling and a tiny scorpion

Today I was finally able to feed the Caribena versicolor sling that has been in my care since the first of this month. I held a small mealworm, Tenebrio molitor, with tweezers close to it, and it "jumped" on the small prey item. In the past the small tarantula had refused food items of the same size, no idea why. Maybe still getting used to its enclosure.

Caribena versicolor with prey
Caribena versicolor with prey: a small Tenebrio molitor larva.

I was also finally able to feed a second instar Chaerilus sp. "Java"; a very small scorpion that I have been keeping since the 7th of April 2020. While it has small springtails in its enclosure, which maybe it eats, I prefer to actually see it eat. So in the late afternoon I managed, after a few attempts, to start it accepting and eating a very tiny mealworm larva.

Diving into Go by building a CLI application

In this blog post we’ll build a CLI application in Go, which we’ll call go-grab-xkcd. This application fetches comics from XKCD and provides you with various options through command-line arguments.

Source: Diving into Go by building a CLI application.

Julia vs. Python: Which is best for data science?

Among the many use cases Python covers, data analytics has become perhaps the biggest and most significant. The Python ecosystem is loaded with libraries, tools, and applications that make the work of scientific computing and data analysis fast and convenient.

But for the developers behind the Julia language — aimed specifically at “scientific computing, machine learning, data mining, large-scale linear algebra, distributed and parallel computing”—Python isn’t fast or convenient enough. Python represents a trade-off, good for some parts of data analytics work but terrible for others.

Source: Julia vs. Python: Which is best for data science?, an article by Serdar Yegulalp.

How Linux pipes work under the hood

Piping is one of the core concepts of Linux & Unix based operating systems. Pipes allow you to chain together commands in a very elegant way, passing output from one program to the input of another to get a desired end result.

Source: How Linux pipes work under the hood, an article by Brandon Wamboldt.

Tarantula molts and feeding

In the afternoon I noticed that the Psalmopoeus irminia sling I keep had molted; I saw the exoskeleton dangling from a piece of moss coming out of the cork tube it lives in. Because I had an appointment I couldn't take photos, so that I did in the evening.

Psalmopoeus irminia molt
The exoskeleton of a Psalmopoeus irminia.

In the evening, after I had taken the above photo I also spotted a cast-off exoskeleton in the terrarium in which I keep a Pterinochilus murinus sling. On the 12th of this month it had already opened its burrow, and I suspected back then that it had molted. And now I had proof. Maybe I overlooked the exoskeleton earlier because it was underneath the leaf of a plastic plant.

Pterinochilus murinus molt
The (partial) exoskeleton of a Pterinochilus murinus.

I also checked on the Chromatopelma cyaneopubescens sling I keep, which recently also molted. Because it moved out of its webbing I decided to try to feed it, and it readily accepted a pre-killed mealworm, Tenebrio molitor.

5 Types Of ZSH Aliases You Should Know

I use Docker, Kubernetes, and Microsoft Azure every day. That said, it makes sense for me to have aliases supporting me with these tools and environments. However, maybe you are using different clouds and command-line tools so that you will end up with different aliases. The key takeaway should be that you create and use aliases to help you get your job done.

Source: 5 Types Of ZSH Aliases You Should Know, an article by Thorsten Hans.

Better git diffs with FZF

Sometimes I find the git diff command a little inconvenient. It can throw a lot of information at the screen at once. I use git diff not only for verifying my changes before a commit, but also to review pull requests, or for finding bugs introduced between two commits. In the situations when you’re looking at a lot of changed files, having to scroll up and down so much is tedious.

Source: Better git diffs with FZF, an article by Rafael Mendiola.

Mocking time and testing event loops in Go

Initially I wanted to write articles on those two topics separately (mocking time and testing event loops), but during the process I realized that the things I want to talk about are too interrelated: when I need to mock time, it's usually to test some event loop with it, and when I test event loops, typically mocked time is also involved in that.

So in the end, it felt better to just combine all that in a single article.

Source: Mocking time and testing event loops in Go, an article by Dmitry Frank.

CSS Basics for Typography

In 2020 there are a lot of developers and designers who want to learn the basics of CSS. In this series of articles, I will teach you those main topics. In this specific article, I will review the essential CSS properties of typography while using many visual examples.

Source: CSS Basics for Typography, an article by Elad Shechter.

Creating and Modifying PDF Files in Python

The PDF, or Portable Document Format, is one of the most common formats for sharing documents over the Internet. PDFs can contain text, images, tables, forms, and rich media like videos and animations, all in a single file.

This abundance of content types can make working with PDFs difficult. There are a lot of different kinds of data to decode when opening a PDF file! Fortunately, the Python ecosystem has some great packages for reading, manipulating, and creating PDF files.

Source: Creating and Modifying PDF Files in Python, an article by David Amos.

Latency in Asynchronous Python

This week I was debugging a misbehaving Python program that makes significant use of Python’s asyncio. The program would eventually take very long periods of time to respond to network requests. My first suspicion was a CPU-heavy coroutine hogging the thread, preventing the socket coroutines from running, but an inspection with pdb showed this wasn’t the case. Instead, the program’s author had made a couple of fundamental mistakes using asyncio. Let’s discuss them using small examples.

Source: Latency in Asynchronous Python, an article by Chris Wellons.

Five tips to be a more effective command-line user

This article intends to be helpful for those who are already command line (CLI) users. Complete beginners are of course encouraged to read on, even though they may not grasp all the advantages immediately and perhaps there is a lot of other more important things to learn when starting. On the other hand, I expect long time CLI users to already work similarly. I do hope they might also find interesting tricks to adopt.

Source: Five tips to be a more effective command-line user, an article by Cosimo Streppone.

maybedoer: the Maybe Monoid for Go

This file is used to sequence functions that could fail together, allowing you to avoid doing an if err != nil check on every single fallible function call.

Source: maybedoer: the Maybe Monoid for Go, an article by Christine Dodrill.

Optional JSON fields in Go

One common kind of data stored in a configuration file is options. In this post, I'll talk about some nuances we have to be aware of when storing options in JSON and unmarshaling them to Go.

Specifically, the most important difference between options and any other data is that options are often, well... optional. Our program can have a large number of possible configuration options, but we may want to configure any particular invocation with only a subset - leaving all the others at their default values.

Source: Optional JSON fields in Go, an article by Eli Bendersky.

Bash Romance

It’s not a particularly new or exciting take to say that we’ve outgrown Bash. The Unix shell environment was novel and powerful in its day, but its age is really starting to show.

Source: Bash Romance.

Using make and git diff for a simple and powerful test harness

People regularly go for complex test harnesses; and when you’re in an ecosystem, they can be a good fit. But outside ecosystems, it’s actually quite easy to produce a fairly good test harness.

Source: Using make and git diff for a simple and powerful test harness, an article by Chris Morgan

Simple Haskell is Best Haskell

Almost two years ago, I moved from a cushy corporate Scala job to a riskier mid-stage startup using Haskell. This post summarises my thoughts about Haskell after experiencing it first hand.

Source: Simple Haskell is Best Haskell, an article by Sam Halliday.

Forget Bash, take Python!

I wrote bash scripts for several years now. You can find old bash scripts of mine on GitHub but most of them aren’t maintained anymore - and i have a reason for that.

Source: Forget Bash, take Python!, an article by Christoph Miksche.

Why is This Website Port Scanning me?

Recently, I was tipped off about certain sites performing localhost port scans against visitors, presumably as part of a user fingerprinting and tracking or bot detection. This didn't sit well with me, so I went about investigating the practice, and it seems many sites are port scanning visitors for dubious reasons.

Source: Why is This Website Port Scanning me, an article by Charlie Belmer.

How to Create a Threaded Web Scanner in Python

The great thing about Python is that it makes developers’ lives easy — import a couple of libraries to do the hard stuff for you, and you’re off to the races. This holds true when creating a threaded web scanner that’s capable of making multiple concurrent requests — with Python it’s easy to accomplish in a short amount of dev time.

Source: How to Create a Threaded Web Scanner in Python, an article by Kevin Dawe.

Python Caches Integers

An integer in Python is not a traditional 2, 4, or 8-byte implementation but rather it is implemented as an array of digits in base 230 which enables Python to support super long integers. Since there is no explicit limit on the size, working with integers in Python is extremely convenient as we can carry out operations on very long numbers without worrying about integer overflows. This convenience comes at a cost of allocation being expensive and trivial operations like addition, multiplication, division being inefficient.

Source: Python Caches Integers, an article by Arpit Bhayani.

Adam's Acanthoscurria geniculata arrived

Today, shortly after Adam returned from school, one of his birthday presents finally arrived; a female Acanthoscurria geniculata. I had ordered this tarantula the 6th of May, 2020 with Robert of Fatbob's Exotics:

  • 0.1 Acanthoscurria geniculata 2,5-3cm body €25

Together with shipping within the Netherlands it came down to €32. Originally, Adam wanted to have a female Brachypelma hamorii but when I contacted Robert the 5th of May the tarantula was no longer available.

Adam holding the vial with his female Acanthoscurria geniculata
Adam holding the vial with his female Acanthoscurria geniculata.

Due to the colder nights it took a while before Robert could ship the spider safely. But yesterday it got picked up by DPD and today it arrived. Not on time for Adam's birthday, which was the 12th of May, so he had been asking now and then "When is my tarantula coming?"

Female Acanthoscurria geniculata on Adam's hand
Female Acanthoscurria geniculata on Adam's hand.

For the terrarium I drilled plenty of holes in two sides of a plastic box and several more in the lid. Together with Adam I added plenty of substrate; slightly moist coco peat. I had also cut a cork tube in half, which Adam added, together with two plastic plants. Finally, I added a plastic bottle lid with water to provide a source of moisture to the spider.

Acanthoscurria geniculata in its terrarium
Acanthoscurria geniculata in its terrarium.

We tried to feed the tarantula a mealworm larva, Tenebrio molitor, but it was not interested. In my experience sometimes tarantulas accept food after travelling and sometimes they don't. Later in the afternoon when I checked the terrarium I noticed that she had been moving substrate out of the cork tube and made a nice pile next to it.

Close up of the female Acanthoscurria geniculata
Close up of the female Acanthoscurria geniculata.

In the evening Adam and I checked again, but she was still well hidden in her new burrow. We also tried to feed his Heterometrus petersii. While the scorpion did pick up the mealworm I put next to it, and even held it for a while, it was too stressed out because I had lifted the piece of wood it normally hides under to actually eat it. Adam and I decided to try again tomorrow. Adam is a little worried about his scorpion since it hasn't been eating since we bought it, the 23rd of February, 2020 at the Invertebrate Show in Harmelen.

We did leave a pre-killed mealworm next to the burrow entry of his tarantula, though. Let's see if it's gone tomorrow.

Python performance: it’s not just the interpreter

I have a particular view of Python performance from my experience on the Pyston project, and since this view is somewhat nonstandard I wanted to take some time to explain it and give a motivating example.

A common explanation for Python's low performance is that it is an interpreted language. In this post I hope to show that while the interpreter adds overhead, it is not the dominant factor for even a small microbenchmark. Instead we'll see that dynamic features -- particularly, features inside the runtime -- are to blame.

Source: Python performance: it’s not just the interpreter, an article by Kevin Modzelewski.

What is Nix

The most basic, fundamental idea behind Nix is this:

Everything on your computer implicitly depends on a whole bunch of other things on your computer.

  • All software exists in a graph of dependencies.
  • Most of the time, this graph is implicit.
  • Nix makes this graph explicit.

Source: What Is Nix, an article by Burke Libbey.

Defensive BASH Programming

Here is my Katas for creating BASH programs that work. Nothing is new here, but from my experience pepole like to abuse BASH, forget computer science and create a Big ball of mud from their programs. Here I provide methods to defend your programs from braking, and keep the code tidy and clean.

Source: Defensive BASH programming, an article by Kfir Lavi.