Plurrrr

In my last post, I talked about how Haskell newtypes are great tools for modeling JSON data when writing API clients in Haskell, and I included some examples on how to write custom ToJSON and FromJSON methods that incorporated these newtypes. That post generated some discussion on Lobsters, from which I learned about this interesting library called autodocodec. Given the advantages laid out in that discussion, I decided to give that library a try on my project’s codebase, and it worked so well that, so I ended up refactoring basically all of my types to use autodocodec to generate JSON parsers for my types. In fact, I enjoyed the experience of using autodocodec so much that I thought it was worth blogging about.

I want to create a personal web crawler. As I download each page, I collect a new list of URLs. To increase the likelihood that I am downloading the most important pages first, I want to keep the list of URLs to visit sorted by length (shorter URLs are more likely to be closer to the front page). That presents a problem though, because I'm also adding URLs.

A naïve list (Vec<T> in Rust), won't be very efficient. After every batch of inserts, I would need to re-sort the list. Wouldn't it be good if there were a data structure that could maintain the sort order itself? Yes! It's called a binary heap.

I’m an idiot. And git is hard. A lot of places use a rebase-based Git workflow, and I’ve made git less hard with a set of handy aliases. Put these in your ~/.gitconfig and turn git into an actually less painful command line tool to use.