Plurrrr

A bloom filter is one of those data structures you are probably already aware of, or have at least heard about. For those looking for a simple recap, they are a probabilistic data structure which can be used to determine if something is in a set or not, giving a slight chance of returning a false positive result for some checks but while using less space than a full hashmap.

What you may not know is that while you can use them as a space efficient hash/dictionary there are other use cases you might not be aware of.

After seeing how to extract content from a website and how to avoid being blocked, we'll take a look at the crawling process. To get data at scale, getting a few URLs by hand is not an option. We need to use an automated system that will discover new pages and visit them.

Recently a blog post came out which I quite like, it describes how to use the concrete base transformers. It’s very thorough and gives a concrete example for using transformers. Although it looks quite low level and I think you’ll get more out of transformers by using full MTL.