Sun 13 Dec 2020

Bulk loading into PostgreSQL: Options and comparison

You have a file, possibly a huge CSV, and you want to import its content into your database. There are lots of options to do this but how would you decide which one to use. More often than not the question is how much time would the bulk load would take. I found my self doing the same few days back when I wanted to design a data ingestion process for PostgreSQL where we needed to bulk load around 250GB of data from CSV files every 24 hours.

Source: Bulk loading into PostgreSQL: Options and comparison, an article by Muhammad Usama.

Use polling for resiliency

How should two microservices communicate?

Let’s analyze two different communication patterns:

  • Polling: Service B periodically query Service A for the current state of the users and updates its local storage.
  • Event driven: Service A publishes in a queue every time user information is updated. Service B consumes the updates to stay up to date.

Using TLA+ we are going to model the two different patterns to see how well they fit our requirements. In our specification, we are also going to take into account unexpected failures that can affect the system and show how to model them in TLA+.

Source: Use polling for resiliency, an article by Georgios Chinis.