Tue 15 Mar 2022

Processing large JSON files in Python without running out of memory

If you need to process a large JSON file in Python, it’s very easy to run out of memory. Even if the raw data fits in memory, the Python representation can increase memory usage even more.

And that means either slow processing, as your program swaps to disk, or crashing when you run out of memory.

One common solution is streaming parsing, aka lazy parsing, iterative parsing, or chunked processing. Let’s see how you can apply this technique to JSON processing.

Source: Processing large JSON files in Python without running out of memory, an article by Itamar Turner-Trauring.

Swift 5.6 Released!

Swift 5.6 includes a number of enhancements to the type system, improved interaction with pointers, and adds the ability to run new plugin commands using the package manager.

Source: Swift 5.6 Released!, an article by Ted Kremenek.

Monorepos done right

There’s a lot of arguments for and against monorepos. Done right, I believe monorepos will help you move with urgency and focus as your engineering organization grows. That doesn’t mean it’s easy! I’ve also seen monorepos turn into monoliths that block teams from getting anything out in a timely manner.

Source: Monorepos done right, an article by Felix Mulder.