Removing duplicate lines with awk
A one liner to remove duplicate lines from a text file while
preserving the original order using awk:
awk '!seen[$0]++' in.txt > out.txt
Source: How to remove duplicate lines from files keeping the original order, which explains how this works and discusses alternatives.