Unix Pipes

A pipe bridges two processes together. The pipe, written with the | character, connects the output of one command to the input of the second. It’s one of the fundamental building blocks of a Unix shell.

A very basic example is using cat to feed the contents of a file to the word count tool wc:

cat <filename> | wcCode language: plaintext (plaintext)

Just in case you didn’t know these commands:

  • cat prints the contents of a file.
  • wc counts all the lines, words, and bytes it is fed.

Another often-used version is wc -l, which only counts the number of lines in a file.

Let’s try this. First, create a file called words.txt with this content:

hello
world
python
ninja
are
you
?
3
2
1Code language: plaintext (plaintext)

Then run this command:

cat words.txt | wc -lCode language: plaintext (plaintext)

The output should be 10 since there are 10 lines in the file. Another example is using sort to sort the words in your file:

$ cat words.txt | LC_ALL=C sort
1
2
3
?
are
hello
ninja
python
world
youCode language: plaintext (plaintext)

As we can see, all the words from words.txt are fed to sort, which sorts them. With LC_ALL=C, sort simply orders the characters by their number in the ASCII table: digits first, then the question mark, then the letters.

Why the LC_ALL=C? Without it, sort uses the language settings (the locale) of your system, and the order can differ per system. On my Mac with an English locale, for example, the question mark comes first, before the digits. If you need the same result everywhere, for example in a script, set LC_ALL=C.

Alright. Here’s one last questionable example of using pipes. There’s this awesome tool called cowsay. You probably need to install it since it’s likely not installed by default. It generates a text-based representation (also called ASCII art) of a cow saying anything that you feed it:

$ echo "Are we going to learn anything useful?" | cowsay
 ________________________________________
< Are we going to learn anything useful? >
 ----------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||Code language: plaintext (plaintext)

This is the stuff that will make a lasting impression on your colleagues.

Using Python in a pipe

Your own Python scripts can be part of a pipeline too. Whatever comes in through the pipe is available as sys.stdin, which you can loop over line by line, just like a file. Everything you print() goes out on the other side. Save this as shout.py:

import sys

for line in sys.stdin:
    print(line.strip().upper())Code language: Python (python)

Now you can put it between other commands. Here, head -3 only shows the first three lines of the result:

$ cat words.txt | python3 shout.py | head -3
HELLO
WORLD
PYTHONCode language: plaintext (plaintext)

The strip() removes the newline at the end of each line; otherwise print() would add a second one. This is the Unix philosophy at work: small programs that each do one thing, combined with pipes. Your Python script doesn’t need to know where its input comes from or where its output goes.

Related articles

Leave a Comment