Process Substitution in Bash

Process substitution is a handy feature in Bash that lets you redirect the input and output of a process as if it were a file. This opens up some neat possibilities for chaining programs together in creative ways.

This article aims to provide a comprehensive understanding of process substitution in Bash, along with some practical examples.

Process Substitution Syntax

There are two forms of process substitution in Bash:

• <(command) feeds the output of the process into another command.
• >(command) takes the output of a command and feeds it as input to a process.

So, <() treats a process as file input, while >() treats it as file output. The operator >(...) is not as commonly used as <(...).

Using Process Substitution in Bash

Let's take a look at a few examples to illustrate how process substitution can be used.

Comparing Files

Process substitution can simplify complex pipelines by eliminating the need for unnecessary temporary files. Instead of creating intermediate temporary files, process substitution manages redirection via automatic temporary files, which are cleaned automatically after the command finishes.

For instance, if you want to compare two files sorted line by line, you would typically do this without process substitution:

sort data.txt > sorted-data.txt
sort backup-data.txt > sorted-backup-data.txt
diff sorted-data.txt sorted-backup-data.txt

With process substitution, this can be done in one line:

diff <(sort data.txt) <(sort backup-data.txt)

Comparing Command Outputs

Suppose you've changed a directory structure and want to compare the file listings before and after the changes to ensure accuracy:

diff <(ls /data) <(ls /data-backup)

This command compares the output of the ls command for two different directories (/data and /data-backup). The process substitution syntax <(command) treats the ls output as if it were a file, allowing diff to compare the "files" directly.

Searching through Command Output

Suppose you want to search for a specific error message or pattern across multiple log files' output. You would run:

grep 'error' <(cat /var/log/syslog /var/log/apache2/error.log)

In this example, process substitution treats the combined output of cat /var/log/syslog and cat /var/log/apache2/error.log as a single file. The grep command then searches for the pattern 'error' across this combined output stream.

This is equivalent to:

cat /var/log/syslog /var/log/apache2/error.log | grep 'error'

Sorting and Saving Command Output

The following example sorts the output of the ps aux command and saves the sorted output to a file for further analysis or processing:

sort <(ps aux) > processes.sorted

Here, the ps aux command's output (which lists running processes) is treated as a file and piped into the sort command. The sorted output is then redirected to a file named processes.sorted.

Opening Multiple Files

Process substitution can be handy if you want to open multiple text files in a text editor like vim, without having to manually concatenate them first.

vim <(cat data.txt new.txt)

This command combines the contents of data.txt and new.txt into a single stream, which is then treated as a file and opened in the vim text editor.

Transferring Compressed Files

Often, you need to transfer and extract files or directories between different servers for backup, deployment, or maintenance purposes. Instead of creating a local archive, transferring it over SSH, and then extracting it on the remote server (which involves multiple steps and temporary files), you can streamline the process using process substitution.

tar -cf >(ssh remote_server tar xf -)

The above command allows you to create a tar archive of the current directory and directly pipe it over an SSH connection to the remote server, where it is extracted in a single step.

Advantages of Process Substitution

Some key advantages of process substitution include:

• Avoiding Temporary Files: Process substitution eliminates the need to create temporary files on disk, which can save disk space and improve efficiency, especially when dealing with large amounts of data
• Streamlined Pipelines: By treating the output of one command as the input file for another command, process substitution allows you to create more streamlined and concise command pipelines, making your scripts and workflows more readable and maintainable.

Conclusion

This article provided a concise overview of process substitution in Bash, a handy feature that allows you to treat the input and output of processes as if they were files. Through several practical examples, you learned how process substitution can simplify complex pipelines, enable efficient file comparisons, filter command output, transfer compressed data, and open multiple files simultaneously. Though the article was short, it showed many useful process substitution methods. This technique can help make command-line tasks easier and faster.