Why Python Is Great: Subprocess — Running Linux Shell From Python Code.

Python is a brilliant object oriented programming language. In artificial intelligence/deep-learning circles, Python is often referred to as the default language (lingua franca) of artificial intelligence. But, the charm of Python extends way beyond running highly complicated deep-learning code. Python is first and foremost a general purpose programming language.

One of the key features of Python is its close integration with Linux. In this post I am going to explore one particular feature inside Python called ‘subprocess’. Subprocess, just like the name suggests initiates a linux shell command in python. This makes a lot of things easier inside Python, including, let us say, creating and manipulating file-system entries.

Here is a Python 3 function that utilizes subprocess to run a list of commands in linux terminal.

import subprocess

def execute_in_shell(command=None, 
                     verbose = False):
    """ 
        command -- keyword argument, takes a list as input
        verbsoe -- keyword argument, takes a boolean value as input
    
        This is a function that executes shell scripts from within python.
        
        Keyword argument 'command', should be a list of shell commands.
        Keyword argument 'versboe', should be a boolean value to set verbose level.
        
        Example usage: execute_in_shell(command = ['ls ./some/folder/',
                                                    ls ./some/folder/  -1 | wc -l'],
                                        verbose = True ) 
                                        
        This command returns dictionary with elements: Output and Error.
        
        Output records the console output,
        Error records the console error messages.
                                        
    """
    error = []
    output = []
    
    if isinstance(command, list):
        for i in range(len(command)):
            try:
                process = subprocess.Popen(command[i], shell=True, stdout=subprocess.PIPE)
                process.wait()
                out, err = process.communicate()
                error.append(err)
                output.append(out)
                if verbose:
                    print ('Success running shell command: {}'.format(command[i]))
            except Exception as e:
                print ('Failed running shell command: {}'.format(command[i]))
                if verbose:
                    print(type(e))
                    print(e.args)
                    print(e)
                
    else:
        print ('The argument command takes a list input ...')
    return {'Output': output, 'Error': error }

This function returns dictionary with two key values: Output and Error. Output records the console output, Error records the console error messages.

To run the function above, next, I will create a Python command:

This command will:

1. Create a folder in Desktop called ‘New_photos’.
2.  Randomly copy 10 .jpg files from  ~/Desktop/My_folder/Photos/ to ~/Desktop/New_photos/
3.  Count the total number of files in the folder: ~/Desktop/New_photos/

command = ['mkdir ~/Desktop/New_photos/'
           'cd ~/Desktop/My_folder/Photos/ ; shuf -n 10 -e *.jpg | xargs -i cp {} ../../New_photos/',
           'ls ~/Desktop/New_photos/  -1 | wc -l']

print ((execute_in_shell(command = command)).get('Output'))

 

That’s it. Running Linux commands inside python is as straightforward as passing a list of commands to a Python function.