Subprocess In Python (Run Command Line)

Good evening, Roberts Greibers here.

We will get into my Python subprocess story and how I even got the point of writing blog posts about Python a bit later. 

But at this moment, I want you to pay attention and read very carefully!

What I’m about share with you is a real subprocess Python example! 🚨

How to use subprocess in Python to run command line commands!

The difference between running a shell command in Python with:

• Python subprocess run command
• Python subprocess popen command
• Python subprocess call command
• And why you should use one or the other to execute shell commands with Python (e.g. run Java applications from Python code)

Also, a real example of me actually executing a shell command in Python.

Most “Python subprocess example” tutorials you’ll find online won’t give you a REAL project example..

And they won’t tell you what steps you’ll need to take in your specific situation. 

In this post, I’ll share how you should THINK in order to be able to come up with a solution for your own “how to run shell command in Python?” problem.. 🚀

If this sounds interesting to you, keep on reading, I’ll also give you an opportunity to become a Python developer at the end of the post 👇🏻

Why You Must Learn To Use Subprocess In Python?

Learning to execute shell command in Python by using one of the Python subprocess methods, like subprocess run or subprocess popen is the SECRET knowledge most beginner Python developers don’t know about!

Understanding shell command execution from Python gives you the chance to run not only Python code, but also any external application outside of Python, for example, any Java or GoLang applications. 

🚨 BIG tech companies are willing to pay more than $100,000/year to Python developers who understand such programming concepts..

Not only that, Python subprocess knowledge is one of those tricky topics that come up in Python developer interviews pretty often!

How Yuliia Got Hired For $70,000/Year

In Python Mentoring, one of my recent graduates – Yuliia actually got hired for a $70,000 / year salary 💰

This is a really big achievement for someone with NO PRIOR EXPERIENCE in Python software development! 🐍

You can read the whole story of how Yuliia got hired in about 6 months by clicking on the image below! 👇🏻

In short: Yuliia went through a bunch of Python interviews (In some of them I was a bit of help) and there were questions regarding running a subprocess in Python. 

Just makes you wonder, if Yuliia would not know how to run shell command in Python, she might not have gotten the job. 

Of course, I’m just speculating here, but that’s one of the main reasons you should know how to execute shell command in Python.  

Using Python Subprocess In A Real Project For A Payment Integration

In my day job, I work as a backend Python/Django developer for a local company here in Riga, Latvia. We’re building a Payment Gateway platform as a B2B service. (find more details on my Linkedin)

On a daily basis, I’m working with dozens of payment providers, banks, and banking APIs.

As you may know, with legacy banking systems there are ups and downs, nothing ever goes smoothly..

So, In this case, I had to use a Python subprocess package to run a shell command which would then actually send a real SOAP request to an external banking system. 

Also, after a successful Python subprocess call, I had to figure out how to capture output from it. 

Let’s dive into the code and I’ll explain how I did it.. 👇🏻

Python Subprocess Run (Code Example)

Before we get into any Python subprocess run examples, I have to explain why there’s an example of Go code listed down below.

In my case, the shell command I had to execute from Python was actually a command to run a Java executable.

But since I’m not a fan of Java and it’s a headache to even create a simple executable I used Go to simulate the same situation.

For the sake of the example, we’re just going to use a simple Go executable that prints out arguments passed to it. (you can download it here or create your own with the Go code below)

This will let us simulate the same exact situation I had when I was working on a payment integration. 

This webservice can be executed by running the following shell command:

./webservice -Dservice.url=https://astra.citadele.lv/gate/B4BGateway -Dservice.sign=true -Dservice.subst=false -Dservice.businessdate=19-10-2022 -Dservice.ks=/projects/pythonic.me/keystore.ks -Dservice.sys.ks=/projects/pythonic.me/sys.keystore.ks -Dservice.sys.alias=SampleAlias -Dservice.sys.password=SamplePassword123 -Dservice.dump=true -Dfile.encoding=UTF-8

Webservice For Python Subprocess Run Example

And here’s the code for the webservice executable available in the webserivce.zip above.

To follow the next steps, you need to download webserivce.zip and extract the webservice executable in a folder where you plan to develop your Python subprocess script.

package main

import (
    "fmt"
    "os"
)

func main() {

    // Accessing args
    dServiceUrl := os.Args[1]
    dServiceSign := os.Args[2]
    dServiceSubst := os.Args[3]
    dServiceBusinessDate := os.Args[4]
    dServiceKs := os.Args[5]
    dServiceSysKs := os.Args[6]
    dServiceSysAlias := os.Args[7]
    dServiceSysPassword := os.Args[8]
    dServiceDump := os.Args[9]
    dServiceEncoding := os.Args[10]

    // Parsing args
    dServiceUrl = dServiceUrl[14:len(dServiceUrl)]
    dServiceSign = dServiceSign[15:len(dServiceSign)]
    dServiceSubst = dServiceSubst[16:len(dServiceSubst)]
    dServiceBusinessDate = dServiceBusinessDate[23:len(dServiceBusinessDate)]
    dServiceKs = dServiceKs[13:len(dServiceKs)]
    dServiceSysKs = dServiceSysKs[17:len(dServiceSysKs)]
    dServiceSysAlias = dServiceSysAlias[20:len(dServiceSysAlias)]
    dServiceSysPassword = dServiceSysPassword[23:len(dServiceSysPassword)]
    dServiceDump = dServiceDump[15:len(dServiceDump)]
    dServiceEncoding = dServiceEncoding[16:len(dServiceEncoding)]

    fmt.Printf("Sending an encoded SOAP request to: %s \n", dServiceUrl)
    fmt.Printf("Service request signing: %s \n", dServiceSign)
    fmt.Printf("Service request substitution: %s \n", dServiceSubst)
    fmt.Printf("Service request business date: %s \n", dServiceBusinessDate)
    fmt.Printf("Service keystore path: %s \n", dServiceKs)
    fmt.Printf("Service system keystore path: %s \n", dServiceSysKs)
    fmt.Printf("Service system keystore alias: %s \n", dServiceSysAlias)
    fmt.Printf("Service system keystore password: %s \n", dServiceSysPassword)
    fmt.Printf("Service logs dump enabled: %s \n", dServiceDump)
    fmt.Printf("Service encoding: %s \n", dServiceEncoding)
}

Python Subprocess Run Explained

Before I share my own subprocess Python code, you have to understand what Python subprocess run command is all about. 

I couldn’t explain it better than it’s already explained in subprocess run documentation (see screenshot below).

But, if I had to explain it in my own words, I’d say you’re passing args to the .run() command which is supposed to be a Python List object. 

args would be the list of strings (shell command parts separated), so for example, let’s say you’d want to execute ls -l inside one of your directories to see the list of available files and folders..

An equivalent Python subprocess run command would be:

import subprocess
import typing

cmd = ['ls', '-l']
EXECUTE_IN_PATH = '/Users/robertsgreibers/projects/pythonic.me'

p: typing.Union[subprocess.CompletedProcess] = (
    subprocess.run(
        cmd,
        cwd=EXECUTE_IN_PATH,
        shell=False,
        capture_output=True
    )
)

shell_command_result = p.stdout.decode()

print(
    f'shell_command_result: \n'
    f'{shell_command_result}',
)

And then the Python run command output could look similar to mine:

shell_command_result: 
total 64
drwxr-xr-x  8 robertsgreibers  staff    256 Oct 23 17:13 7_python_subprocess
-rw-r--r--  1 robertsgreibers  staff    138 Apr  8  2022 Pipfile
-rw-r--r--  1 robertsgreibers  staff    453 Apr  8  2022 Pipfile.lock

Popen Python Subprocess Command (Don’t Use It – Here’s Why)

You might have seen Popen Python subprocess code examples online with similar results.

You might even ask what’s the difference between Python subprocess run command and Popen Python command.

Answer: Just take another look at the documentation screenshot I shared above, it’s right there – Python subprocess run command is basically just calling the same Python subprocess popen command. 

Link to Popen Python documentation (if you’re curious to explore more details)

I personally prefer Python subprocess run approach, the one I already explained above. 

Keep things simple and just stick with high-level functions (already designed to save time for devs)

Python Subprocess Call (Why You Should Use .run() Instead)

Subprocess run Python example is not the only one you’ll find online when Googling for “execute shell command Python”.

From my own experience, I’ve seen dozens of old “Python Subprocess Call Example” code snippets.

The thing is, to execute shell command Python documentation suggests you use .run() method – the one I’ve already explained above.

subprocess.call() is considered to be an older high-level API method. 

Back in the day, when Python 2.7 used to be the latest Python version, I remember using subprocess.call() method all the time.. but that time has passed, just use .run() instead.

Python Subprocess Run (The Final Example)

Alright, let’s put it all together!

In summary, you want to stick with using .run() method to run subprocess in Python.

Any other methods you’ve seen online are either older code examples or used to handle a specific case.

In general, .run() should be good enough for you to use in any case when you have the urge to Google for “Python run command line”.

This is HOW I THINK 💡, you don’t need to know every little detail to become a successful Python developer, you just need one solution that works in most cases. 

import datetime
import subprocess
import typing

# Settings to configure environment (dev or production)
DEBUG = True  # Dev if equals to True


class Gateway:
    # define path to where your executable is held
    # for the sake of a simple example, keep "webservice" executable
    # in the same folder as this Python file
    WEBSERVICE_PATH: str = './'

    def __init__(self):
        self.sandbox_url = 'https://dev.banking.com/gate/B4BGateway'
        self.production_url = 'https://prod.banking.com/gate/B4BGateway'

    def url(self) -> str:
        return (
            self.sandbox_url
            if DEBUG else
            self.production_url
        )

    def get_current_time(self) -> datetime.datetime:
        return datetime.datetime.now()

    def get_execution_date(self) -> datetime.date:
        return self.get_current_time().date()

    def get_keystore_path(self) -> str:
        # returning an abstract path, in a real project this would be
        # a real path to a real keystore file
        return '/projects/pythonic.me/keystore.ks'

    def get_sys_keystore_path(self) -> str:
        # returning an abstract path, in a real project this would be
        # a real path to a real keystore file
        return '/projects/pythonic.me/sys.keystore.ks'

    def get_sys_alias(self) -> str:
        return 'SampleAlias'

    def get_sys_password(self) -> str:
        return 'SamplePassword123'

    def request(self):
        GO_OPTS: typing.List[str] = [
            f'-Dservice.url={self.url()}',
            f'-Dservice.sign=true',
            f'-Dservice.subst=false',
            f'-Dservice.businessdate={self.get_execution_date()}',
            f'-Dservice.ks={self.get_keystore_path()}',
            f'-Dservice.sys.ks={self.get_sys_keystore_path()}',
            f'-Dservice.sys.alias={self.get_sys_alias()}',
            f'-Dservice.sys.password={self.get_sys_password()}',
            f'-Dservice.dump=true',
            f'-Dfile.encoding=UTF-8',
        ]

        cmd: typing.List[str] = ['./webservice'] + GO_OPTS

        p: typing.Union[subprocess.CompletedProcess] = (
            subprocess.run(
                cmd,
                cwd=self.WEBSERVICE_PATH,
                shell=False,
                capture_output=True
            )
        )

        print(
            f'Go process (p.returncode): '
            f'{p.returncode}',
        )
        print(
            f'Go process (p.args): '
            f'{p.args}',
        )
        print(
            f'Go process (p.stderr): '
            f'{p.stderr} \n\n',
        )

        return p.stdout.decode()


gateway = Gateway()

res = gateway.request()

print(res)

# In a real project, you'd parse the response
# and finalize payment initiation

So, this is my approach to using subprocess Python package. 

You can bookmark this blog post page and use it as a template to start your work any time you need to run shell command in Python.

In fact, if you’re a part of any Python coding forums or groups on Facebook, Reddit, etc.

I’d appriciate it if you’d share this post! 📝

Here’s the expected output with webservice executable (shared at the beginning of the post)

Go process (p.returncode): 0
Go process (p.args): ['./webservice', '-Dservice.url=https://dev.banking.com/gate/B4BGateway', '-Dservice.sign=true', '-Dservice.subst=false', '-Dservice.businessdate=2022-10-19', '-Dservice.ks=/projects/pythonic.me/keystore.ks', '-Dservice.sys.ks=/projects/pythonic.me/sys.keystore.ks', '-Dservice.sys.alias=SampleAlias', '-Dservice.sys.password=SamplePassword123', '-Dservice.dump=true', '-Dfile.encoding=UTF-8']
Go process (p.stderr): b'' 


Sending an encoded SOAP request to: https://dev.banking.com/gate/B4BGateway 
Service request signing: true 
Service request substitution: false 
Service request business date: 2022-10-19 
Service keystore path: /projects/pythonic.me/keystore.ks 
Service system keystore path: /projects/pythonic.me/sys.keystore.ks 
Service system keystore alias: SampleAlias 
Service system keystore password: SamplePassword123 
Service logs dump enabled: true 
Service encoding: UTF-8 

Why Aren’t You A Python Developer Already?

Well, to be honest, I might even know why… ⚠️

It’s really NOT that easy to figure out the STEPS to take in order to become a Python developer WITHOUT HELP so don’t blame yourself just yet!

It’s actually really really HARD to become a Python developer without GUIDANCE! ❌

Believe me, I’ve tried..

Why “I’ll Do It Myself” Doesn’t Work!

I used to THINK: “It can’t be that hard, I’ll do it myself.. everything is online anyway..”

Oh boy, the years I wasted with such an ego-driven mindset! 😩

The thing is, of course, there’s information you can find online and you can figure out how to do things on your own..

But that’s just the beginning.. the REAL problems arise when you’re thrown into real project situations. 

You literally have NO IDEA what to do and how to solve the CASCADE of problems you create as a beginner developer..

You might say “Yeah, but I work harder than anyone else, I’ll figure it out..”

🚨 Listen, NO YOU WON’T! 🚨

I’m telling you, I was in the same exact position as you, working harder than anyone else around me, THERE ARE PROGRAMMING CONCEPTS you just can’t “figure out”.

Those programming concepts are the ones you learn from your mentors..

I was lucky enough to be surrounded by mentors (more experience Python developers) earlier in my career – mentors who pointed out to me the significant mistakes I kept making and how to avoid them..

Keep this in your mind – I’m not an exception, I used to be YOU..

I also struggled quite a bit before my mentors guided me in the RIGHT DIRECTION..

What Should You Do As A Beginner Python Developer

It doesn’t matter how smart you are..

If you’re NOT a Python developer yet and you want to become one ASAP..

You have to humble yourself.. You have to find a way to ask for help! 💡

You have to find YOUR MENTORS, people you can learn from.

When I say “humble yourself”, I mean IT’S NOT EASY to ask for help..

You might even feel like admitting “well, I’m dumb.. I need help..”, etc.

I know for a fact, that it’s one of my insecurities, why I don’t want to ask for help when I’m stuck.. But it’s just a trick your mind plays on you! (It’s bullshit) 🧠

You have to find ways to go around your insecurities, accept you’re not there yet, and just ask for HELP!

No one is ever going to smash your head if you ask intelligent, well-structured questions.

In fact, experienced developers love to help out, you just have to be smart enough to ask proper, detailed questions. 

Your Opportunity To Become A Python Developer

I used to be a quality assurance engineer (more on my Linkedin profile)

It took me years to get out of it and become a Python developer (now already coding in Python for around 7+ years)

… just because I made so many mistakes along the way. 

There are literally SECRET programming concepts Python developers use every day that I had to learn from my mentors to become a professional Python developer. 

The funny thing is – coding bootcamps, online resources, none of them even talk about these programming concepts and that you should know them by heart as a developer which is frustrating when you’re trying to LEARN. 

I’m open to having a call with you to see if there’s a chance I can help you become a Python developer.. 

⚠️ See if there are still details on how you can contact me below the post..

My time is quite valuable at this point and I might not be coaching by the time you read this post..

I won’t tolerate laziness, I’d rather spend time with my girlfriend, traveling or enjoying life than try to force someone to learn Python..

That’s why I want to have a call with you first, just to see if we’re a good fit.

How To Find A Mentor To Learn Python From

My Python Mentoring consists of three major learning strategies.

First, we have a bunch of videos and documents explaining how you can fully develop real project features (for example, sign-up, sign-in, sign-out, survey, etc.)

Secondly, students are able to have a weekly LIVE Zoom call with me personally and share their problems through screen sharing. 

You can find video cuts from LIVE Zoom calls here on this blog somewhere..

Thirdly, we have a private Discord server where students can ask questions and I share additional coding videos, explaining programming concepts, etc. (I’ve shared a couple of screenshots in this post already – scroll up to see them)

🚨 If you take your career seriously, you need to find someone who can provide a minimum of the strategies I’ve mentioned above! 🚨

I just can’t see anyone succeeding in this industry without major help, it’s literally that difficult..

That’s why developers are so well-paid.. it’s really HARD to learn and correctly apply all the necessary programming concepts!

As I said, see if there are my contact details below the post if you’re interested.. tell me your story and I’ll see if there’s anything I can do to help you! 👇🏻

But if not, that’s fine.. I’m already in living the way I always wanted to live, I’m free, well-paid, and have the opportunity to work from anywhere in the world. 

This is about your life, your career – just make sure you don’t waste time “figuring out things on your own”, essentially just wasting your life away..

Talk soon!