Python Requests POST GET (HTTP Request Explained)

Welcome back or nice to see you for the first time!

Roberts Greibers here with another experience-based post about Python requests.

I’ve been working professionally as a Python developer for well over 7 years, I’ll share more details about my story later in the post.. (just scroll down the page to find out more about me). 

But before we talk about how qualified I am to write posts about Python requests, I’ll just give you all the answers you’ve been Googling for. 🚀

WARNING: You’re about to see the most important Python requests library use cases with clearly defined examples. 

This is the BIGGEST breakthrough you’ll ever achieve as a Python developer.

Learning and implementing this way of calling API with a Python request is a major achievement in your career that you’ll use on a daily basis for years to come. 

In this post, you’re going to explore:

• How to install and use the most popular Python HTTP library (Python requests library)
• Why requests Python module is the recommended way for HTTPRequest in Python?
• How to use requests Python library in your project (LIVE Zoom call with a student) 
• How I’ve used Python requests library for the past 3 years at my dev job.

🚨 I’ll explain in detail everything you should know about Python3 requests (why Python2 requests are NOT used anymore)

And I’ll show you how API calls in Python are done in a real project (from Django projects I’ve worked on)

The intention here is to give you a real Python HTTP request example you could use in your own projects. 

Just a quick note before we start: At the end of the post I’ll share details about what helped my students to get hired as Python developers, read till the end and see if that helps you too!👇🏻

Why Python Requests Are Crucial For Your Career?

Having mastered Python requests library and having your own framework (system to follow) for making HTTPRequest in Python makes you excel in your Python abilities…

… and let me explain WHY 👇🏻

This might differ from field to field, but in general, most integrations between systems are handled through REST APIs. 

In most projects where you’ll work as a backend Python developer, you’ll be involved in the process of creating a Python HTTP client and actually making API calls in Python. 

Let me give you a quick list of examples:

• Want to handle payments through Stripe in your project? That’s a bunch of Python HTTPS client calls.
• Want to implement a Zoom call scheduling system? – again, API Requests Python
• Want HTML-templated and beautifully styled emails through SendGrid? once again, HTTPS requests Python..

You get the idea..

As a Python developer you’re NOT going to re-invent the wheel, you’ll use a bunch of already-built solutions and you’ll integrate with them to build your own project! 🔥

Requests In Python Django Project

Just to give you a bit more context, take for example the company I work for and the product we’re building. (More on my Linkedin)

We’re building a payment gateway platform. (trying to put a bunch of payment integrations into one organized system and sell it to businesses in need of this service)

For the past 3 years, I’ve developed a large portion of our “Payment Initiation Service (PIS)” integrations.

All of them involve some sort of Python requests execution. 

Of course, the core system is built with Django and we have our own innovations too, but for payments, we’re literally a gateway.

We’re handling all payment API calls in Python requests. 👌🏻

Python Requests Library In Your Project

Requests Python module is the recommended way to create an HTTPRequest in Python just because it’s the most straightforward and easiest way to make a Python HTTP request. 

Python requests library literally has a headline Requests: HTTP for Humans on their website.

And when it comes to coding there’s a misconception “code should be complex” …

Which is incorrect!

In reality, you’re always writing code for humans, everyone should be able to read and understand your code..

Click on the video ( play button ▶️ ) to watch a video cut from a LIVE Zoom call recorded a couple of months ago when one of my students Yuliia was discovering requests in Python for the first time. 

In general, for calling API Python requests you don’t need any other libraries. (unless there’s a specific requirement)

Python3 requests is the one and only library used in most Python projects, Python examples and that’s why I suggest my students learn only requests, disregarding any other libraries. 

If you believe you’d be willing to work hard and achieve similar results to Yuliia’s…

… see if there’s any way you can contact me at the end of this post!

🚨 In Python Mentoring we tend to focus on preparing you for a real Python developer career, teaching you only the stuff that’s actually used in the industry by large tech companies!

Building An API Wrapper Class With Python Requests

The best way of explaining my workflow is to just take you through an example of a situation that’s as close as possible to the real one.

But, it would be difficult to set up a real API for the sake of an example or make you sign up for one of the previously mentioned services only to follow my workflow..

Instead, what we will use is this simple service called httpbin.org

Install Python Requests (pip install requests)

I won’t go into details of setting up the whole Python environment in this post (there are plenty of articles online for that)

And there are multiple ways of using Python on your local machine – global Python, virtual environment, even virtual machine, etc. 

In this example, I’m going to use my favorite and the one I’ve seen used the most – pipenv. 

I already have a pipenv environment set up in my pythonic.me project folder 

If you’re in a similar situation, then to start you just need to execute the following command to install requests:

pipenv install requests

It should give you a very similar output to the one I’ve shared below..

✔ Installation Succeeded – is what you’re looking for!

(pythonic.me) robertsgreibers@MacBook-Pro-2 pythonic.me % pipenv install requests
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
Installing requests…
✔ Installation Succeeded 
Pipfile.lock (16c839) out of date, updating to (fe5a22)…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Building requirements...
✔ Success! 
Updated Pipfile.lock (fe5a22)!
Installing dependencies from Pipfile.lock (fe5a22)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00
(pythonic.me) robertsgreibers@MacBook-Pro-2 pythonic.me % 

Notice, I’m not using pip install requests or pip3 install requests

pip is a part of pipenv, study pipenv from this link to understand the difference and why you might be interested to use pipenv too. 

Python Import Requests

Right after installing requests, Python will let you import the library and use it.

To import requests in Python, simply add the following line at the top of your Python script:

import requests

Python Requests GET Method Explained

One of the simplest HTTP requests you’ll ever make is Python HTTP GET request.

Once you have the requests module installed, you can start by typing out requests.get to see if you can get any suggestions from your IDE.

This is on PyCharm (Great support for Python, totally recommended for Python beginners)

PyCharm suggests all the possible Python request GET params, but to make a very simple HTTP GET request Python only requires only one param which is url.

GET request usually means you’re asking to receive data from a server somewhere on the web. 

Some of the simple, very common examples would be:

• GET /users – Retrieve data of available users (a list of users)
• GET /countries – Retrieve data of available countries (a list of countries)
• GET /options – Retrieve data of available options (a list of options)
• and more, etc.. 

import json

import requests


res: requests.Response = requests.get(
    url='https://httpbin.org/get'
)

print('res.status_code: ', res.status_code)
print('res.json(): ')
print(json.dumps(res.json(), indent=4))

For testing purposes, here’s an example of a simple GET request using httpbin.org

You won’t receive any significant results here, what httpbin.org does is simply returns information about the details of the request you made. 

Which can be confusing, keep that in mind.

In a real API, you’d get new information – info, that you requested. 

Here’s the response:

res.status_code:  200
res.json(): 
{
    "args": {},
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip, deflate",
        "Host": "httpbin.org",
        "User-Agent": "python-requests/2.28.1",
        "X-Amzn-Trace-Id": "Root=1-63820e7f-281682d63b9b56016cd8fbfc"
    },
    "origin": "91.188.39.199",
    "url": "https://httpbin.org/get"
}

The above example is fine in the beginning, but usually when it comes to building a development feature (e.g. payment integration)

Eventually, you’ll be required to wrap your requests in a Python class (that’s what I found from my experience)

If you’ll leave naked requests in your project, it will be harder to do refactoring, logging, or any other handling on a larger scale. 

Imagine having 50+ Python requests GET calls spread throughout the whole project and you need to make a small change in all of them. 

That’s 1h+ of manual work, you don’t want to be in this situation ever.. 

Here’s an example of starting to build a wrapper class in Python:

import json

import requests


class HTTPBinAPI:

    def __init__(self):
        self.base_url = 'https://httpbin.org'

    def get(self):
        res: requests.Response = (
            requests.get(
                url=f'{self.base_url}/get'
            )
        )
        print('res.status_code: ', res.status_code)
        print('res.json(): ')
        print(json.dumps(res.json(), indent=4))
        breakpoint()


api = HTTPBinAPI()

api.get()

As I’d build the wrapper class, I’d also use json.dumps together with breakpoint() just the see a prettified response and analyze if it makes sense what I’m doing. 

When you see GET request makes sense from your get() method, just simply return res variable from it.

Output should be the same as in the first example:

res.status_code:  200
res.json(): 
{
    "args": {},
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip, deflate",
        "Host": "httpbin.org",
        "User-Agent": "python-requests/2.28.1",
        "X-Amzn-Trace-Id": "Root=1-63820923-0e4709d6410a6fdc477b2d9c"
    },
    "origin": "91.188.39.199",
    "url": "https://httpbin.org/get"
}
--Return--
> /Users/robertsgreibers/projects/pythonic.me/9_python_requests/1.py(20)get()->None
-> breakpoint()
(Pdb) 

Requests POST Python Example

Building on top of the GET example, here’s a requests POST Python example.

The only difference for this simple example is that requests POST is intended for object creation or some kind of submission to the API you use.

And when it comes to HTTP requests POST method, we usually use JSON format to submit data through API.

You can already see, there’s a built-in parameter in requests.post called json

You don’t have to do anything else besides submitting your JSON as a Dict through json param and you should be good. 

import json
import typing

import requests


class HTTPBinAPI:

    def __init__(self):
        self.base_url = 'https://httpbin.org'

    def get(self) -> requests.Response:
        res: requests.Response = (
            requests.get(
                url=f'{self.base_url}/get'
            )
        )
        return res

    def post(self, json_data: typing.Dict[str, str]) -> requests.Response:
        res: requests.Response = (
            requests.post(
                url=f'{self.base_url}/post',
                json=json_data
            )
        )
        return res


api = HTTPBinAPI()

# api.get()

res: requests.Response = api.post(
    json_data={
        'full_name': 'Roberts Greibers',
        'project': 'Pythonic.me'
    }
)

print('res.status_code: ', res.status_code)
print('res.json(): ')
print(json.dumps(res.json(), indent=4))

Notice, this time httpbin.org is returning a response that contains json key

This json key tells you what JSON data you would submit through your POST request to a real API if this would have been a request to a real API, not httpbin.org

httpbin.org can be used also as a testing tool to see if you’re sending data correctly to the API you’re working with. 

res.status_code:  200
res.json(): 
{
    "args": {},
    "data": "{\"full_name\": \"Roberts Greibers\", \"project\": \"Pythonic.me\"}",
    "files": {},
    "form": {},
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip, deflate",
        "Content-Length": "59",
        "Content-Type": "application/json",
        "Host": "httpbin.org",
        "User-Agent": "python-requests/2.28.1",
        "X-Amzn-Trace-Id": "Root=1-63822ece-4e3b408f314138513b311f78"
    },
    "json": {
        "full_name": "Roberts Greibers",
        "project": "Pythonic.me"
    },
    "origin": "62.85.95.199",
    "url": "https://httpbin.org/post"
}

httpbin.org seems to be returning Python HTTP status as 200, but for resource creation usually APIs are configured to return 201.

Here are docs on a proper status code for POST requests.

Python Header Requests Explained

When it comes to using headers in Python requests, one of the most common situations you’ll run into is some sort of authorization by providing a tokenized header alongside your request body. 

Bearer token authorization is quite common alongside APIs and you’ll run into it all the time. 

The idea is that each user can generate their own API token and hold it dearly (preferably never exposing it to the public)

User would use such a token to access restricted/protected resources (only the user with the token should be able to access the resource)

Let’s use /bearer API endpoint from httpbin.org to test this functionality through Python. 

Here’s what happens when you try to access the protected resources without the token:

import json
import typing

import requests


class HTTPBinAPI:

    def __init__(self):
        self.base_url = 'https://httpbin.org'

    def get(self) -> requests.Response:
        res: requests.Response = (
            requests.get(
                url=f'{self.base_url}/get'
            )
        )
        return res

    def post(self, json_data: typing.Dict[str, str]) -> requests.Response:
        res: requests.Response = (
            requests.post(
                url=f'{self.base_url}/post',
                json=json_data
            )
        )
        return res

    def get_bearer(self) -> requests.Response:
        res: requests.Response = (
            requests.get(
                url=f'{self.base_url}/bearer',
            )
        )
        return res


api = HTTPBinAPI()

# api.get()

# res: requests.Response = api.post(
#     json_data={
#         'full_name': 'Roberts Greibers',
#         'project': 'Pythonic.me'
#     }
# )

res: requests.Response = api.get_bearer()

print('res.status_code: ', res.status_code)
print('res.json(): ')
print(json.dumps(res.json(), indent=4))

You can see, we were able to retrieve status code 401.

401 means the following:

The HyperText Transfer Protocol (HTTP) 401 Unauthorized response status code indicates that the client request has not been completed because it lacks valid authentication credentials for the requested resource.

Here’s the explanation for it, 401 Unauthorized pretty much explains it all.

You need to pass authorization details to be able to access this resource.

Traceback (most recent call last):
  File "/Users/robertsgreibers/.local/share/virtualenvs/pythonic.me-GsTWWFLs/lib/python3.9/site-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
  File "/Users/robertsgreibers/.pyenv/versions/3.9.0/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/Users/robertsgreibers/.pyenv/versions/3.9.0/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Users/robertsgreibers/.pyenv/versions/3.9.0/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/robertsgreibers/projects/pythonic.me/9_python_requests/1.py", line 53, in <module>
    print(json.dumps(res.json(), indent=4))
  File "/Users/robertsgreibers/.local/share/virtualenvs/pythonic.me-GsTWWFLs/lib/python3.9/site-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
res.status_code:  401
res.json():

But if you’d update get_bearer method to use Authorization headers, you’d receive a successful response.

def get_bearer(self) -> requests.Response:
    res: requests.Response = (
        requests.get(
            url=f'{self.base_url}/bearer',
            headers={
                'Authorization': 'Bearer 1234'
            }
        )
    )
    return res

I hope this helped you,  here’s the successful response.

Bookmark this blog post page, use it as a template to start your API work any time and I’ll guarantee you’ll be more productive.

res.status_code:  200
res.json(): 
{
    "authenticated": true,
    "token": "1234"
}

Also, if you’re a part of any Python coding forums or groups on Facebook, Reddit, etc.

I’d appreciate it if you’d share this post! 📝 

What Makes Python Requests HARD To Understand?

If it’s a bit HARDER for you to grasp the concept of making an HTTP Python request it’s fine..

DON’T WORRY! 

It’s a bit HARD for everyone new to the development world..

If you’re trying to figure out how to become a Python developer alone, it’s obvious to me you’re going to STRUGGLE!

Web development is one of those skills where you need a helping hand and it’s okay to ask for help! 🐍

Your System: Making Python Requests EASY

To be honest, when you’re starting out with HTTP and API requests at all – you’ll struggle no matter what approach you choose!

It’s inevitable!

And it’s okay, DON’T be too hard on yourself..

🚨 You could be investing a lot of time into HTTP request research and STILL struggle to use Python requests library properly!

That’s just the nature of learning and understanding development concepts.

Concepts are fairly simple, but NOT always straightforward for newcomers.

⚠️ The real problem is you DON’T have a systemized approach to learning...

As a newcomer, you’ll learn a thing here or there, but won’t fully understand concepts like HTTP requests in Python.

Also, who’s there to answer your questions?

You should have A TON of questions while you’re learning to program..

And this is fine, MOST people starting out with Python and HTTP will have this problem. 

I used to have it, my students had that, it’s a NORMAL learning curve issue.. 

How To MASTER Python Requests FAST

I let you in on a SECRET.. 🔥

When I first started with Python years ago I used to think like you..

I used to work my Quality Assurance day job (even my first blog post is about it) and constantly work on my Python projects to learn more!

I thought books could help me.. I thought I can Google search Python examples online..

.. and eventually, I would become a Python developer by myself at some point..

But I was WRONG! 

What I actually needed was someone with Python experience to just guide me in the RIGHT direction!

The above screenshot is taken from inside my Python Mentoring Discord server.

This is where my students are able to ask any questions related to Python/Django and general software development career advice.

Back in the day, I dreamed about having a place where I could ask a coding-related question and get back a quick response.

I had none of that, I had to try and fail.. and I had to attend pointless coding bootcamps that didn’t lead anywhere..

And eventually, I had to just change jobs until I had the opportunity to be surrounded by experienced Python developers so I could learn from them.

THIS TOOK ME YEARS!

Help For Learning Requests In Python

This is a way of me giving back to the Python community! 🐍

I’m happy to share all the knowledge I’ve collected over the years that you most likely won’t find anywhere else online.

But I might NOT be around for a long time..

I have other goals besides Python programming & coaching.

I’m willing to discuss your situation and maybe even have a call with you if I can see potential in you.. 

⚠️ If you can find my contact info below this post, there’s a chance you can still reach out to me and I’ll see if I can help you in any way!

But be aware of the fact my time is quite valuable at this point..

People constantly offer me new job opportunities, my current company requires a lot of attention and I don’t have time for everyone’s BS.. 💩

Please DON’T SEND me messages if you’re NOT serious about your career and life!

HTTP Requests In Python Mentoring

A certain part of my Python Mentoring is where I teach students how to work with external APIs – It can be any API you choose to use in your project!

I show them the EXACT process I follow every day when I’m building payment integrations for the company I work for at the moment.. (explained earlier)

That’s just one part of the Python mentorship, overall my students have access to over 400+ documents I’ve personally written + videos where I explain the whole process of building your own projects from start to finish! 🔥

On top of that, we’re having LIVE Zoom calls every week to discuss common problems where students can directly ask questions through screen sharing and get LIVE code reviews from me. 

🚨 I know from my own experience.. 🚨

It’s hard to learn software development alone, close to impossible!

As I said, see if you can find my contact details below the post.. tell me your story and I’ll see if there’s anything I can do to help you! 👇🏻

If you DO NOT have at least a small percentage of support from more experienced developers, I’d suggest you find it as soon as possible!

Otherwise, you’ll just struggle alone, you’ll waste a lot of time..

This is fine if you want to waste years of your life on learning, but I believe years after getting hired as a developer are spent better..

THE CHOICE IS YOURS!

Talk soon!