Wait For Elements In Python Selenium (5 Powerful Examples)

Alright, welcome back to the blog, Roberts Greibers here, and in this post, I want to talk about five powerful examples you can use in Python Selenium to wait for elements when working on automation projects.

Whether you’re just getting started as a Python developer, or you are already in the industry as a test automation engineer and looking for more effective ways in Python selenium to wait for element – this post is for you!

These are not just things that I’ve just randomly come up with!

These are not just theoretical concepts!

These are code examples in Python selenium to wait for element that I have saved in my notes since a long time ago and I literally used them back when I used to work on test automation projects. 

The code examples I’m about to show you are few of the best ways in Python selenium to wait for element that allowed me to improve the speed of my test automation scripts!

I’m going to show you a code to:

• wait for element to be found (any situation)
• wait for element to be visible
• wait for element to be clickable
• wait for element to load
• wait for element to be displayed

… which one to use depends on a specific situation you’re working on. But I can assure you, in the end, these code examples led me to write more advanced Selenium automation test cases. 

And NO! It’s not going to be about using just the typical time.sleep(1) solution. I’ll show you where it’s appropriate to use time.sleep(1) and I’ll show you the best ways I know in Python Selenium to wait for element.

import time

time.sleep(1)

I’ve been working professionally as a Python developer for more than 5 years – a part of it was as a quality assurance engineer where I worked with Python and selenium framework a lot. 

Eventually, I became a Python/Django developer (see my Linkedin here) and now I’m also helping relatively new engineers (quality assurance engineers, analysts, etc.) to become Python developers. (see a cut from a LIVE mentoring call)

If you like the idea of becoming a Python developer, learning about Selenium framework tricks, and eventually maybe even becoming a Django developer, then this post is for you!

Python time wait for element

When you’re just a beginner Python developer and you’re Google searching all the time for Python selenium wait for element code examples or any other cases when you have to wait in Python, the first thing that you will find is a built-in package called time. 

And if you use time package in Python selenium to wait for element, I’m not gonna blame you, it’s really NOT your fault that you don’t know any better. 

import time

time.sleep(1)

The above code waits for 1 second and then continues your Python script execution.

… Most of the time, using time.sleep(1) in Python selenium to wait for element is NOT the best way of handling such a situation.

But as a beginner you really don’t have a lot of choices either… you just google search “python selenium wait for element” and try to come up with something, right? At least that’s how I started.

No one really will tell you that you’re probably developing an unsustainable code that will often break and won’t be accepted in a real development environment when you’re working for a big tech company. 

Why you think you’re doing all the “right things”

All these bad habits are coming from the fact that you think you’re already doing all the “right things”.

Because you’re probably constantly surrounded by co-workers and other people around telling you what to do. (This, of course, depends on your environment and people around you – it’s not always the same case for everyone)

But it doesn’t mean that people around you know what they’re talking about. And when I say this – I don’t mean people around you are wishing you bad things or that they’re bad people – they just probably don’t know any better, but still want to help you.

And the things you hear might sound something like this:

• Just build your portfolio of projects (you can find Python tutorials online)
• Just use google search “python selenium wait for element” and you’ll find a solution
• Practice building your own projects – build a calculator app or automate a website with Python selenium
• Or… build a to-do app and include it on your portfolio page
• You can do it on your own, you don’t need anyone to help you – see that guy did it!
• Or the worst one: Take a one-week Python Selenium training and you’ll be good

And you might even think…

Well, there’s a lot of information about Python out there, I could probably do google research, practice Python selenium every day for a few hours and eventually, I’ll become a Python developer.

It’s not like I’m lazy, I’m pretty smart too – Yes, I can do it on my own.

I used to think the same and I even tried to do the same, but let me tell you why this is a bit of bad advice in a second…

The real reason your code is BAD

I’m going to give you an example of how and when I’d use time.sleep(1) in Python selenium to wait for element and when it would be appropriate for the situation in a short moment… 

But keep in mind, the real reason why your Python code is BAD and why you’re probably not a Python developer yet…

• Is NOT because you’re not smart enough to become a Python developer on your own (you’re probably pretty smart if you’re reading this post) 
• Is NOT because you haven’t practiced Python coding enough on your own
• is NOT because you don’t have a Python portfolio of to-do and calculator projects
• And it’s also NOT because you don’t work hard enough (aspiring Python developers are usually the hardest working people I know)

The REAL problem here is you don’t have enough FEEDBACK and proper Python code examples – real solutions – used in real Python projects.

Think about it, when was the last time someone told you:

“Your code is bad here, let me show you a better way!”

And it’s NOT really your fault, how could you know when you’re just starting out?

Even if you have managed to learn a lot of Python on your own – you might have hit a glass ceiling of knowledge or might hit one soon. 

By glass ceiling of knowledge, I mean, there’s a limit to the Python knowledge you can find online and learn on your own. 

Python time package (appropriate usage)

The best “tricks” or ways of Pythonic coding – writing code more professionally you can only learn from more experienced Python developers. 

The moment I realized this, it completely changed my career and I had to find ways to reach out to more experienced Python developers – people who are where I wanted to be. 

The first person you can reach out to – is me – follow me on Linkedin, send me a DM if you have any questions and I’ll see if I can help you out.

So, let me give you an example of when using time.sleep(1) would be appropriate. 

This is what I usually do with my clients – I let them make mistakes, use their own solutions – realize it’s not going to work – and then I’ll give them the best solution I know and explain why it’s the best. 

You can see that in a recent call we did where it’s clearly working and it’s making a large difference in the learning process for a client of mine… By the way, she just started to learn about Python and development and she’s already coding like a PRO.

The code example below is taken from a real project I’m working on right now.

You won’t be able to execute it as it’s taken out of the context of the project. I’ve replaced real classes and objects with sample class objects. (see explanation below to get the idea of what’s going on here)

import time

pending_serializer = 'example_serializer'


class EPurchaseActionExecutionFailed(Exception):
    pass


class RelatedAttempt:

    @property
    def successful(self):
        return


class Operation:

    @property
    def related_attempt(self):
        return RelatedAttempt()

    @property
    def related_attempt_successful(self):
        return self.related_attempt.successful

    def refresh_from_db(self):
        pass


class Purchase:

    def refresh_from_db(self):
        pass


class Spec:

    def operation(self):
        return Operation()

    def serialize_finalized(self, pur, op):
        pass


def example_function():
    op = Operation()
    pur = Purchase()
    spec = Spec()

    start = time.monotonic()

    while True:
        if time.monotonic() - start > 50:
            return pending_serializer

        op.refresh_from_db()
        
        if op.related_attempt is not None:

            if not op.related_attempt_successful:
                raise EPurchaseActionExecutionFailed(spec.operation)

            pur.refresh_from_db()
            return spec.serialize_finalized(pur, op)

        time.sleep(1)

Assume you have a function called example_function and in Python selenium you want to wait for element using that function.

In this case, you can define a start = time.monotonic() variable to catch a starting point of wait time.

Then, with while True: you can create a context of an infinite loop – meaning, it will repeat the code under while indent until you somehow break out of it.

(lines 54 – 55) – Here you actually can define the “break out” point with an if statement.

        if time.monotonic() - start > 50:
            return pending_serializer

with time.monotonic() you get the “new” current time on each while loop iteration.

And you subtract the time amount from current time using start variable to get the difference – meaning, you get the “already waited time” here and check it if it’s larger than 50.

If it is larger, simply return pending_serializer in order to exit example_function (you can also exit just by having an empty return statement – depends on how you want your function to work)

(lines 56 – 66) – here you can have the context of what are you waiting for – you can have different ways of waiting for element or thing to happen. In the case of the example above, we’re refreshing database objects in Django and waiting for them to change.

Of course, you don’t want to refresh a database as fast as you can – it can cause problems, that’s why in this case you can use time.sleep(1) to have a one-second delay between each database refresh.

Also, keep in mind, this is a very specific case where time.sleep(1) worked with a limitation that we break out of an infinite loop if the wait time reaches 50 seconds.

In general, you have to be careful with using time.sleep(1) inside a serious project – depending on the context of your project or feature you’re working on there might be better ways than time.sleep(1), but that’s a topic for another day.

Take the above code as an example to wait for any kind of change in your Python projects – the same code example can be applied in Python selenium to wait for element.

Python wait for element (any situation)

I can’t give you an example for any situation and you should never apply any techniques blindly. You always have to consider the context of the code you’re working on.

And by context, I mean, you have to think – what is the code I’m writing supposed to do?

Especially for solutions like time.sleep(1) – you should always think – what are the cases when time.sleep(1) will break?

What if X happens? What if Y happens?

Before you commit to using a code, you have to make a decision. 

And a coding decision can’t be taken blindly without considering possible breaking points as it in the future can cost a lot of money for the tech company you’re working for. 

This is the mindset I always try to bring to my clients. When you’re coding by yourself and try to come up with your own ideas – there’s no one to save you from yourself!

If you make a dumb decision and write code without paying too much attention – you will make mistakes and introduce bugs that you won’t be even aware of and that’s the WORST situation to be in!

Also, realize in order to become a real professional Python developer, you have to be a well-rounded person.

You can’t just copy-paste code and call yourself a developer, there’s way more to it, my client said it really well when he texted me about the experience he has had working together with me, see the screenshot below:

ALWAYS think from all angles before you commit to a code solution!

Python selenium wait for element (time package)

When you’re in a situation where you have to wait for element – using time package is one of the ways to do it – not the best, but still possible. (explained why above)

Here (selenium example below) you’re applying the same principles explained above – the appropriate way of using time.sleep(1) – as you can see it can work your way and wait for element in Python selenium.

import time

from selenium import webdriver
from selenium.common.exceptions import (
    NoSuchElementException,
    ElementNotVisibleException
)
from selenium.webdriver.common.by import By

browser = webdriver.Firefox()
browser.get('https://www.yourproject.com/')


def wait_for_element(element_id):
    while True:
        try:
            return browser.find_element(
                by=By.ID,
                value=element_id
            )
        except (NoSuchElementException, ElementNotVisibleException):
            time.sleep(1)

This could be your “custom wait for X” function that you can apply also outside of Python selenium.

So, while loop with a True boolean value is an infinite loop.

Inside an infinite loop, you’re going to try to find an element using browser.find_element() method.

Everytime browser.find_element() will fail you’re going to get NoSuchElementException or ElementNotVisibleException

(depends on your situation – could be a different tuple of exceptions – so you have to adjust this exception list to your situation)

If the exception happened you catch it with try / except and in a case of exception indent under except is going to be executed.

This is where you can wait for a second with time.sleep(1) and repeat the while loop until exception doesn’t happen anymore and you just exit the function with the element you found using return statement at line 17.

So this is something that could work for you in any situation…

It’s just that I still have better examples to give you later in the post – Selenium framework gives you its own tools to wait for elements which generally is a better approach than using time.sleep(1)

But with that being said, time.sleep(1) is also a solution you can consider if for some reason Selenium framework tools don’t work.

Python selenium wait for element to be visible

When it comes to waiting for element to be visible in Selenium you have to use what framework has provided for you – otherwise, you’ll have a bad time!

For example, recently I dug into exploring the whole process of amazon.com product scraping – I was looking for ways to scrape amazon products by categories using search menu and selenium, of course.

One of the first problems I had to deal with was about how to handle an element invisible exception called ElementNotInteractableException – it usually appears when you have to wait for the element to be visible. 

Here’s a full code example you can use in Python selenium to wait for element to be visible. Let me give you details about how it actually works down below…

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

browser = webdriver.Firefox()
browser.get('https://www.yourproject.com/')
browser.maximize_window()

wait = WebDriverWait(
    driver=browser,
    timeout=15,
)
element = wait.until(
    method=EC.visibility_of_element_located(
        locator=(
            By.ID,
            'element_id'
        )
    )
)
element.click()

First, you’ll fire up your browser with lines 6, 7, 8 – depending on your situation there will be more steps and this part will be extended to a couple of more actions.

Probably extended as far as you’ll reach the part of the code where you will in Python selenium wait until element is visible.

Use Selenium WebDriverWait

Now, you have reached the part where you want to wait until element is visible. It’s time to use Selenium WebDriverWait.

Selenium’s WebDriverWait object gives you an option to define certain conditions – Expected Conditions. There’s a large difference between using time.sleep(1) and using expected conditions.

By the way, you can get access to expected conditions with the following line. Also used in the code example above.

from selenium.webdriver.support import expected_conditions as EC

You see – when you just use time.sleep(1) it does not have any constraints, it just waits for a second and continues.

Use visibility_of_element_located()

While in the meantime, expected conditions can let you wait until a certain expected condition is reached, for example, wait until element is visible by using a method called .visibility_of_element_located()

method=EC.visibility_of_element_located(
    locator=(
        By.ID,
        'element_id'
    )
)

One thing I have to mention is how the locator works.

If you dig deeper and explore how Python Selenium wait until element is visible code is developed – meaning you just open Selenium framework methods and read the code.

… You’ll find that methods like .visibility_of_element_located() are just a combination of other Selenium methods.

I’m talking about _element_if_visible, driver.find_element() where you can see clearly that .find_element() method requires two arguments by= and value=

Docs for visibility_of_element_located()

And if dig into exploring .visibility_of_element_located() method in documentation, you’ll see how it works. In general, I recommend you to try to understand WHAT you’re trying to develop and HOW tools you use actually work – will always make you a better developer.

Here’s the code just for reference.

def visibility_of_element_located(locator):
    """ An expectation for checking that an element is present on the DOM of a
    page and visible. Visibility means that the element is not only displayed
    but also has a height and width that is greater than 0.
    locator - used to find the element
    returns the WebElement once it is located and visible
    """

    def _predicate(driver):
        try:
            return _element_if_visible(driver.find_element(*locator))
        except InvalidSelectorException as e:
            raise e
        except StaleElementReferenceException:
            return False

    return _predicate

Python selenium wait for element to be clickable

The next powerful way in Python Selenium to wait for element to be clickable is also kind of similar to a way of waiting until element is displayed on a page.

You literally don’t have to change a lot in order to wait for element to be clickable.

Take a look at the working code example, I’ll give you a bit of detail below…

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

browser = webdriver.Firefox()
browser.get('https://www.yourproject.com/')
browser.maximize_window()

wait = WebDriverWait(
    driver=browser,
    timeout=15,
)
element = wait.until(
    method=EC.element_to_be_clickable(
        locator=(
            By.ID,
            'element_id'
        )
    )
)
element.click()

Use element_to_be_clickable()

.element_to_be_clickable() is another one of those expected conditions, take a look at the whole list down below. This will give you an overall idea of how you should use expected conditions.

The screenshot below is taken from Selenium documentation page.

So, again here – all you need is simply change the expected condition you use and you should be good.

method = EC.element_to_be_clickable(
    locator=(
        By.ID,
        'element_id'
    )
)

Docs for element_to_be_clickable()

And similar to the example of how in Python Selenium wait for element to be visible, you also want to explore the code that’s been put together to wait for element to be clickable.

Python selenium wait for element to load

Sometimes there’s no other way in Python selenium to wait for element to load until you can continue. Also, not always there’s a way to wait for a specific element to load.

This is what I recently noticed when I was working on amazon.com product scraping. It was just easier to wait for the page to load until you let the script continue the execution.

Use driver.execute_script()

Notice, in my Selenium automation scripts I like to assign Selenium’s driver to a variable called browser – it just makes more sense in my head when I read the code.

But the idea stays the same.

You can use driver.execute_script() to execute JavaScript code inside a page you’re browsing with Selenium driver.

ready_state = None

while ready_state != 'complete':
    ready_state = browser.execute_script(
        'return document.readyState;'
    )
    time.sleep(1)

How to wait for page to load

And if you’re familiar with JavaScript a bit, you’ll know that 'return document.readyState;' is just a way to determine if the page is completely loaded.

So, essentially the above code with a while loop is doing 1-second iterations and in each iteration, you’re checking if the page is loaded.

If a page is not loaded yet, meaning 'return document.readyState;' is still not returning a 'complete' string, you keep waiting.

I was working on a multi-page Amazon product scraping where it was better to just wait for a page to fully load before continuing any further and the above code example solved the problem.

More details about how it works you can find in my Amazon product scraping blog post.

Python selenium wait for element to be displayed

When it comes to waiting for a specific element to be displayed in Python Selenium there are a couple of ways you can go about it.

So essentially it all comes down to a specific situation and the best recommendation I can give you is to use a combination of the following expected conditions.

And when I say a combination, I mean exactly that, depending on the situation you might need to check for a multiple of these expected conditions to determine if the element is displayed.

It comes back to what I have already mentioned before – you need to wait for surrounding elements to be visible in order to find exactly the element you’re waiting for to be displayed.

Plus, I’d also recommend using the previous point – wait for a page to fully be loaded first before you try to wait for element to be displayed.

And this depends on each situation, code would be very similar to the previous examples, just with more expected conditions.

Nothing works, what to do?

Ultimately, every time I share code examples with practical usage and theory, heavy posts such as this one, a lot of people just kind of skim through the content, and essentially they don’t do anything with the code that’s provided here. 

Find someone to help you!

And if you know me from Linkedin or just searching Python-related topics through Google, you might already have noticed, I’m always big on executing, I’m always big on implementing the steps you need to take!

No BS, just real talk!

I’m always just giving you code to implement, and it doesn’t matter if you’re NOT perfect at it or NOT ready for it, YOU HAVE TO START TODAY!

And I’m here to help you with your journey, start with posts here and if it’s still too hard for you, you can always send me a message on Linkedin – I’m usually very busy, but eventually, I’ll get to your message.

Conclusion

But either way, whatever it is, go take action, go implement these things!

Whether you do that alone or with me, it’s your choice – But with me is what I actually would recommend you.

Just because I’m really good with Python/Django and I could save you a lot of time and struggle – as you already might know, overall this has been my focus for the past 5+ years. 

There are so many lessons I’ve learned over this time. Lessons you can’t usually find online, you only can learn them from the experience or a mentor.

Which is also a reason why am I sharing it here in this blog and with my clients.

Take action and don’t give up!

Just take action! And don’t give up! Don’t you ever goddamn give up!

Because every time I take the time to write up a post, I remember the first post I wrote here which was about 5 years ago.

When I first started – I didn’t know anything compared to now, but I was still willing to share what I learned just to refresh my mind and let the knowledge set in.

Don’t let your dreams go to waste!

When I’m writing a post here, I always think back to the times when I didn’t have the money, when I was a poor student just griding out.

…And trying to work on my development skills. 

I didn’t have any real experience. 

If I look back, it’s so amazing to kind of know what you’ve done and how far you’ve come and how much you suffered, how much you sacrificed to be where you are right now. 

Please don’t let your dreams go down to waste, go execute on those!