Fixing 'Cannot Read Properties of Null (Reading'shadowRoot')' Selenium Web Scraping Error

Understanding and Fixing Common JavaScript Errors in Selenium

When web scraping with Selenium WebDriver, encountering JavaScript-related errors is not uncommon, especially when dealing with dynamic web elements like shadow DOMs. One frequent error that developers face is the JavascriptException: Cannot read properties of null (reading 'shadowRoot'), which often occurs when interacting with complex page elements.

This error typically arises when Selenium is unable to access or interact with elements inside a shadow DOM, a unique type of encapsulated DOM structure used by many modern websites for better modularity. In Python, using Selenium to control the browser can be tricky with such elements.

In the context of web scraping from platforms like Shopee, popups or banners often utilize shadow DOMs, which may be challenging to close programmatically. This problem can hinder the smooth flow of automated tasks and disrupt data collection.

This guide will walk you through a clear solution to address the 'Cannot Read Properties of Null' error and provide a practical approach to close popups embedded within shadow DOMs in Shopee using Python Selenium.

How to Handle Shadow DOM and Popups in Selenium Web Scraping

The scripts provided above aim to address a common issue encountered in web scraping with Selenium WebDriver when interacting with shadow DOM elements. A shadow DOM is a part of a web page that operates separately from the main DOM, often used in complex web components. In the context of scraping sites like Shopee, popups frequently appear inside shadow DOMs, which can lead to errors if accessed with traditional Selenium methods. The first script is designed to close the popup using JavaScript execution through execute_script(), a powerful tool that allows Selenium to run raw JavaScript within the browser context.

The key challenge is that elements inside a shadow DOM aren't accessible with common Selenium commands like find_element_by_xpath(). Instead, we use JavaScript to traverse into the shadow DOM using the shadowRoot property. The script targets the Shopee popup's close button by first accessing its shadow host element, and then querying its internal structure. By utilizing driver.execute_script(), the script is able to manipulate and close elements inside this isolated DOM. This solution works well when combined with explicit waits to handle dynamic page elements that load asynchronously.

The second script introduces WebDriverWait, an essential tool for managing the timing of dynamic page elements. Since Shopee’s popups load asynchronously, directly interacting with these elements can cause errors. To avoid this, WebDriverWait() ensures that the elements we wish to interact with are fully loaded and ready. This script waits for the presence of both the main DOM element and the shadow DOM elements. The method EC.presence_of_element_located() ensures that Selenium interacts with elements only after they are visible and present, which is crucial for avoiding null reference errors.

In both scripts, we handle error situations with a try-except block to ensure the program doesn't crash due to unexpected errors, such as elements not being found. Error handling is particularly important when scraping websites that frequently update their structure or change popup behavior. Additionally, these scripts follow best practices by terminating the browser session using driver.quit() after execution to avoid memory leaks or performance issues.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.common.exceptions import JavascriptException
import time
# Initialize WebDriver with Chrome
options = Options()
driver = webdriver.Chrome(service=Service(), options=options)
# Open Shopee website
driver.get('https://www.shopee.co.th/')
# Click the Thai language button
th_button = driver.find_element(By.XPATH, '/html/body/div[2]/div[1]/div[1]/div/div[3]/div[1]/button')
th_button.click()
# Pause to allow popups to load
time.sleep(3)
# Try to close the shadow DOM popup
try:
    close_button = driver.execute_script('return document.querySelector("shopee-banner-popup-stateful")'
                                      '.shadowRoot.querySelector("div.shopee-popup__close-btn")')
    close_button.click()
except JavascriptException as e:
    print("Error: ", e)
# Close the browser
driver.quit()

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
# Initialize WebDriver with Chrome
options = Options()
driver = webdriver.Chrome(service=Service(), options=options)
# Open Shopee website
driver.get('https://www.shopee.co.th/')
# Click the Thai language button
th_button = WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.XPATH, '/html/body/div[2]/div[1]/div[1]/div/div[3]/div[1]/button'))
)
th_button.click()
# Wait for the shadow DOM popup to be present
try:
    shadow_host = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CSS_SELECTOR, 'shopee-banner-popup-stateful'))
    )
    shadow_root = driver.execute_script('return arguments[0].shadowRoot', shadow_host)
    close_button = shadow_root.find_element(By.CSS_SELECTOR, 'div.shopee-popup__close-btn')
    close_button.click()
except Exception as e:
    print("Error closing the popup: ", e)
# Close the browser
driver.quit()

Handling Dynamic Content with Selenium WebDriver

Another key aspect to consider when working with Selenium WebDriver for web scraping is how to handle dynamic content that continuously updates or changes after page load. Many modern websites, like Shopee, use JavaScript to load and update content dynamically. This means that elements on the page might not be immediately available after the page loads. In such cases, Selenium’s default behavior of waiting for the page load event may not be sufficient. Using explicit waits like WebDriverWait can solve this issue by waiting for specific elements to appear or become clickable.

For scraping sites with popups, banners, or complex UI components that rely on shadow DOMs, it’s essential to know how to interact with them. These components hide elements within an isolated DOM structure that can’t be accessed by traditional methods like XPath or CSS selectors. Using the execute_script() command helps bridge this gap by allowing you to run JavaScript directly within the browser, giving you access to the shadow DOM and allowing for interactions with elements like close buttons or form fields within those hidden parts of the page.

Additionally, error handling becomes crucial in such cases. Websites can often change their structure, leading to broken scrapers. Proper use of try-except blocks in Python allows you to catch errors such as JavascriptException and handle them gracefully, ensuring the scraper doesn’t crash unexpectedly. Incorporating logging to capture the error details can help identify the root cause and resolve it in future scrapes.