Random ramblings about Mac, Python, TeX, programming, and more   |     |        |     |  


Web page interaction in Python

November 14, 2024  |  web, python, programming

Some of my routine tasks involve interaction with web pages, often to fill in one or more web forms with data that can be provided programmatically. With my previous email provider, I used this to add new email aliases to my email account. Now, I need it to send documents to our work printers. To ease these tasks, I developed a very simple (and limited) Python-based web interaction scripting language. These scripts can be executed using my webinteract Python module implemented with the Python framework splinter. The splinter framework provides an interface for web application automation. Default, the webinteract module uses the splinter framework with the chrome driver. For this to work, both Chrome and ChromeDriver have to be installed.

The webinteract module has also been tested with, and should work with, the firefox and edge drivers. For all of these drivers to work, the selenium module has to be installed (see ChromeDriver below for more details). All three drivers have been tested on macOS and Windows 11. If you have a problem with one of them, try one of the other two.

The webinteract module has been developed to fit my needs. New features have been added when needed. It is not made to be complete or robust, but it is made to be easily extendable. New action (command) types can be easily added by adding methods to a class in the module (or extending that class). For more advanced use cases, it might be better to use the splinter framework directly. But for some types of automated web page interactions, the webinteract module might be an easy-to-use approach.

The source code of the webinteract module is available here:

https://www.pg12.org/dist/py/lib/webinteract/

The easiest way to install and use the module is to use pip (see webinteract at PyPi): 

pip install webinteract

This pip command will install the module, and a console script webinteract to use the module as a program. If you execute webinteract without a script as an argument, it will open a browser window and present you a prompt in the terminal where you do web interaction. Try either the -h argument to webinteract or type help in the prompt.

Table of contents

In the following text, I will first introduce web interaction actions and how you write scripts for web interaction tasks. Next, I will show how to run such scripts from the command line. Then, I will document the currently available web interaction actions of the module. At the end, I have added a section with some comments and hints for maintaining and troubleshooting the software involved.

Web interaction actions

My webinteract module interprets and executes web interaction scripts. The web interaction script commands are called web interaction actions, and files containing such scripts use the wia extension. This is an example of a wia script with web interaction actions:

setvals(url="https://a.web.page/")
setvals(account="an@email.address", pw="s3crEt p4s5w0rd")
visit(url)
fill(account, "id", "loginForm:id")
fill(pw, "id", "loginForm:password")
click("id", "loginForm:loginButton")
verify( \
  is_text_present("Login succeeded", wait_time=10), True, \
  "Login failed")
fill("A text", "id", "textForm:field1")
fill("Another text", "id", "textForm:field2")
click("id", "registerForm:addText")
verify( \
  is_text_present("Add text failed"), False, \
  "Add text failed")

The actions in the first two lines of the script set the values of three variables, url, account and pw. In line 3, the action opens the web page at the address given in the variable url. The actions in lines 4 and 5 fill in the two fields with the IDs "loginForm:id" and "loginForm:password" in the web form with values from the variables account and pw, respectively. The action in line 6 clicks a login button (with the ID "loginForm:loginButton"). In line 7 to 9, we perform a check: is the text "Login succeeded" present on the web page? Since it might take a while for a web page to completely load (after the action at line 6 pressed the button), we specify that the check should wait (at most) 10 seconds for this to become true. These three lines combine two web interaction actions: verify and is_text_present. The result of the is_text_present action is the first argument to the verify action. We also see that these three lines are actually combined to one line when interpreted. The \ at the end of a line removes the line break (and any extra spaces at the beginning of the next line) before the combined line is interpreted. We expect the result to be true; if not, the script fails and terminates here. The following actions in lines 10, 11, 12, 13, 14 and 15 fill in another web form with two text fields and check that the error message "Add text failed" is not present after the register button is pressed.

Use the module as a program

There are two approaches to using the web module as a program. This first approach can be used if you have downloaded the module file from my file repository. The second apprach can be used if you intalled the module with pip.

Install module manually

The preferred way to install the module is using pip (see Install module with pip); however, a guide to install it manually explains some of the magic.

If the wia script example above is stored in a file webinteraction-example.wia it can be executed using the following command:

python3 webinteract.py webinteraction-example.wia

If the module is installed as the program webinteract at a location found by your shell, a simpler way to execute the script is possible:

webinteract webinteraction-example.wia

The first line of the webinteract.py module contains this magic string:

#!/usr/bin/env python3

The magic string says this is a Python program, and the system knows it should use Python 3 to execute the program. When this file is copied with the name webinteract to a location (path) where your shell looks for executables, the program can be executed from the command line independent of the current directory of your shell. You have to ensure that the program file is executable. If the directory ~/bin is one of the directories your shell is looking for executables, you can make webinteract.py such an executable with the following two steps:

cp webinteract.py ~/bin/webinteract
chmod +x ~/bin/webinteract

The first line copies the file to a location where your shell will find it and renames it to webinteract. The second line makes the file executable. The locations your shell looks for executables (programs) can changed by updating the environment variable PATH. It is out of the scope of this blog post to discuss how this is done.

If the webinteract program is installed at a location found by the shell, we can further simplify the executions of such scripts using the magic string trick again. Instead of using webinteract at the command line to execute a wia script, we can insert a magic string into the script itself. Insert the following magic string as the first line in your wia scripts:

#!/usr/bin/env webinteract

If you have made the script executable with the chmod command, you should be able to execute the script from the current directory directly:

./webinteraction-example.wia

The webinteract program (the webinteract Python module used as a program) provides a few command line arguments that can modify its behaviour (see the documentation of all command line arguments below). We can, for example, run the example script headless using this command:

webinteract --headless webinteraction-example.wia

Or, if we have applied the necessary magic to the script itself, this will also work:

./webinteraction-example.wia --headless

Install module with pip

The preferred way to install webinteract on your system is using pip:

pip install webinteract

When you have installed the module with pip, and the version you have installed is version 1.29 or newer, a console script webinteract will be generated and available on your system. This script behaves exactly like the webinteract program above (see Install module manually). This also means that when the module is installed with pip, you can use the same magic string in your wia scripts as described above:

#!/usr/bin/env webinteract

And, of course, command line arguments will still work in the same way:

./webinteraction-example.wia --headless

Use namespace and the keychain with command line arguments

A more advanced usage of command line arguments to the webinteract program might involve your computer's keychain (system keyring) and a provided namespace where variables are assigned values. Let us start with this wia script example add-alias.wia. Remember to make the script executable, and if the moudle is installed with pip use the alternative magic string in the first line:

#!/usr/bin/env webinteract
visit("https://email.provider/controlpanel")
fill(account, "id", "loginForm:id")
fill(pw, "id", "loginForm:password")
click("id", "loginForm:loginButton")
verify( \
  is_text_present("Login succeeded", wait_time=10), True, \
  "Login failed")
fill(alias, "id", "textForm:alias")
click("id", "registerForm:addAlias")
verify( \
  is_text_present("Add alias failed"), False, \
  "Add alias failed")

In this example, we log in to a web page with an account and a password. If this succeeds, we add an alias and verify that the error message does not occur. In the example, three unset variables are used: account, pw, and alias. The pw variable is the password used to log in. This should not be stored in clear text anywhere. To avoid this, webinteract can fetch the password from a keychain. It uses the keyring module to implement it. By providing the account name and service name used when the password was registered in the keychain, webinteract will fetch the password from the keychain and store it in the variable pw. When executing the script, the two other unset variables can be given values using the namespace argument. If the log in password is stored under the account name my@email.address and the service name email.provider, an example of running the script could be:

./add-alias.wia \
    --pw-account my@email.address \
    --pw-service email.provider \
    --name-space '{"account": "my@email.address", "alias": "new-alias"}'

Since the account name used to access the password from the keychain is the same as the account variable in the namespace, we can simplify the usage of the script by removing the account variable from the namespace and use the pw_account variable instead:

./add-alias.wia \
    --pw-account my@email.address \
    --pw-service email.provider \
    --name-space '{"alias": "new-alias"}'

In the script, we then have to change line 3 to use the pw_account variable instead of the account variable:

fill(pw_account, "id", "loginForm:id")

Interactive usage

The webinteract module can also be used as an interactive program with a prompt where you can write your web interaction actions. For example, start the webinteract program with the command webinteract (no wia file given) and you will be given the following prompt:

wia>

The web-browser will start with no page loaded. In this prompt, you can type in the web interaction actions and watch the effect of the actions in the open web-browser window:

wia> visit("https://a.web.page/")
wia> click_link("partial_href", "news")

You can also pass arguments to the program when used in interactive mode. For example, if log-in information is needed, you should grab it from the keychain:

webinteract --pw-account my@email.address --pw-service email.provider

Then you can use this information interactively in the web interaction actions:

wia> visit("https://a.web.page/")
wia> fill(pw_account, "id", "loginForm:id")
wia> fill(pw, "id", "loginForm:password")
wia> click("id", "loginForm:loginButton")

The interactive prompt is also useful when debugging your scripts. For example, I use it to see what find actions return or the value or text of web elements of a web page. If an action returns a value, the value is displayed in the interactive prompt. Some examples (including the values displayed in the prompt):

wia> setvals(select_elem = find("id", "selector"))
wia> element_get_value(select_elem)
'tag'
wia> element_get_text(select_elem)
'tag\nyear\nsearch'
wia> get_text("tag", "h1", 0)
'A web page'

Besides the web interaction actions, a few other commands are available when webinteract is used interactively. These commands are used to print and manage the action command history and to show help texts:

If you add an action after the help command, the help text for that specific action is provided (e.g., the command «help click» print the documentation for the click action). With the help command, the interactive prompt of webinteract is a nice way to access the documentation of the webinteract module and its web interaction actions.

Scrape web page content

Another usage of the webinteract module is to scrape a web page for content (data). In this example script, get_text.wia, we get the content of the third paragraph on the web page:

#!/usr/bin/env webinteract
visit("https://a.web.page/")
get_text("tag", "p", 3)

If we run this script without any arguments, the content is written to standard output (stdout). If we want to write it to a file, we can use the --output-file argument:

./get_text.wia --output-file thetext.txt

The output is then written to the file thetext.txt. In interactive mode, we can use the --output-file argument to save the results to a file:

webinteract --output-file thedata.txt
wia> visit("https://a.web.page/")
wia> get_text("tag", "p", 3)
'The text at paragraph three on the web page.'
wia> exit
cat thedata.txt
The text at paragraph three on the web page.

Command line arguments

Positional arguments:

Options:

The web interaction actions reference

The webinteract module provides the following web interaction actions:

Above, you can see that almost all actions with an argument pair SelectorType stype and its value sval have a version where an ElementList is the argument instead of these two arguments (this is also true for actions with a LinkType/link-value argument pair). The exceptions are actions that only search for elements (e.g., find and is present actions). An example of an action with an ElementList version is the fill action. It has the companion element_fill. The element_ versions of these actions can be combined with variables set to web elements, or actions returning web elements. The two lines below perform the same interaction with a web page (fill in the text "LaTeX" in the web element with the id "searchtext"):

fill("LaTeX", "id", "searchtext")
element_fill("LaTeX", find("id", "searchtext"))

In the listing of the web interaction actions from the webinteract module, the following non-standard Python types were used in the type hints:

Comments and troubleshooting

When executing wia scripts, a lot of different software and systems are involved. This includes a web browser and a matching WebDriver (a remote control interface that enables introspection and control of user agents), Python 3, a keychain, and so on. This section includes comments and hints for maintaining and troubleshooting this software complexity.

ChromeDriver

If the webinteract module is used with ChromeDriver, the Chrome browser and a matching version of ChromeDriver have to be installed. To ensure that the Chrome browser is up-to-date, go to the About page of the browser. To ensure that the ChromeDriver is up-to-date on my Mac, I install it (and keep it updated) using Homebrew:

brew install chromedriver
pip3 install selenium
chromdriver		# run "chromedriver" once to get system approval

On a Mac, you might run into a problem since the ChromeDriver is not properly signed:

warning chromedriver

To solve this, we have to make an exception for chromedriver (if we trust the installed chromedriver program). In macOS 14 or earlier: locate the chromedriver program file in Finder, control-click on the file, choose Open from the shortcut menu, and click the Open button on the pop-up. In macOS 15 or later: Open System Settings, choose Privacy & Security in the left panel, scroll down to the Security section, and press the Open Anyway button on the right side of the text about blocking ChromeDriver (for this to appear, you have to run chromedriver at least once):

You might have to do this every time the Chrome Browser and the ChromeDriver are updated.

In the steps above installing ChromeDriver, I also installed the Python module selenium. This is a Python binding for WebDrivers, and it supports Chrome and ChromeDriver. The splinter framework uses the selenium module.

Keychains and the keyring module

The keyring module supports macOS Keychain, Freedesktop Secret Service (requires SecretStorage), KDE4/KDE5 KWallet (requires dbus), and Windows Credential Locker. Other keyring services are available through third-party backends. I have only tested the webinteract module with the macOS Keychain.

You can use the keyring console script to add items to the keyring service. If you want to add a password for the examples above, you can use this command:

keyring set email.provider my@email.address

When using macOS, you can also use the built-in command line tool security directly to add items to the macOS Keychain. If you want to add a password for the examples above to the macOS keychain, you can use this command:

security add-generic-password -a my@email.address -s email.provider -w

The -w flag at the end makes the security tool prompt you for the password.

Windows computers

Windows computers are now fully supported (since version 1.31). You should install webinteract with pip on Windows computers. The Python module and the webinteract console script will both be available after you have installed the module with pip. Try webinteract --version in the command prompt on your Windows computer.

On Windows computers, you can use the keyring console script to populate the keyring with entries (passwords) used in the wia scripts (see Keychains and the keyring module above).

Last updated: December 17, 2024