Web Automation - dealing with navigation timeout in Open Twebst

The most important aspect of web automation is object recognition. In Open Twebst, FindElement method makes use of HTML tag name and attributes to find objects inside web pages. But before accessing the DOM, the web page must be loaded. Deciding when the web page is completely loaded is not always an exact science, the loading time depends on various things like:

• the speed of the internet connection
• the size of the HTML document
• the size and number of the images inside web document
• some web pages seem to update and load continuously
• sometimes Open Twebst library fails to detect if the page is completely loaded when the page embeds resources like PDF or office documents

To address page loading, Open Twebst has three useful properties: core.loadTimeout, core.searchTimeout and core.loadTimeoutIsError which affect Find methods and work together like this:

• wait a maximum core.loadTimeout milliseconds for the page to be completely loaded. If core.loadTimeout is zero don’t wait and start searching the element right away.
• if the page is completely loaded start searching the element in core.searchTimeout milliseconds. If the element is not found returns null.
• if the page is not completely loaded and core.loadTimeoutIsError is true (which is default value) then throw an exception. If core.loadTimeoutIsError is false try  to find the element in core.searchTimeout milliseconds; returns null if not found.
• searchTimeout is useful for elements that are dynamically created by web page scripts + Ajax after the DOM is created.

Usually we have to wait for the page to load before performing actions on HTML controls; some web pages just don't work if we access the DOM as soon as the controls are available because not all the content is loaded and the initialization scripts are not yet executed.

However it is possible to set core.loadTimeout to zero (don't wait for the page to load) and set core.searchTimeout to a greater value like 1 minute. This way the element is retrieved as soon as it is loaded.

Other related features: core.isLoading, browser.WaitToLoad.