How to automate existing instances of Internet Explorer - Tutorial(4)

As explained in a previous tutorial, a web macro can start by connecting to an already running instance of Internet Explorer browser. A web macro usually starts a new Internet Explorer page and then navigates to a URL to begin with. There are times when automating an existing IE window is a more natural alternative. Let's start with a JScript/WSH simple web macro:

// Create core object.
var core = new ActiveXObject("Twebst.Core");

// Find a browser for which the displayed URL contains "yahoo".
core.useRegExp = true;
var b1 = core.FindBrowser("url=.*yahoo.*");
b1.Navigate("www.google.com");

// Find a browser for which the title is "Google" (exact match).
core.useRegExp = false;
var b2 = core.FindBrowser("title=Google");

b2.Close();

Basically FindBrowser method searches for a browser page based on TITLE and/or URL. If core.useRegExp is false then an exact search is used. If core.useRegExp is true then the search uses regular expressions. Once connected to a IE browser instance, you get a Browser object to play with.

Of course, this works for IE6, IE7, IE8 on Windows XP, Vista and Windows 7 (32 and 64 bit) and it's compatible with Internet Explorer Protected Mode.

What can you do with a Browser object? Quite a lot of things: Navigate to a URL, FindElement and FindFrame inside it, Close the browser, get information about the browser, automate modal and modeless HTML dialgos and wait to complete navigation and loading.

Enough talk! It's time to Download Twebst Automation Studio and see it for yourself.

.

IE macro - how to get programmatic control over HTML elements - Tutorial(3)

Finding elements inside HTML frames/iframes hierarchy with Twebst Web Automation Library

Once you have started a web macro and you've got a Browser object to play with, it's time to get programmatic access HTML controls inside the web page. As any web developer should know, you can easily find elements inside a document using document.getElementById.

In the world of web automation things are a bit different; web macros need to access HTML elements in any web document and at any level of DOM hierarchy. The target HTML element can be inside frames/iframes loaded from various domains, subject of cross-frame scripting security restrictions (see my older posts: "When IHTMLWindow2::get_document returns E_ACCESSDENIED" and "When IHTMLWindow2.document throws UnauthorizedAccessException").

Here's a simple VBscript web macro that automates a Google search in Internet Explorer:

Dim core
Dim browser
Set core    = CreateObject("Twebst.Core")
Set browser = core.StartBrowser("http://www.google.com/")

Call browser.FindElement("input text",   "name=q").InputText("codecentrix")
Call browser.FindElement("input submit", "name=btnG").Click()

What is so great about the code above is the way FindElement method works: first it waits for the browser to load the HTML document then it searches for the element thru all frame/iframe hierarchy.

Once an Element is found you can perform actions on it like: Click, RightClick, InputText, Check, Uncheck, Select, Highlight etc. What is even cooler than this, is that you don't have to write these statements by yourself. They will be automatically generated by Twebst Web Recorder which you can get for free:

Get Free Web Recorder for Internet Explorer

.

WSH and clipboard access

I did some Windows Script Host programming recently and I was pleasantly surprised by its power, features and flexibility. One thing that I couldn't accomplish was accessing the clipboard from WSH. Digging the internet I found some solutions like this one based on Internet Explorer Automation. There are several problems with this approach as you can read in my article about Internet Explorer Automation: What's wrong with Internet Explorer Automation?

My solution for scripting the clipboard content in WSH is a regular COM object created with VC++ and ATL.

Download full source code and compiled DLL: WSH_clipboard.zip
To install the COM object run register.bat

I found scripting the clipboard useful enough to add this feature to the next release of Twebst Web Automation Library.

Free Web Macros for Internet Explorer

As I presented in my previous post, automating Internet Explorer can be a difficult task.
Twebst Web Automation Library can make things easier.

It gives full programmatic control over the Internet Explorer browser. Twebst is a library of COM object that can be used within any environment that supports COM, from scripting languages (JScript, VB Script) to high level programming languages (C#, C++). For more information, see Twebst Libray Online Documentation. And yes, it's free!

Get it FREE!

What Twebst can do?

• increase productivity by automating repetitive web tasks
• automate regression testing of web applications
• automate web actions and data-entry
• automatically log in to different web sites
• fill out web-forms automatically
• extract data from web pages (web scraping).
• monitor web pages

Twebst features

• Start new browsers and navigate to a specified URL.
• Connect to existing browsers.
• Search and access HTML elements and frames inside browsers.
• Intuitive names for HTML elements using the text that appears on the screen.
• Advanced search of browsers and HTML elements using regular expressions.
• Perform actions on all HTML controls (button, combo-box, list-box, edit-box etc).
• Simulates user behavior generating hardware or browser events.
• Get access to native interfaces exposed by Internet Explorer so you don't need to learn new things if you already know IE web programming.
• Synchronize web actions and navigation by waiting the page to complete in a specified timeout.
• Available from any programming or script language that supports COM
• Optimized search methods and collections.
  

What's wrong with Internet Explorer Automation?

The Microsoft Office products (Word, Excel, Power Point, Access, Outlook) allow their users to manipulate Office documents from Visual Basic or Visual Basic for Applications (VBA) code. It is possible to write a VBA macro in Excel that initializes a series of cells, and uses the cells to display a chart for instance.

Automation is the process of controlling one product from another product with the result that the client product can use the objects, methods, and properties of the server product. The client has access to the object model of the server.

Though Internet Explorer browser is not part of the Office suite, it supports automation. Here is a short sample:

// Create an IE automation object.
var ie = new ActiveXObject("InternetExplorer.Application");

// Make it visible and navigate to a given URL.
ie.Visible = true;
ie.Navigate("http://www.google.com/");

// Give it some time to load the page and then get the document.
WScript.Sleep(3000);
var doc = ie.Document;

// Fill out search field.
var edit = doc.getElementsByName("q").item(0);
edit.value = "codecentrix";

// ... and press the submit button.
var submit = doc.getElementsByName("btnG").item(0);
submit.click();

Here is ie_auto.js file for download.
However there are problems with Internet Explorer automation:

• it may not work at all on Windows Vista unless the script is running at the same integrity level as iexplore.exe process. Simply clicking the js file won't do it. The script will run at medium integrity level and Internet Explorer has low integrity level and as result the script fails. If you run the script at high integrity level the newly started IE instance will have the same high integrity level and the script works (but this is not the best option from a security point of view). Changing the integrity level of the running script (or application) is not always the most desirable or easiest thing to do.

• no support to "connect" to already existing IE documents.

• sub-documents in different domains are not accessible for scripting due to cross scripting security issues (see my older posts: "When IHTMLWindow2::get_document returns E_ACCESSDENIED" and "When IHTMLWindow2.document throws UnauthorizedAccessException").

• difficult search of elements across all sub-documents inside frames/iframes (and sometimes impossible, see the point above).

• difficult and time consuming search of HTML elements on attributes other than id or name (getElementById and getElementsByName are the only methods I know that search elements directly wihtout browsing element collections which might be very slow when performed out of process).

• no tab support (only launch a new tab).

• no direct support for synchronizing input actions (clicks, keys) with the HTML document loading (it could be implemented by registering to IE events like document complete or looping while the browser becomes ready to accept inputs).

• no advanced search criteria like regular expression or searching on multiple attributes.

If you are interested in solving the issues above, let me introduce a project I've been working on for some time now. Here's Twebst, web automation library for Internet Explorer!

Get it FREE!

(to be continued)

focus vs fireEvent("onfocus")

While working on Twebst web automation library I encountered this problem: how to simulate setting the focus on HTML edit controls in Internet Explorer? There are two ways to do this.

1. Call IHTMLElement2::focus() method on target element that "causes the element to receive the focus and executes the code specified by the onfocus event".
2. Rise onfocus event on target element by calling IHTMLElement3::fireEvent() method.

The two approaches are quite similar but there are some interesting differences.

1. fireEvent("onfocus") does not actually set the focus on the element, it just executes the code of the onfocus handler event.
2. Calling focus method sets the focus on target element and call the onfocus event handler but not immediately. The onfocus event seems to be inserted in a queue and its handler is executed asynchronously after the current handler is finished.
3. If focus method is called from inside the onfocus handler nothing happens if the control already has the focus (that prevents an infinite recursion).

Example:


<html>
<script type="text/javascript" language="javascript">
function BtnFocusClick()
{
     document.getElementById('editTest').focus();
     window.status += "b";
}

function BtnOnFocusClick()
{
     document.getElementById('editTest').fireEvent('onfocus');
     window.status += "c";
}

function EditOnFocus()
{
     window.status += "a";
}
</script>

<body>
     <input type="text" onfocus="EditOnFocus()"; id="editTest"/><br/>
     <input type="button" value="focus" id="btnFocus" onclick="BtnFocusClick();"/>
     <input type="button" value="fire onfocus" id="btnOnFocus" onclick="BtnOnFocusClick();"/>
</body>
</html>

If pressing the button "fire onfocus" button the message in the Internet Explorer status bar is the expected one "ac". If pressing the "focus" button, the message is in reverse order than expected: "ba". That suggests that EditOnFocus handler is called after BtnFocusClick exit.

codeproject

IHTMLDocument3::getElementsByTagName and IHTMLElementCollection

A common task when writing Internet Explorer extensions is to browse a collection of objects based on a specified element tag-name. To work with collections IE provides IHTMLElementCollection interface that represents a collection of elements in an HTML document.

Usually a collection is retrieved by calling methods of IHTMLDocument2 interface. For some tag-names there specialized methods to retrieve collection (IHTMLDocument2::get_anchors , IHTMLDocument2::get_applets, IHTMLDocument2::get_forms, IHTMLDocument2::get_images, IHTMLDocument2::get_links, IHTMLDocument2::get_scripts).

To get all elements collection there is IHTMLDocument2::get_all.
One way to get a collection of elements having a specified tag-name is:

// CComQIPt<IHTMLDocument2> spDocument is a document object.

CComQIPtr<IHTMLElementCollection> spAllCollection;
HRESULT hRes = spDocument->get_all(&spAllCollection);
_ASSERTE(SUCCEDED(hRes) && (spAllCollection != NULL));

// Get the sub-collection of elements that have the "input" tag name.
CComVariant varTagName(CComBSTR("input"));
CComQIPtr<IDispatch> spDispCollection;
hRes = spAllCollection->tags(varTagName, &spDispCollection);

CComQIPtr<IHTMLElementCollection> spInputCollection = spDispCollection;
_ASSERTE(spInputCollection != NULL);

// Now you can browse spInputCollection using
// IHTMLElementCollection::item and IHTMLElementCollection::get_length methods.

The second method is:

// CComQIPt spDocument is a document object.
// Query for IHTMLDocument3 interface
CComQIPtr<IHTMLDocument3> spDoc3 = spDocument;
_ASSERTE(spDoc3 != NULL);

// Get the collection of elements that have the "input" tag name.
CComQIPtr<IHTMLElementCollection> spInputCollection;
HRESULT hRes = spDoc3->getElementsByTagName(CComBSTR("input"), &spInputCollection);
_ASSERTE(SUCCEDEED(hRes));

// Now you can browse spInputCollection using
// IHTMLElementCollection::item and IHTMLElementCollection::get_length methods.

ATL thunks and Windows DEP story

When using older ATL versions the program may generate an access violation due to Data Execution Prevention. This is basically a memory protection feature that prevents executing code from memory pages marked as non-executable.

The ATL implementation of CWindow class uses a technique called thunk. A thunk is a small piece of code that ATL generates in a region of memory allocated on the heap. Older versions of ATL did not set the execution flag on the allocated memory pages and that generates a crash on Windows machines where DEP is enabled.

Usually ATL is used in Internet Explorer extensions. If DEP is enabled then the result is a browser crash. This is a good reason to upgrade to Visual Studio 2005 or later. Find more on how to activate DEP in IE7 here.

Repairing Internet Explorer

Creating Internet Explorer extensions is not always an easy task. Erroneous extensions can make your IE to crash, become unstable or to behave unpredictably.

Even worse, uninstalling extensions can leave IE in a broken state, some things stop working properly. This usually happens when IE components are unregistered and IE registry keys are altered and not properly restored on uninstall. Here's a short list of damages I personally encountered:

1). 'Open in New Window' Command Does Not Work in Internet Explorer

2). When you try to add a Search Provider in Internet Explorer 7, a script error occurs, and the provider is not added successfully.
3). When you open a new tab in Internet Explorer 7, does the You've opened a new tab message appear every time, and are you unable to turn off this option
4). Open a link in a new tab takes forever to load.

I'll come later with examples of how an extension can break IE and how to avoid it. For now let's concentrate on how we can fix it. Download RepairIE.zip file, extract files inside and run "fixie.cmd"

Points of interest:

• Internet Explorer components are registered using regsvr32.exe tool.
• For IE7 mshtml.dll can not be registered as explained above; instead mshtml.tlb is registered using regtlib.exe tool.
• Registering shdocvw.dll using regsvr32.exe is not a good idea on IE7 because it damages [HKEY_CLASSES_ROOT\Typelib\{EAB22AC0-30C1-11CF-A7EB-0000C05BAE0B}\1.1\0\win32 registry key.
• reg_ieframe.reg file fixes the broken key above.
  

Open TextRange selection as URL

On many forums, people post URLs as plain text and this forces me to:

• select the text URL
• ctrl+C to copy it in the clipboard
• ctrl+T to open a new tab
• paste the URL in the address bar of the newly created tab
• press "Enter" key to start navigation

I found this whole process very annoying and it seems I have to go through this many times a day. Here's my attempt to automate it by creating an extension for "Internet Explorer" similar to Linkification extension (for "Fire Fox" browser). I called my extension: "Make Link".

How it works:

• right click on text URL
• choose "MakeLink: use clicked text as link" menu item (the URL is open in a new tab automatically)

or you can

• fully select the URL text or only a part of it
• right click on selected text
• choose "MakeLink: use clicked text as link" menu item (text link is open in a new tab automatically)

DOWNLOADS:

• MSI setup file
• EXE setup file
• extension source code

Points of interests:

• HKEY_CURRENT_USER\Software\Microsoft\Internet Explorer\MenuExt registry key to add a item in the IE context menu
• gain access to window object passed in external.menuArguments
• document.selection property to get the selected TextRange
• TextRange methods: moveStart, moveEnd, moveToPoint, select
• The extension is entirely written in JScript (no BHO, no COM).

Known issues:

• on some web pages with horizontal scroll simply right click a text URL won't work. You need to select a part of the URL and then right click it.

How to properly catch RBN_CHEVRONPUSHED notification?

This is actually a thread I started on MSDN forum but unfortunately it remained unanswered:

Virtually any IE toolbar needs a chevron to happily live along with other toolbars in the same re-bar. So does my toolbar. To implement chevron functionality in IE toolbars I need to handle RBN_CHEVRONPUSHED. According to MSDN, when the chevron button is pushed, the notification is sent by the rebar in the form of WM_NOTIFY message to its parent. Here is the windows hierarchy in IE7:

WorkerW <- ReBarWindow32 <- ToolbarWindow32

where the last toolbar window is my toolbar. So I need to catch notifications from ReBarWindow32 that are sent to WorkerW window. To do that the first idea that came to my mind was to subclass the WorkerW window. I don’t like this idea because:

• I subclass a window that does not belong to me, it was created by IE.
• I don’t know what is the best time to subclass it: on IObjectWithSite.SetSite or on IDockingWindow.ShowDW ? (Those functions are implemented by my toolbar component)
• I don’t know what is the best time to un-subclass it.
• I don’t know when other toolbar might subclass/un-subclass the same window (I actually got a conflict with other toolbar resulting in IE stack overflow crash because of the order of subclassing/unsubclassing).

My second approach uses RB_SETPARENT to modify the parent of ReBarWindow32 window to be one of my windows. I process the RBN_CHEVRONPUSHED notification for my chevron button and send the other notifications to the original parent window (that is WorkerW). I change the parent on toolbar initialization/un-init (IObjectWithSite.SetSite). It seems a safer approach but I’m still worried about other toolbars using the same technique and the possibility of conflicts.

Take care of standard IE "Links" toolbar that also sends WM_COMMAND, WM_DRAWITEM and WM_MEASUREITEM messages to WorkerW window (and now you'll get those messages too). On IE6, "Go" button also do the same.

So the question remains: what is the best way to catch RBN_CHEVRONPUSHED notification when creating an IE toolbar extension?

From IAccessible to IHTMLElement and back

When developing Internet Explorer plug-ins you might want to take advantage of the dual nature of the browser objects. Internet Explorer exposes the DOM (document object model) using IHTMLElement interface as the building block of the hierarchy. It also offers accessible objects through Active Accessibility IAccessible interface.

// From IAccessible to IHTMLElement.
CComQIPtr<IHTMLElement> AccessibleToHTMLElement(IAccessible* pAccessible)
{
     ATLASSERT(pAccessible != NULL);

     // Query for IServiceProvider interface.
     CComQIPtr<IServiceProvider> spServProvider = pAccessible;
     if (spServProvider != NULL)
     {
         // Ask the service for a IHTMLElement object.
         CComQIPtr<IHTMLElement> spHtmlElement;
         HRESULT hRes = spServProvider->QueryService(IID_IHTMLElement, IID_IHTMLElement,
                                                      (void**)&spHtmlElement);

         return spHtmlElement;
     }

     return CComQIPtr<IHTMLElement>();
}

// From IHTMLElement to IAccessible.
CComQIPtr<IAccessible> HTMLElementToAccessible(IHTMLElement* pHtmlElement)
{
     ATLASSERT(pHtmlElement != NULL);

     // Query for IServiceProvider interface.
     CComQIPtr<IServiceProvider> spServProvider = pHtmlElement;
     if (spServProvider != NULL)
     {
         // Ask the service for a IAccessible object.
         CComQIPtr<IAccessible> spAccessible;
         HRESULT hRes = spServProvider->QueryService(IID_IAccessible, IID_IAccessible,
                                                      (void**)&spAccessible);

         return spAccessible;
     }

     return CComQIPtr<IAccessible>();
}

Not all IHTMLElement objects support Active Accessibility. Here is the list of the HTML elements that are also accessible elements:
A, AREA, BUTTON, INPUT type=BUTTON, INPUT type=RESET, INPUT type=SUBMIT, FRAME, IMG, INPUT type=checkbox, INPUT type=image, INPUT type=password, INPUT type=radio, MARQUEE, OBJECT, APPLET, EMBED, SELECT, TABLE, TD, TH, TEXTAREA, INPUT type=TEXT.

This technique was successfully implemented and tested in My web automation library.
codeproject