Web Data Extraction with C++ Web Macro

Web data extraction or web scraping can be implemented in various ways. Today I will use Twebst Web Automation Library to extract search results from Google using DOM parsing method and Internet Explorer automation (you need to install Twebst Library first).

Here are the steps that C++ web macro will perform in order to extract results from Google search:

• Open an Internet Explorer browser and navigate to Google site.
• Find the search edit box and fill out the word to search.
• Find the submit button and click it.
• Wait until the page is loaded and find a DIV with id=res
• Find the collection of all H3 elements inside the DIV element.
• Extract the text and URL and display it.

Enough talk! Let the code speak for itself.

// Start a new Internet Explorer instance and navigate to a given URL.
IBrowserPtr pBrowser = pCore->StartBrowser("http://www.google.com/");

// Find search edit box in page and type some text into it.
IElementPtr pSearchEdit = pBrowser->FindElement("input text", SearchCondition("name=q"));
pSearchEdit->InputText("codecentrix");

// Find search button and click it.
IElementPtr pSearchBtn = pBrowser->FindElement("input submit", SearchCondition("text=Google Search"));
pSearchBtn->Click();

// Find the DIV element where the result are displayed.
IElementPtr pResultDiv = pBrowser->FindElement("div", SearchCondition("id=res"));

// Get all found results and print them in console.
IElementListPtr pResultList = pResultDiv->FindAllElements("h3", SearchCondition());

// Display only the header result (text and url).
for (int i = 0; i < pResultList->length; ++i)
{
    // Get current H3 in the list.
    IElementPtr pCrntResult = pResultList->Getitem(i);

    // Find first and only anchor inside H3
    IElementPtr pCrntAnchor = pCrntResult->FindElement("a", SearchCondition());
    CComQIPtr<IHTMLAnchorElement> spCrntAnchor = pCrntAnchor->nativeElement;

    // Get URL from IHTMLAnchorElement.
    CComBSTR bstrURL = "";
    spCrntAnchor->get_href(&bstrURL);

    // Display results.
    wcout << pCrntResult->text << L"\n" << bstrURL.m_str << L"\n\n";
}

Download:

• CppMacro.zip (full source code and exe)
• Twebst Web Automation Library

What's wrong with Internet Explorer Automation?

The Microsoft Office products (Word, Excel, Power Point, Access, Outlook) allow their users to manipulate Office documents from Visual Basic or Visual Basic for Applications (VBA) code. It is possible to write a VBA macro in Excel that initializes a series of cells, and uses the cells to display a chart for instance.

Automation is the process of controlling one product from another product with the result that the client product can use the objects, methods, and properties of the server product. The client has access to the object model of the server.

Though Internet Explorer browser is not part of the Office suite, it supports automation. Here is a short sample:

// Create an IE automation object.
var ie = new ActiveXObject("InternetExplorer.Application");

// Make it visible and navigate to a given URL.
ie.Visible = true;
ie.Navigate("http://www.google.com/");

// Give it some time to load the page and then get the document.
WScript.Sleep(3000);
var doc = ie.Document;

// Fill out search field.
var edit = doc.getElementsByName("q").item(0);
edit.value = "codecentrix";

// ... and press the submit button.
var submit = doc.getElementsByName("btnG").item(0);
submit.click();

Here is ie_auto.js file for download.
However there are problems with Internet Explorer automation:

• it may not work at all on Windows Vista unless the script is running at the same integrity level as iexplore.exe process. Simply clicking the js file won't do it. The script will run at medium integrity level and Internet Explorer has low integrity level and as result the script fails. If you run the script at high integrity level the newly started IE instance will have the same high integrity level and the script works (but this is not the best option from a security point of view). Changing the integrity level of the running script (or application) is not always the most desirable or easiest thing to do.

• no support to "connect" to already existing IE documents.

• sub-documents in different domains are not accessible for scripting due to cross scripting security issues (see my older posts: "When IHTMLWindow2::get_document returns E_ACCESSDENIED" and "When IHTMLWindow2.document throws UnauthorizedAccessException").

• difficult search of elements across all sub-documents inside frames/iframes (and sometimes impossible, see the point above).

• difficult and time consuming search of HTML elements on attributes other than id or name (getElementById and getElementsByName are the only methods I know that search elements directly wihtout browsing element collections which might be very slow when performed out of process).

• no tab support (only launch a new tab).

• no direct support for synchronizing input actions (clicks, keys) with the HTML document loading (it could be implemented by registering to IE events like document complete or looping while the browser becomes ready to accept inputs).

• no advanced search criteria like regular expression or searching on multiple attributes.

If you are interested in solving the issues above, let me introduce a project I've been working on for some time now. Here's Twebst, web automation library for Internet Explorer!

Get it FREE!

(to be continued)

ATL + STL = CAdapt

I didn't know about CAdapt class until I tried to convert a VS2003 project to VS2005. Everything worked OK but I couldn't compile it! The compiler complained about:

std::list<CComQIPtr<IHTMLElement> >.

I found that what was accepted by VC++ 2003 compiler is not accepted by VC++ 2005 compiler. The reason of for is the address operator overloaded by CComQIPtr class. std::list class needs the address of a CComQIPtr object but & operator returns a IHTMLElement* address.

Here's where CAdapt class comes to save us. The list of smart pointers becomes: std::list<CAdapt<CComQIPtr<IHTMLElement> > >
Also you need to use m_T member where needed.

.Net and COM interop story

.Net allows programmers to reuse COM components in their managed code. To make this possible a managed wrapper object around the native object is needed. Besides that, one can use the COM object like any other managed object. Even if it sounds simple, you have to be aware of the differences between the CLR's object lifetime management and the COM version of object lifetime management.

COM programmers have to call Release on every interface that has been AddRef'ed. For C# programmers using COM objects that means AddRef is called when:
- a COM object is created.
- a COM object is returned by calling a method or a property.
- a COM object is cast'ed to another COM interface type.

To release a COM object in C# there are two options:
- leave the GC to collect managed wrappers and to call their finalizers that will call Release on native COM object.
- manually call Marshal.ReleaseComObject on every interface used in the code.

Let's see a short example using COM objects exposed by IE. The code bellow changes the color of every link in a HTML document.

// IHTMLDocument2 doc;
foreach (IHTMLElement elem in doc.all)
{
    IHTMLAnchorElement anchor = elem as IHTMLAnchorElement;
    if (anchor != null)
    {
        elem.style.color = "red";
    }
}

This first approach leaves the task of releasing COM objects to garbage collector. Let's manually release COM objects now:

// IHTMLDocument2 doc;
IHTMLElementCollection allCollection = doc.all;
foreach (IHTMLElement crntElem in allCollection)
{
   IHTMLAnchorElement anchor = crntElem as IHTMLAnchorElement;
   if (anchor != null)
   {
       IHTMLStyle style = crntElem.style;
       style.color = "red";

       Marshal.ReleaseComObject(style);
       Marshal.ReleaseComObject(anchor);
   }

   Marshal.ReleaseComObject(crntElem);
}

Marshal.ReleaseComObject(allCollection);

As you can see the number of code lines doubles! I personally prefer to leave the task of releasing COM objects to GC even if they will be eventually released after some time when GC comes into action.

Some might be tempted to call GC.Collect after a large chunk of code that work with COM objects but this could be even worse because other managed objects could be promoted to next GC generation and their lifespan is therefore longer than necessary.

In theory it is possible to create a lot of large COM objects that will exceed the native heap while the managed heap has a lot of available memory because managed wrappers are smaller in size. GC won't be called in this scenario so the native heap won't be freed.

If your application suffers from this kind of memory allocation problem, maybe using COM objects from managed code is not the best approach for you.

When IHTMLWindow2.document throws UnauthorizedAccessException

This is basically a C# translation of one of my older articles "When IHTMLWindow2::get_document returns E_ACCESSDENIED". Some .Net people encountered difficulties to use it, so I decided to make their life easier.

The main problem is the confusion created by System.IServiceProvider .Net interface because it has the same name as the COM interface. Once this issue is passed the code translation is straightforward. Here's the interop code to declare the COM interface IServiceProvider.

// This is the COM IServiceProvider interface, not System.IServiceProvider .Net interface!
[ComImport(), ComVisible(true), Guid("6D5140C1-7436-11CE-8034-00AA006009FA"),
InterfaceTypeAttribute(ComInterfaceType.InterfaceIsIUnknown)]
public interface IServiceProvider
{
   [return: MarshalAs(UnmanagedType.I4)][PreserveSig]
   int QueryService(ref Guid guidService, ref Guid riid, [MarshalAs(UnmanagedType.Interface)] out object ppvObject);
}

You find here full source code of the sample assembly.

This technique was successfully implemented and tested in Twebst web automation library.

When IHTMLWindow2::get_document returns E_ACCESSDENIED

Internet Explorer extensions usually needs to access HTML elements. When extensions are initialized they get a IWebBrowser2 pointer representing the browser. Starting with this pointer one can get any HTML element in the web page but to do that we need to browse a hierarchy of frames first. The simplest web pages only have one frame and one document. Web pages containing <frame> or <iframe> have a hierarchy of frames, each frame having its own document.

Here are the objects involved and the corresponding interfaces:

browser      - IWebBrowser2
frame/iframe - IHTMLWindow2
document     - IHTMLDocument2
element      - IHTMLElement

The list bellow shows what method to call to get one object from another:

browser      -> document        IWebBrowser2::get_Document
document     -> frame           IHTMLDocument2::get_parentWindow
frame        -> document        IHTMLWindow2::get_document
frame        -> parent frame    IHTMLWindow2::get_parent
frame        -> children frames IHTMLWindow2::get_frames

A normal call chain to get a HTML element is:

browser -> document -> frame -> child frame -> ... -> child frame -> document -> element

This will work almost all the time. The problems arise when different frames contain documents loaded from different internet domains. In this case IHTMLWindow2::get_document returns E_ACCESSDENIED when trying to get the document from the frame object. I think this happens to prevent cross frame scripting atacks.

Here is HtmlWindowToHtmlDocument function I wrote to be used instead IHTMLWindow2::get_document to bypass the restriction:




// Converts a IHTMLWindow2 object to a IHTMLDocument2. Returns NULL in case of failure.
// It takes into account accessing the DOM across frames loaded from different domains.
CComQIPtr<IHTMLDocument2> HtmlWindowToHtmlDocument(CComQIPtr<IHTMLWindow2> spWindow)
{
     ATLASSERT(spWindow != NULL);

     CComQIPtr<IHTMLDocument2> spDocument;
     HRESULT hRes = spWindow->get_document(&spDocument);
    
     if ((S_OK == hRes) && (spDocument != NULL))
     {
          // The html document was properly retrieved.
          return spDocument;
     }

     // hRes could be E_ACCESSDENIED that means a security restriction that
     // prevents scripting across frames that loads documents from different internet domains.
     CComQIPtr<IWebBrowser2>  spBrws = HtmlWindowToHtmlWebBrowser(spWindow);
     if (spBrws == NULL)
     {
          return CComQIPtr<IHTMLDocument2>();
     }

     // Get the document object from the IWebBrowser2 object.
     CComQIPtr<IDispatch> spDisp;
     hRes = spBrws->get_Document(&spDisp);
     spDocument = spDisp;

     return spDocument;
}


// Converts a IHTMLWindow2 object to a IWebBrowser2. Returns NULL in case of failure.
CComQIPtr<IWebBrowser2> HtmlWindowToHtmlWebBrowser(CComQIPtr<IHTMLWindow2> spWindow)
{
     ATLASSERT(spWindow != NULL);

     CComQIPtr<IServiceProvider>  spServiceProvider = spWindow;
     if (spServiceProvider == NULL)
     {
          return CComQIPtr<IWebBrowser2>();
     }

     CComQIPtr<IWebBrowser2> spWebBrws;
     HRESULT hRes = spServiceProvider->QueryService(IID_IWebBrowserApp, IID_IWebBrowser2, (void**)&spWebBrws);
     if (hRes != S_OK)
     {
          return CComQIPtr<IWebBrowser2>();
     }

     return spWebBrws;
}

Here is the C# version of the code: "When IHTMLWindow2.document throws UnauthorizedAccessException".

This technique was successfully implemented and tested in My web automation library.
codeproject

From IAccessible to IHTMLElement and back

When developing Internet Explorer plug-ins you might want to take advantage of the dual nature of the browser objects. Internet Explorer exposes the DOM (document object model) using IHTMLElement interface as the building block of the hierarchy. It also offers accessible objects through Active Accessibility IAccessible interface.

// From IAccessible to IHTMLElement.
CComQIPtr<IHTMLElement> AccessibleToHTMLElement(IAccessible* pAccessible)
{
     ATLASSERT(pAccessible != NULL);

     // Query for IServiceProvider interface.
     CComQIPtr<IServiceProvider> spServProvider = pAccessible;
     if (spServProvider != NULL)
     {
         // Ask the service for a IHTMLElement object.
         CComQIPtr<IHTMLElement> spHtmlElement;
         HRESULT hRes = spServProvider->QueryService(IID_IHTMLElement, IID_IHTMLElement,
                                                      (void**)&spHtmlElement);

         return spHtmlElement;
     }

     return CComQIPtr<IHTMLElement>();
}

// From IHTMLElement to IAccessible.
CComQIPtr<IAccessible> HTMLElementToAccessible(IHTMLElement* pHtmlElement)
{
     ATLASSERT(pHtmlElement != NULL);

     // Query for IServiceProvider interface.
     CComQIPtr<IServiceProvider> spServProvider = pHtmlElement;
     if (spServProvider != NULL)
     {
         // Ask the service for a IAccessible object.
         CComQIPtr<IAccessible> spAccessible;
         HRESULT hRes = spServProvider->QueryService(IID_IAccessible, IID_IAccessible,
                                                      (void**)&spAccessible);

         return spAccessible;
     }

     return CComQIPtr<IAccessible>();
}

Not all IHTMLElement objects support Active Accessibility. Here is the list of the HTML elements that are also accessible elements:
A, AREA, BUTTON, INPUT type=BUTTON, INPUT type=RESET, INPUT type=SUBMIT, FRAME, IMG, INPUT type=checkbox, INPUT type=image, INPUT type=password, INPUT type=radio, MARQUEE, OBJECT, APPLET, EMBED, SELECT, TABLE, TD, TH, TEXTAREA, INPUT type=TEXT.

This technique was successfully implemented and tested in My web automation library.
codeproject