Retrieving HTTP content in .NET

We can walk and chew gum at the same time!

Events are cool, but they're not all that useful if you're running in blocking mode as I've shown so far. HttpWebRequest/HttpWebResponse can also be run in asynchronous modes using the BeginGetResponse/EndGetResponse methods. Most of the stream classes provide this mechanism which allows you to specify a callback method to collect output retrieved from these requests (you can also send data asynchronously this way).

However, after playing with this for a while and then looking at the native thread support in the .NET framework it turned out to be easier to create a new thread of my own and encapsulate the thread operation in a class. The following example runs multiple wwHttp objects on a couple of threads simultaneously while also updating the form with information from the OnReceiveData event. Figure 1 shows what the form looks like. While the HTTP requests are retrieved the main form thread is still available to perform other tasks, so you can move around the form, so the form's UI remains active all the while.

This process is surprisingly simple in .NET partly because .NET makes it easy to route thread methods into classes. This make it both easy to create a thread's processing into a nicely encapsulated format as well as providing an easy packaging mechanism for passing data into a thread and keeping that data isolated from the rest of an application.

Make sure you add the System.Threading namespace to your forms that use threads. The following code defines the thread handler class that fires the HTTP request with the FireUrls() method (Listing 5).

Listing 6: Implementing a Thread class

public class GetUrls
  public string Url = "";
  public wwHttpMultiThread ParentForm = null;
  public int Instance = 1;

  public void FireUrls()
   wwHttp oHttp = new wwHttp();
   oHttp.OnReceiveData +=
             new wwHttp.OnReceiveDataHandler(this.OnReceiveData);
   string lcHTML = oHttp.GetUrlEvents(this.Url,4096);
   if (oHttp.Error)
     this.ParentForm.lblResult.Text = oHttp.ErrorMsg;

  public void OnReceiveData(object sender,
             wwHttp.OnReceiveDataEventArgs e)
   if (this.Instance == 1)
     this.ParentForm.lblResult.Text =
        e.CurrentByteCount.ToString() + " bytes";
     this.ParentForm.lblResult2.Text =
        e.CurrentByteCount.ToString() + " bytes";

There's nothing special about this class – in fact any class would do as thread handler (as long as you write thread safe code). This simplified implementation includes a reference back to the ParentForm that makes it possible to access the status labels on the form. The Instance property is used to identify which request is updating which form control. The actual code here is very much like code I've previously shown using the wwHttp object. Note that this code assigns the event handler a method of the thread action class. This method then calls back to the ParentForm and updates the labels.

The calling code on the form creates two threads that call the FireUrls method as follows (Listing 6):

Listing 7: Creating and running the Thread

private void cmdGo_Click(object sender, System.EventArgs e)
  GetUrls oGetUrls = new GetUrls();
  oGetUrls.Url = this.txtUrl.Text;
  oGetUrls.ParentForm = this;
  ThreadStart oDelegate =
        new ThreadStart(oGetUrls.FireUrls);
  Thread myThread = new Thread(oDelegate);

  GetUrls oGetUrls2 = new GetUrls();
  oGetUrls2.ParentForm = this;
  oGetUrls2.Url = this.txtUrl2.Text;
  oGetUrls2.Instance = 2;
  ThreadStart oDelegate2 =
       new ThreadStart(oGetUrls2.FireUrls);
  Thread myThread2 = new Thread(oDelegate2);

To start a thread the ThreadStart() function is called which takes a function pointer (basically a reference that points at a specific method in a class) as an argument. This returns a Delegate that can be used to create a new thread and tell it to start running with this pointer. You can pass either an instance variable of a static address of static class method. In most cases you'll want to use a method of a dynamic instance object variable, because it gives you the ability to fully set up the instance by setting properties that you'll need as part of the processing. Think of the actual thread implementation class as wrapper that is used as the high level calling mechanism and parameter packager to your actual processing code. If you need to pass data back to some other object you can make this instance a member of another object. For example, I could have made this object part of the form which would then allow the form to access the members of the 'thread' class and both could share the data.

Creating threads and running them is very easy but make sure you manage your data cleanly to prevent access of shared data from different threads at the same time. Shared data needs to be protected with synchronization of some sort. In fact, you can see why this is an issue if you click the Go link on the sample form a few times while downloads are still running. You'll see the numbers jump back and forth as multiple threads update the same fields on the form. As multiple instances are writing to the labels at the same time the code will eventually blow up. The workaround for this is to use synchronized methods to handle the updates or to use separate forms to display the update information (new form for each request). The topic of synchronization is beyond the scope of this article and I’ll cover basic multi-threading concepts in a future article.

The Web at your fingertips

This cliché has been so overused, but I can say .NET really delivers on this promise in a number of ways. Yes you could do all of this and most other Internet related things before, but .NET brings a consistent model to the tools. In addition you get straight forward support for advanced features like easy to implement multi-threading and (albeit somewhat complex) event support that make it easy to created complex applications that utilize the tools in ways that were either not possible in, or took a lot more work previously.

The tools are all there to access the Web very easily whether it's through the high level tools in .NET or the more lower level network protocol classes. The HttpWebRequest class is a fairly easy and powerful implementation that provides an excellent balance between flexibility and ease of use. For me the wwHttp class has been an exercise in understanding and utilizing the HttpWebRequest/Response classes. I hope you find the class useful if not as is then as an example of a variety of things that need to be done with HTTP requests frequently. The class is not complete and some features will need expansion in the future, but it's good starting point.

You might also like...



Why not write for us? Or you could submit an event or a user group in your area. Alternatively just tell us what you think!

Our tools

We've got automatic conversion tools to convert C# to VB.NET, VB.NET to C#. Also you can compress javascript and compress css and generate sql connection strings.

“A computer is a stupid machine with the ability to do incredibly smart things, while computer programmers are smart people with the ability to do incredibly stupid things. They are, in short, a perfect match” - Bill Bryson