Community discussion forum

Dealing with poor web protocols

• • Reply

  4 years ago

  by crc32

  crc32

  , United States

  Joined 4 years ago

  So, I'm trying to write a web spider for some data gathering at the US Patent Office (I work at a university - my group is studying the impact of software patents).  Anyway, in response to a POST, the PTO's webserver returns "200 Script Results Follow" instead of "201 CREATED".  This response kills my HttpWebRequest with a WebException - "System.Net.WebException: The server committed a protocol violation. Section=ResponseStatusLine".
  

  
  Does anyone know of a workaround for this?
  

  
  Thanks,
  Colin
  

  
  PS Code:
  

  Code:

  
  using System;
  using System.Collections.Generic;
  using System.Text;
  using System.Text.RegularExpressions;
  using System.Net;
  using System.IO;
  using System.Web;
  

  
  namespace ConsoleApplication1
  {
     public class InsertTest {
     public static void Main() {
  

  
         loHttp.Method = "POST";
         byte[] lbPostBuffer = System.Text.
                               Encoding.GetEncoding(1252).GetBytes(lcPostData);
         loHttp.ContentLength = lbPostBuffer.Length;
  

  
         Stream loPostData = loHttp.GetRequestStream();
         loPostData.Write(lbPostBuffer, 0, lbPostBuffer.Length);
         loPostData.Close();
         try
         {
             HttpWebResponse loWebResponse = (HttpWebResponse)loHttp.GetResponse();
             Encoding enc = System.Text.Encoding.GetEncoding(1252);
  

  
             StreamReader loResponseStream =
               new StreamReader(loWebResponse.GetResponseStream(), enc);
  

  
             string lcHtml = loResponseStream.ReadToEnd();
  

  
             loWebResponse.Close();
             loResponseStream.Close();
         }
         catch (WebException J)
         {
             Console.WriteLine("Protocol violation!\n {0}", J.ToString());
          }
       
         string adj22 = Console.ReadLine();
  

  
     }
     }  
     
  }
  

  Report abuse
• • Reply

  4 years ago

  by James Crowley

  James Crowley

  London, United States

  Joined 8 years ago

  You need to set an application configuration setting to tell the .NET framework to use "Unsafe" header parsing (ie ones that break the protocol). Place the following in your app.config file:
  

  
  <system.net>
  <settings>
  <httpWebRequest useUnsafeHeaderParsing = "true" />
  </settings>
  </system.net>
  

  
  (the file is named YourApplication.exe.config)

  Report abuse
• • Reply

  3 years ago

  by brianmrush

  Brian Rush

  , United States

  Joined 3 years ago

  James

  Hi.  Thanks for you comments.  I have read multiple articles identifying the useUnsafeHeaderParsing in either the app.config or machine.config.  I have done this but I am still see issues.  Here is my code.

  myWebReqest = CType(System.Net.WebRequest.Create("http://twister.sbs.ohio-state.edu/text/ncep/FOUS61.KWNO"), System.Net.HttpWebRequest)

  'myWebReqest = CType(System.Net.WebRequest.Create("http://www.yahoo.com"),   System.Net.HttpWebRequest)

  myWebReqest.ProtocolVersion = System.Net.HttpVersion.Version10

  aReader = myWebReqest.GetResponse()

  What is odd is this code works from development but not not from aspx page as anonymous user. The exception I get is:

  System.Net.WebException: The server committed a protocol violation. Section=ResponseStatusLine
  
  

  I am at my wits end.  I am almost ready to do the unthinkable, create a vb6 com component to get access to the data.  Any thoughts here.  Are their special rights the anonymous user may need.  What is odd is anonymous can run OK for sites like yahoo.  Only breaks on the weird URL above.

  Help!!  I am sure I have useUnsafeHeaderParsing = "true" set correctly

   

  Thanks in advance.

   

  Brian

  Report abuse
• • Reply

  3 years ago

  by brianmrush

  Brian Rush

  , United States

  Joined 3 years ago

  James

  Disregard my previous post.  I finally figured this out and thought I would highlight my stupidity and maybe save someone else some time if they see this issue. There was an obvious thing I was not checking. Recently I switched my internal IP and I had some rule sets set up in Norton to allow anonymous web traffic out.
  
  Norton AntiVirus was preventing traffic going out of from my new IP. As a result I was always getting the protocol exception.
  
  Jeez...Well at least this made me read a bunch of articles. Even implemented code to use XMLHTTP the old way from VB. I did eventually go back to HTTPWebRequest once I figured out it was Norton.
  
  
  
  
  
  

  Thanks

  
  Brian
  

  Report abuse