Library tutorials & articles

Create a Site Search Engine in ASP.NET

By Stevan Rodrigues, published on 07 Feb 2004
Page 8 of 8
  1. Introduction
  2. SiteSearch.aspx
  3. Page.vb
  4. UserSearch.vb
  5. FileContent.vb
  6. PagesDataset.vb and CleanHtml.vb
  7. Site.vb and Web.config
  8. How to integrateand Enhancing the code

How to integrateand Enhancing the code

How to integrate

The application has been tested with the web form SiteSearch.aspx in the root directory. So my suggestion is that you do the same. Later on, you can try moving it to any subfolder. All my classes I have placed in components folder. You can move them to any folder of your choice.

Errors

When the application is place in the root you may get the following errors. The remote server returned an error: (401) Unauthorized.
OR
The remote server returned an error: (500) Internal Server Error.

This error is caused because
1)If server returns (401) Unauthorized, the application is unable to read the file beacuse of right access.
2)If server returns (500) Internal Server Error, the page that it was trying to read returned an error. The page that application was trying to read either has an error or requires parameters because of which it returns error

Follow the steps to rectify the error
1)In the Web.config ensure that the BarredFolders list is comprehensive
aspnet_client,_private,_vti_cnf,_vti_log,_vti_pvt,_vti_script,_vti_txt,cgi_bin,_bin,bin,_notes,images,scripts
2)Ensure that the BarredFiles list is comprehensive and contains localstart.asp,iisstart.asp

Globalization

The Search Engine module can be easily globalize. For this purpose we will see how to convert it into Chinese language.

Web.config

The XML declaration must appear as the first line of the document without other content, including white space, in front of the start <.

The XML declaration in the document map consists of the following:<?xml version="1.0" encoding="Your Encoding" ?>. By default the visual studio uses the utf-8 encoding this needs to be changed to encoding that you want to use. Here we will change to gb2312. Hence the XML Declaration needs to be modified as follows.

English

<?xml version="1.0" encoding="utf-8" ?>

Chinese

<?xml version="1.0" encoding="gb2312" ?>

The requestEncoding and responseEncoding specifies the assumed encoding of each incoming request and outgoing response. The default encoding is UTF-8, specified in the <globalization> tag included in the Machine.config file created when the .NET Framework is installed. If encoding is not specified in a Machine.config or Web.config file, encoding defaults to the computer's Regional Options locale setting. We will need to change requestEncoding and responseEncoding to reflect the change in encoding.

English

<globalization requestEncoding="utf-8" responseEncoding="utf-8" />

Chinese

<globalization requestEncoding="gb2312" responseEncoding="gb2312" />

In order to avoid building the code when the encoding changes we need to add the encoding key to the appsettings.

<!-- Set this to the Encoding of the web site-->     
<add key="Encoding" value="gb2312" />

Also change the English Language key to false.

 <!-- Set this boolean to False if you are not using 
  an English language web site--> 

  <add key="EnglishLanguage" value="True" />

SiteSearch.aspx

Last but not the least, the codepage attribute has to be added in the page directive.

English

<%@ Page Language="vb" Trace="False" AutoEventWireup="false" Codebehind="SiteSearch.aspx.vb" Inherits="SearchDotnet.SiteSearch" debug="false" %>

Chinese

<%@ Page Language="vb" Trace="False" AutoEventWireup="false" Codebehind="SiteSearch.aspx.vb" Inherits="SearchDotnet.SiteSearch" debug="false" codePage="936" %>

Enhancing the code

The application is meant for small sites. For bigger sites, the code can be further enhanced. Infact you will need to write to database say XML file periodically and then read from it. I will offer a few tips to do so.

1)In my code I search and filter the data using regular expression. Instead of this you will have to write entire data (not filtered data) to XML file using the following method.

      Private Shared Sub WriteXmlToFile(ByVal thisDataSet As DataSet)
            If thisDataSet Is Nothing Then
                Return
            End If
            
            thisDataSet.WriteXml(XMLFile)
        End Sub

2)Later you will need to read the xml from file save it to the shared dataset say Searchs.Site.PageDataset.Tables("Pages").

        Private Shared Function ReadXmlFromFile() As DataSet
            ' Create a new DataSet.
            Dim newDataSet As New DataSet("New DataSet")

            ' Read the XML document back in. 
            ' Create new FileStream to read schema with.
            Dim fsReadXml As New System.IO.FileStream(XMLFile, System.IO.FileMode.Open)

            ' Create an XmlTextReader to read the file.
            Dim myXmlReader As New System.Xml.XmlTextReader(fsReadXml)

            ' Read the XML document into the DataSet.
            newDataSet.ReadXml(myXmlReader)

            ' Close the XmlTextReader
            myXmlReader.Close()
            Return newDataSet

        End Function

3)For each search you will later need use the Select method of PageDataset.Tables to filter it according to search results. Filter dataset might look like this. FillDataset method contains logic to create and add search results (array of DataRow) to database. 

       Private Sub FiterPagesDatset()
            Dim strExpr As String
            Dim foundRows As DataRow()

            Dim Field() As String = {"Title", "Description", "Keywords", "Contents"}

            strExpr = SomeFunction 'Your function to build the query.

          
            foundRows = Searchs.Site.PageDataset.Tables("Pages").Select(strExpr)
            FillDataset(foundRows)
        End Sub
4)The filtered result store it into another dataset and use it to display results

Points of Interest

When I was working on the project, the question was how to display the results. DataGrid was my choice as we can exploit a lot of its features which are not present in other list controls. Once my question was solved, the next was how to pass content to the DataGrid. DataSet was the only alternative. As I worked further before storing the information in DataSet, I had to move to and fro with bulk information about the page. I decided to use site object to store the information.

One of the authors suggested the following best practices:

  1. The class should be small to have approximately half a dozen methods and properties.
  2. Methods be short with again approximately half a dozen lines.

After carefully analyzing the code and keeping the best practices in mind, I redesigned the code to what it is now.

Histoy

Modified the code to read dynamic pages.

Enhanced the code to support globalization.

Comments

  1. John Loser
    28 May 2008 at 04:09

    hello all

    i am trying hard to integrate the search engine code but have failed, my site is on local lan server with the root folder as a virtual folder , my site is in c# , i have placed the contents in the folder named searchdotnetc , but the code dosnt works, i modified the code made the class partial for the cs file and also tried placing the contents folder in the app code , but i still parsers only static pages and that to only in the searchdotnetc folder for the dynamic pages it gives  the error

     


    Line 273:			wcMicrosoft = new System.Net.WebClient();
Line 274:			objUTF8Encoding = new UTF8Encoding();
Line 275:			srchFile.Contents = objUTF8Encoding.GetString(
Line 276:				wcMicrosoft.DownloadData(String.Format("{0}/{1}", Searchs.Site.ApplicationPath, srchFile.Path)));
Line 277:		}

    Source File: v:\SITENAMEwww\App_Code\components\UserSearch.cs    Line: 275

    Stack Trace: 

    [WebException: The remote server returned an error: (404) Not Found.]
   System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request) +338
   System.Net.WebClient.DownloadData(Uri address) +181
   System.Net.WebClient.DownloadData(String address) +26
   Searchs.UserSearch.GetDynamicFileContent(Page srchFile) in v:\SITENAME\www\App_Code\components\UserSearch.cs:275

    please help .....

     

     

  2. shaka
    25 Jun 2007 at 19:01
    Do you have a version of the Search Engine project, for ASP.NET 2.0, that works well with Master Pages?
  3. sree2000
    02 Jun 2006 at 17:53

    Hello,

         I have down loaded the SearchDotNet.zip file from the code project. I am using VS2005, and asp.net 2.0  the files are not getting converted successfully. Can you provide me a solution to integrate this search facility into my web application .

  4. Deepthi P
    31 Jan 2006 at 07:25
    when i used ur solution to implement search engine for my website it is  searching only words .
    for example if hai is given it is finding some files having "hai" but if h is given it is showing " 0 found"
    please suggest me one solution
  5. Cassius Arulkumaran
    18 Jan 2006 at 21:47

    you can use text copy to copy images and text objects inside of SQL server
    http://www.mssqlcity.com/Articles/KnowHow/Textcopy.htm

  6. kody
    03 Nov 2005 at 23:07
    Did you find it ? Because I am also searching for something like that.
  7. trobin7
    04 Oct 2005 at 15:37

    It seems I have this error as well.  Do I need to give write access to the root directory or access to another file or something else?


    As a test, I've given the "Everyone" full access to the root, but doesn't help


    Thanks ..Troy

  8. Stevan Rodrigues
    13 Sep 2005 at 08:53

    The error is related to the permissions. You will need to check the permissions and ensure that the application has permissions expecially  System.Net.WebClient

  9. Najeem M Illyas
    11 Sep 2005 at 13:58

    Hi I'm also facing a similar problem with the search program. I'm getting error like 404. Anybody tell me how to fix the problem, I've configured the web.config file correctly.

  10. oramike
    18 May 2005 at 15:10

    hi, i have an SQL Server database with documents from applications (ms word, excel, etc). i now need to retrieve these documents in their format but don't know how to go about it.


    can anyone lend a hand, please!

  11. oramike
    15 Apr 2005 at 15:30
    Hi James, thanks for the help. I finally got it.

  12. James Crowley
    14 Apr 2005 at 23:17
    In IIS, you either need to make the virtual directory an "application". To do this, right click on the virtual directory (in IIS admin), click properties, and click the "Create" button next to "application name".

    If you don't have an application configured, then there's a set of configuration options in web.config that aren't supported - hence it saying "It is an error to use a section registered as allowDefinition='MachineToApplication' beyond application level.". If you remove the <authentication> tag entirely, then this should work (or possibly it will complain about another unsupported tag )
  13. oramike
    14 Apr 2005 at 19:44
    When i tried to run the code, i get this error message:

    Configuration Error.

    Description: An error occurred during the processing of a configuration file required to service this request. Please review the specific error details below and modify your configuration file appropriately.

    Parser Error Message: It is an error to use a section registered as allowDefinition='MachineToApplication' beyond application level. This error can be caused by a virtual directory not being configured as an application in IIS.

    Source Error:


    Line 34:           "Forms", "Passport" and "None"
    Line 35:     -->
    Line 36:     <authentication mode="Windows" />
    Line 37:
    Line 38:


    Source File: c:\inetpub\wwwroot\SearchDotnet\web.config    Line: 36.


    I changed the authentication mode to "None", but i still get the same error message.

    Is there anything i should do that i have not yet done?


  14. pax
    04 Feb 2005 at 01:23

    i'm trying to search in an asp file but it doesn't work, i'm not getting the error 500, it's solved but i still not searching on the asp file, it appears like there's no match on that file but there's a match. what can i do?

  15. tiglathpileser
    09 Dec 2004 at 10:06

    we make a search engine that we are in the process of porting to .net  Get in touch with me if you are interested in doing some beta testing.  Office document indexing is quite tricky with a lack of documentation available and pdf documents store information in a variety of strange ways.  contact me: paddy phdcc.com

  16. Hugmup
    07 Dec 2004 at 18:39

    I am not a Visual Studio developer, but I have devleped about 8 ASPX projects with a notepad called ScITE for my web site and I've compiled my DLLs with the command-line compiler.


    I downloaded the source files. I put SearchSite.aspx in the wwwroot\website directory on my local IIS server, I put the DLL in wwwroot\website\bin, and I put the web.config file in wwwroot\website. Then I configured web.config according to the instructions, modified the HTML in the ASPX file so that it matches my web site, and changed the name of SiteSearch.aspx to zsearch.aspx to match my file naming scheme.


    It works beautifully on my local IIS server, but it does not work on the production server. The error is Service Unavailable.


    This is not useful.

  17. Stevan Rodrigues
    03 Dec 2004 at 21:31

    My code searches for files that store content as text or removes text from html. It is not made to search for pdf and word documents.

  18. jackthecat64
    03 Dec 2004 at 13:44

    Hi. I read your query to Stevan about dealing with .pdf files in his search engine (searchdotnet).  I don't think he replied to you, right?


    Did you solve this problem?  I'm am having the same issue.


    Thanks in advance for any help you can offer.


    Beth

  19. musa7290
    28 Sep 2004 at 19:31

    I would highly appreciate if someone points me some sample code for simple search engine\crawler\robot that can be used\extended to gather news headlines from news websites.


    Basically I am highly interested to implement a small live new page like news.google.com
    Thanks in Advance
    Musa


  20. fpga
    17 Aug 2004 at 08:19

    Hi,
    I would integrate your solution in my web...but , something I don't understund...
    you explain that we have to "2) Place the SiteSearch.aspx and web.config  in the root  of the application"


    but , I allready have a web.config , for my application.
    ...
    can you give me more information about it ?


    thanks

  21. Bob Reichard
    16 Aug 2004 at 08:30

    I have your Sitesearch code working fine on my development system and development server. However, on my live server I receive the following error. But if I log into the server through terminal and acces the site in a browser from that box it works fine. Any idea why this is happening? Permissions all seem to be fine.


    The underlying connection was closed: Unable to connect to the remote server.
    An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.
    [WebException: The underlying connection was closed: Unable to connect to the remote server.]
      System.Net.HttpWebRequest.CheckFinalStatus() +673
      System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult) +140
      System.Net.HttpWebRequest.GetResponse() +149
      System.Net.WebClient.DownloadData(String address) +151
      clearweld.Searchs.UserSearch.GetDynamicFileContent(Page srchFile)
      clearweld.Searchs.UserSearch.GetFileInfo(String FPath, Page srchFile)
      clearweld.Searchs.UserSearch.GetInfo(String FPath)
      clearweld.Searchs.UserSearch.ProcessFile(String FPath)
      clearweld.Searchs.UserSearch.ProcessDirectory(String targetDirectory)
      clearweld.Searchs.UserSearch.Search(String targetDirectory)
      clearweld.SiteSearch.SearchSite(String strSearch)
      clearweld.SiteSearch.Page_Load(Object sender, EventArgs e)
      System.Web.UI.Control.OnLoad(EventArgs e) +67
      System.Web.UI.Control.LoadRecursive() +35
      System.Web.UI.Page.ProcessRequestMain() +731

  22. Ilkin E Jamalli
    27 Jul 2004 at 23:17
    I have installed on c:\wwwroot\search  folder.
    It works fine but it search only on c:\wwwroot\search\sites  folder. How can I make it to search in c:\wwwroot\
    Can you please suggest ?
  23. Stevan Rodrigues
    23 Jun 2004 at 22:15

    Hi,
    I have sent the updates but the admin here is too too busy to do changes. So my latest code can be found at
    http://www.codeproject.com/aspnet/SearchDotnet.asp. I have uploaded the c# version there. The C# version is older version If you want to use VB.net code and since you have webmatrix then


    1)You can just use the dll of vb.net search engine and place it in the bin of your project


    2) Use the aspx pages in your c sharp project


    3)Convert the code behind vb pages to c sharp



    Steve

  24. Sri Koerniwati
    23 Jun 2004 at 10:06

    Hi Steve,


    I am beginner asp.net programmer. I am developing website using webmatrix, I use C#. I create a folder for all my pages then I created a virtual directory in IIS. I download your search engine vb code, I copy searchdotnet folder to my website folder then I run SiteSearch.aspx. The page was loading and when I tried to search words that are in my pages, I got this message: 0 Files found. I tried to put one of my pages in Sites folder under SearchDotNet Folder and tried again to search word on my pages and I still got the same message: 0 Files found.


    Could you help me with this ? Where should I put my pages so when I run the SiteSearch.aspx, I can find words that are on my pages.... how to convert VB.net code to C#?


    Thank you so much for helping me....

  25. Stevan Rodrigues
    21 Jun 2004 at 00:10

    Hi,
    1)You can just use the dll of vb.net search engine and place it in the bin of your project


    2) Use the aspx pages in your c sharp project


    3)Convert the code behind vb pages to c sharp


    4)My latest code can be found at
    http://www.codeproject.com/aspnet/SearchDotnet.asp


    Steve

  26. Emily Grettel
    20 Jun 2004 at 23:18
    Hi Steve!

    Just wondering, if you knew how much of a problem it would be if we integrated your VB.NET search tool in our C# web application for our intranet... theres roughly 400 pages (dynamic and static) but i'm not sure whether things will cause problems because ours is written in C# and the search engine is VB.NET.

    Also, we would be having it in the root (Eg. /search.aspx) so would this be a problem? or do we need to manually convert all to C# before we try and integrate it?

    Thanks and congratulations on a FANTASTIC and truely excellent piece of work:-)

  27. Emily Grettel
    20 Jun 2004 at 23:18
    Hi Steve,

    Just to let you know I posted a msg about your script here:
    http://www.developerfusion.com/forums/topic.aspx?id=21918
  28. Alan
    09 Jun 2004 at 01:43

    Great code stevan....!!


    Was just wondering if you had any ideas about working with PDF's and word documents


    I am using the XML version to search an entire intranet. The site contains a large number of pdfs and documents that have embedded graphics.


    The search functionality works perfectly. However, the xml file contains code that represents the graphic part of these documents. Therefore when a search is completed, the content field returns meaingless information:



    Search on Marketing returns:


    Title: Marketing
    Content: PDF 1 3 6 0 obj endobj xref 6 36 0000000016 00000 n 0000001064 00000 n 0000001368 00000 n 0000001572 00000 n 0000001730 00000 n 0000002139 00000 n 000000266    



    Any ideas how this could be filtered out...???

  29. rogeriogr
    14 May 2004 at 10:48
    Hi folks,

    I am trying the SiteSearch script in my personal web page and it is working well except by the fact that the portuguese accents are not being showed on the search results.

    For example, when showing the results ú appears as uacute; ç appears as ccedil.

    Thanks in advance.
  30. Stevan Rodrigues
    07 May 2004 at 22:33

    The Latest Code and images can be found at http://www.codeproject.com/aspnet/SearchDotnet.asp.

  31. Stepan Kazakov
    22 Apr 2004 at 10:22

    I got your updated version and when i try to build it i encouter this problem. Any idea why?




    Thanks

  32. Stevan Rodrigues
    08 Apr 2004 at 04:37

    Hi,
    I have updated my article n sent to the admin but there has been a delay.Just try this
    http://www.freewebs.com/stevanrodrigues/searchdotnet.html
    Regards,
    Stevan

  33. Manoj Kumar
    29 Mar 2004 at 07:37

    Dear Sir,


    This is manoj and using your site search engine
    and facing so many errors while using it.
    can you give me the process from A-Z how to use it?


    Thanks & Regards
    manoj(mnj_secret@yahoo.com)

  34. asplover
    20 Mar 2004 at 12:44
    hello,firend:
      i come form china.i appreciate your site search engine very well. but i encount a problem that it can not search chinese word. please tell me how to solve this problem.
                                            thank u very much!
                                                                                             asplover
                                                                                               3.20
  35. Simone75
    09 Mar 2004 at 03:38

    Thank you Stevan,
    Now all work correctly.
    I use your dll and it works fine.
    I've a c# web project so i can't integrate your vb.net project with mine using your guidelines.
    In a later version of your project think about configuration of the root in wich begin the search; i think it is helpful.
    Thank you very much.
    Bye.

  36. Stevan Rodrigues
    09 Mar 2004 at 01:13

    Well just go to the http://www.freewebs.com/stevanrodrigues/SearchDotnet.html where you will get the latest update. Cick on the third link ie..
    Download latest version of demo project (Visual studio.net not required)
    1) Place the SearchDotnet.dll in the bin folder in the root of the application eg: http:\localhost\abitaria.
    2) Place the SiteSearch.aspx and web.config in the root  of the application eg: http:\localhost\abitaria.


    If you still want to integrate the code and not just dll in your page then
    1)Since you already have a project in visual studio .net open the old project(vb.net web application) which is pointing to the root of the application eg: http:\localhost\abitaria.
    2)Place the sitesearch.aspx in the directory "abitaria".
    3)Place the components folder  in root "abitaria" or any folder within "abitaria"
    4)Build the code.



    Regards,


    Stevan Rodrigues

  37. Simone75
    08 Mar 2004 at 04:43

    I'm sorry but i've problems yet.
    1)I'm not able to open In visual studio .net a new project(vb.net web application) pointing to the root of the application eg: http:\localhost\abitaria because visual studio .net tell me that already exists a project pointing in that root.
    2)I tried, also, to  ONLY copy sitesearch.aspx and Components directory in root Abitaria but when i try to load the aspx page this error is shown:
    Description: An error occurred during the parsing of a resource required to service this request. Please review the following specific parse error details and modify your source file appropriately.


    Parser Error Message: Could not load type 'SearchDotnet.SiteSearch'.


    Source Error:



    Line 1:  <%@ Page Language="vb" Trace="False" AutoEventWireup="false" Codebehind="SiteSearch.aspx.vb" Inherits="SearchDotnet.SiteSearch" debug="false" %>
    Line 2:  <HTML>
    Line 3:      <HEAD>



    Source File: C:\Inetpub\wwwroot\Abitaria\sitesearch.aspx    Line: 1


    So i think i must change the code in sitesearch.aspx because it inherits from your SearchDotnet project


    I like your SearchDotnet application and i want to use it in my web sites but i need more help.
    Could you help me again?
    Thank you.
    Simone

  38. Stevan Rodrigues
    07 Mar 2004 at 00:46
    Hi,

    Do you have visual studio .net? Do you have the .Net framework?

    1)In visual studio .net open a new project(vb.net web application) pointing to the root of the application eg: http:\\localhost\abitaria.
    2)Place the sitesearch.aspx in the directory "abitaria".
    3)Place the components folder  in root "abitaria" or any folder within "abitaria"
    4)Build the code.

    There is a work around do all this without visual studio .net. I have a solution without visual studio send me an email and I will let you know.

    Regards,

    Stevan Rodrigues
  39. Simone75
    05 Mar 2004 at 10:46
    Hello,
    i want to knows how i can integrate your site search web application in my web application.
    I have a virtual directory called "abitaria", you use your virtual directory called "searchdotnet".
    I want sitesearch.aspx in my virtual directory and i want the search in my root "http:\\localhost\abitaria" not in "http:\\localhost\searchdotnet\site".
    Can you help me???
    thank you
    email me
  40. Stevan Rodrigues
    29 Feb 2004 at 05:53

    I have sent the latest update to admin till then you can try the code at
    http://www.codeproject.com/aspnet/SearchDotnet.asp
    Also you can get in touch with me by email.

  41. Stevan Rodrigues
    26 Feb 2004 at 00:36

    Well just run a search for 'error' you will get the page listed there. This is because your page may require some parameter(querystring or form) or cannot be read directly. Try calling from the web using that path in address of your browser.

  42. Dennis St. John
    26 Feb 2004 at 00:20

    I have been successful in running this app in a virtual directory in IIS (i.e. /SearchDotNet), however, what if I want to search the files in the root directory as well?  I copied the app to the root and can't seem to get it to work as it does within a virtual directory underneath of the root.  Any help or suggestions would be greatly appreciated.


    Thanks!


    Dennis - PC1, Inc.

  43. Joshua Stengel
    25 Feb 2004 at 13:25

    I made a modification as show in another post
    and that got rid of the error but it still doesn't
    search the dynamic content.  I am investigating...


    See post for 500 and 401 errors:
    http://www.developerfusion.com/forums/topic-19327

  44. Joshua Stengel
    25 Feb 2004 at 12:16

    When searching dynamic .aspx pages I get a 404 error.
    Any thoughts?


    I have the following lines in my web.config file and the
    engine runs fine in directories that do not contain
    dynamic content.


    web.config lines:


            <add key="FilesTypesToSearch" value=".aspx" />
            <add key="DynamicFilesTypesToSearch" value=".aspx" />


    Error:


    [WebException: The remote server returned an error: (404) Not Found.]
      System.Net.HttpWebRequest.CheckFinalStatus() +674
      System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult) +139
      System.Net.HttpWebRequest.GetResponse() +149
      System.Net.WebClient.DownloadData(String address) +151
      SearchDotnet.Searchs.UserSearch.GetDynamicFileContent(Page srchFile) +154
      SearchDotnet.Searchs.UserSearch.GetFileInfo(String FPath, Page srchFile) +345
      SearchDotnet.Searchs.UserSearch.GetInfo(String FPath) +212
      SearchDotnet.Searchs.UserSearch.ProcessFile(String FPath) +13
      SearchDotnet.Searchs.UserSearch.ProcessDirectory(String targetDirectory) +71
      SearchDotnet.Searchs.UserSearch.Search(String targetDirectory) +174
      SearchDotnet.SiteSearch.SearchSite(String strSearch) +173
      SearchDotnet.SiteSearch.Page_Load(Object sender, EventArgs e) +229
      System.Web.UI.Control.OnLoad(EventArgs e) +67
      System.Web.UI.Control.LoadRecursive() +35
      System.Web.UI.Page.ProcessRequestMain() +731

  45. Stevan Rodrigues
    25 Feb 2004 at 01:08

    Thanks James for the support and co operation.


    I had send the update of my search engine. I know you are busy. So if you want me to format it the way you want I will be pleased to do it. Also if I could be of any help let me know.

  46. stix
    24 Feb 2004 at 12:12

    Silly me. FOrgot that one. Works fine now. thanks.

  47. James Crowley
    24 Feb 2004 at 11:52

    Have you got


    <add key="EnglishLanguage" value="True" />


    or False in your web.config file?

  48. stix
    24 Feb 2004 at 11:39

    I get this error when trying to run the sample code. Any ideas?


    Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.


    Exception Details: System.FormatException: Input string was not in a correct format.


    Source Error:


    An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.  


    Stack Trace:



    [FormatException: Input string was not in a correct format.]
      Microsoft.VisualBasic.CompilerServices.DoubleType.Parse(String Value, NumberFormatInfo NumberFormat) +195
      Microsoft.VisualBasic.CompilerServices.BooleanType.FromString(String Value) +219


    [InvalidCastException: Cast from string "" to type 'Boolean' is not valid.]
      Microsoft.VisualBasic.CompilerServices.BooleanType.FromString(String Value) +339
      SearchDotnet.Searchs.UserSearch.Search(String targetDirectory) +34
      SearchDotnet.SiteSearch.SearchSite(String strSearch) +174
      SearchDotnet.SiteSearch.Page_Load(Object sender, EventArgs e) +230
      System.Web.UI.Control.OnLoad(EventArgs e) +67
      System.Web.UI.Control.LoadRecursive() +35
      System.Web.UI.Page.ProcessRequestMain() +725



  49. Stevan Rodrigues
    24 Feb 2004 at 00:58
    Hi,
    The problem is quite basic.
    When you place the folder in wwwroot you will need to make that directory ie searchdotnet directory as virtual directory.
  50. srinath_sankar
    23 Feb 2004 at 11:47
    this is the error that i am getting when i run the same

    Description: An error occurred during the processing of a configuration file required to service this request. Please review the specific error details below and modify your configuration file appropriately.

    Parser Error Message: It is an error to use a section registered as allowDefinition='MachineToApplication' beyond application level. This error can be caused by a virtual directory not being configured as an application in IIS.

    Source Error:


    Line 34:           "Forms", "Passport" and "None"
    Line 35:     -->
    Line 36:     <authentication mode="Windows" />
    Line 37:
    Line 38:


    Source File: c:\inetpub\wwwroot\msg10\searchdotnet\web.config    Line: 36


    --------------------------------------------------------------------------------
    Version Information: Microsoft .NET Framework Version:1.1.4322.573; ASP.NET Version:1.1.4322.573
  51. srinath_sankar
    23 Feb 2004 at 11:40
    Hi I am having problem downloading and running the code any help....I have the searchdotnet directory in my wwwroot andwhen i try to run it it gives me an error saying that the virtual directory missing...and also wanted to know if i can use the same code in my website or do i have to meke changes

    thanks
    srinath
  52. Stevan Rodrigues
    18 Feb 2004 at 07:43

    1)In my code I search and filter the data using regular expression. Instead of this you will have to write entire data (not filtered data)to XML file using the following method.


         Private Shared Sub WriteXmlToFile(ByVal thisDataSet As DataSet)
               If thisDataSet Is Nothing Then
                   Return
               End If
               
               thisDataSet.WriteXml(XMLFile)
           End Sub



    2)Later you will need to read the xml from file save it to the shared dataset .
           Private Shared Function ReadXmlFromFile() As DataSet
               ' Create a new DataSet.
               Dim newDataSet As New DataSet("New DataSet")


               ' Read the XML document back in.
               ' Create new FileStream to read schema with.
               Dim fsReadXml As New System.IO.FileStream(XMLFile, System.IO.FileMode.Open)


               ' Create an XmlTextReader to read the file.
               Dim myXmlReader As New System.Xml.XmlTextReader(fsReadXml)


               ' Read the XML document into the DataSet.
               newDataSet.ReadXml(myXmlReader)


               ' Close the XmlTextReader
               myXmlReader.Close()
               Return newDataSet


           End Function
    3)For each search you will later need to filter it according to search results. Filter dataset might look like this. Searchs.QueryBuilder.Build(Field, m_searchWords) is the function to build the query.



          Private Sub FiterPagesDatset()
               Dim strExpr As String
               Dim foundRows As DataRow()


               Dim Field() As String = {"Title", "Description", "Keywords", "Contents"}


               strExpr = Searchs.QueryBuilder.Build(Field, m_searchWords)


             
               foundRows = Searchs.Site.PageDataset.Tables("Pages").Select(strExpr)
               FillDataset(foundRows)
           End Sub
    4)The filtered result store it into another dataset and use it to display results

  53. Hardy Wang
    17 Feb 2004 at 09:22

    Especially there are large amount of request at same time?

  54. Stevan Rodrigues
    17 Feb 2004 at 04:16

    When the application is place in the root you may get the following errors.
    The remote server returned an error: (401) Unauthorized.
    OR
    The remote server returned an error: (500) Internal Server Error.


    This error is caused because
    1)If server returns (401) Unauthorized, the application is unable to read the file beacuse of right access.
    2)If server returns (500) Internal Server Error, the page that it was trying to read returned an error. The page that application was trying to read either has an error or requires parameters because of which it returns error


    Follow the steps to rectify the error
    1)Modify the GetDynamicFileContent method in the UserSearch.vb file


           '**********************
           '
           ' GetDynamicFileContent Method
           '
           ' File Content is picked in this method
           '
           '    **********************
           Private Sub GetDynamicFileContent(ByVal srchFile As Searchs.Page)
               Dim wcMicrosoft As System.Net.WebClient
               Dim objUTF8Encoding As UTF8Encoding
               Dim strResponse As String


               Try
                   wcMicrosoft = New System.Net.WebClient()
                   objUTF8Encoding = New UTF8Encoding()
                   srchFile.Contents = objUTF8Encoding.GetString( _
                   wcMicrosoft.DownloadData(String.Format("{0}/{1}", Searchs.Site.ApplicationPath, srchFile.Path)))
               Catch ex As System.Net.WebException
                   m_page.Trace.Warn("Error", ex.Message)
               End Try


           End Sub


    2)Rebuild the code
    3)In the Web.config ensure that the BarredFolders list is comprehensive
    aspnetclient,private,vticnf,vtilog,vtipvt,vtiscript,vtitxt,cgibin,bin,bin,_notes,images,scripts
    4)Ensure that the BarredFiles list is comprehensive and contains localstart.asp,iisstart.asp
    5)Send me the mail if this issue gets resolved

  55. DLynes
    16 Feb 2004 at 05:33

    Great code, but I'm concerned about performance issues on large sites or sites with lots of users. Any chance you could post the XML version sometime soon?

  56. Stevan Rodrigues
    10 Feb 2004 at 03:23

    Thanks James,
    Quick work! Code can now be downloaded.
    Regards,
    Stevan Rodrigues
    MCSD.net(Early achiever),
    MCAD.Net(Charter Member),MCP
    Mobile: 00971 50 5381435

  57. Stevan Rodrigues
    09 Feb 2004 at 08:04

    Hi James,
    Thanks for all.
    You have unzipped the searchdotnet.zip file in the parent zip file so the link in html page does not work.
    I could not upload the images. So included  the images and html file. Since you have added the images we do not require the page and images. But the searchdotnet.zip file in the parent zip is required. So just add the demo code in the zip so the users can download it and test.
    Thanks for all.
    Regards,
    Stevan Rodrigues
    MCSD.net(Early achiever),
    MCAD.Net(Charter Member),MCP
    Mobile: 00971 50 5381435

  58. Stevan Rodrigues
    09 Feb 2004 at 07:46

    Hi Thushan,
    Thanks for the compliment. The inspiration was that there was no code available so write your own. Now who wants to reinvent the wheels.  
    Chief I have adopted the best object oriented practises and enhanced the code but some people right away rejected it as not useful at Codeproject. It was heart breaking. So I delayed writing my second article on the xml version. But today I have got compliments from a no of places. I am happy that I wrote the article.

  59. Thushan Fernando
    09 Feb 2004 at 07:15

    Steven,


    Great well James is a busy lad(like most of us here) so sometimes things get delayed. If you sent him the code give it a few days for him to sort it out


    and yes i was the first to rate it 5/5 here and also the codeproject site


    I see you have got some inspiration from WebWizguide's ASP search engine? i used to use that too...

  60. Stevan Rodrigues
    09 Feb 2004 at 06:31

    Thanks Thushan,
    I am happy to know about your comments. It was really encouraging. Code project doesnot have the latest version. I updated the code further and send it to codeproject but no use. Also I uploaded the demo project here. But the admin did n't upload. May be a mistake. Well you can get in touch with me at stevanrin@hotmail.com where I can give you the latest code. Also I have written another version which writes to XML file and reads from it. This code enhances the speed. I still need to document it. I was waiting for a positive feedback for a go ahead.
    But don't forget to rate because that tells me where I stand.

  61. James Crowley
    09 Feb 2004 at 06:25

    The demo code is there - just click the "Download Source Code" link - that zip file includes source code and a demo. (We just can't provide an online demo - you'll have to download it and run it on your own machine)


    I've also added images to the tutorial now.


    Stevan - if you want a different download to be put up, just email me with it

  62. Stevan Rodrigues
    09 Feb 2004 at 06:14

    Hi,
    Thanks for the comments. Its quite encouraging. I did upload the demo code but realised that the admin has not uploaded it. I will send a quick mail. Also you can get in touch by email at stevanrin@hotmail.com. I will pass on the code.
    Regards,
    Stevan Rodrigues
    MCSD.net(Early achiever),
    MCAD.Net(Charter Member),MCP
    Mobile: 00971 50 5381435

  63. Thushan Fernando
    09 Feb 2004 at 05:45

    hi,


    I've been looking at this code since it was posted at CodeProject on the 29th (eagerly awaited article i might add)... Steve has put an example here:
    http://www.codeproject.com/aspnet/SearchDotnet.asp


    I read your other post as well, and i think if its 6000+ pages you should do something about storing common queries in a file(XML?/xSQLx) so that you save CPU cycles...


  64. Emily Grettel
    09 Feb 2004 at 05:38

    Thanks for the post its really really helpful in needy times.


    Would you be able to give us a live demo of this or maybe have a completed version of it uploaded as code so its easier for people in the future?


    Thanks alot!

  65. Developer Fusion Bot
    01 Jan 1999 at 00:00

    This thread is for discussions of Create a Site Search Engine in ASP.NET.

Leave a comment

Sign in or Join us (it's free).

Related tags

asp.net
Stevan Rodrigues I am a Microsoft Certified Solutions Developer in .Net Architecture (MCSD.Net Early Achiever – one among the first 2500 worldwide), Microsoft Certified Application Developer in .Net – MCAD.Net (Cha...
AddThis

Related articles

Ask a question

Related discussion

Related podcasts

Languages

Web Development

Frameworks & Architecture

Other major sections

Cancel
We'd love to hear what you think! Submit ideas or give us feedback