1. Use the full url of your site for the “Starting URL” option. The crawler will explore only the URLs within the starting directory, i.e. when starting URL is “http://www.example.com/path/index.html”, the “http://www.example.com/path/sub/page.html” will be indexed, but “http://www.example.com/other/index.html” will NOT.
2. “Save sitemap to” – is the filename in the “public_html/” folder of your website. This file should be writable by the script. To make sure it is, create this file and set its permissions to 0666.
3. It is recommended to use “Server’s repsonse” for “Last modification” field. In this case the entries for static pages will be filled with their real last modification time, while for dynamic pages the current time is used.
4. “Do not parse” input field contains file types, separated by space. These files will be added to the sitemap, but not fetched to save bandwidth, because they are not html files and have no embedded links. Please make sure these files are indexed by Google since there is no sense in adding them to sitemap otherwise!
5. “Do not parse URLs” works together with the option above to increase the speed of sitemap generation. If you are sure that some pages at your site do not contain the unique links to other pages, you can tell generator not to fetch them.
   For instance, if your site has “view article” pages with urls like “viewarticle.php?..”, you may want to add them here, because most likely all links inside these pages are already listed at “higher level” (like the list of articles) documents as well:
   • viewarticle.php?id=

   If you are not sure what to write here, just leave this field empty. Please note that these pages are still included into sitemap.

6. “Exclude extensions” – these files are not crawled and not included in sitemap.
7. To disallow the part of your website from inclusion to the sitemap use “Exclude URLs” setting: all URLs that contain the strings specified will be skipped.
   For instance, to exclude all pages within “www.domain.com/folder/” add this line:
   • folder/

   If your site has pages with lists that can be reordered by columns and URLs look like “list.php?sort=column2″, add this line to exclude duplicate content:

   • sort=

   Anyway, you may leave this box empty to get ALL pages listed.

8. “Include ONLY URLs” setting is the opposite to “Exclude URLs”. When it is not empty, ONLY the urls that match the substring entered are included into sitemap.
9. “Individual attributes” setting allows you to set specific values for last modification time, frequency andpriority per page. To use it, define specific frequency and priority attributes in the following format: “url substring,lastupdate YYYY-mm-dd,frequency,priority”.
   Example:
   page.php?product=,2005-11-14,monthly,0.9
10. You may want to limit the number of pages to index to make sure it will not be endless if your website have an error like unlimited looped links.
11. To limit the maximum running time of the script, define the “Maximum execution time” field (in seconds).
12. To have a possibility to use “Resume session” feature, define the “Save the script state” field. This value means the intervals to save the crawler process state, so in case the script was interrupted, you can continue the process from the last saved point. Set this value to “0″ to disable savings.
13. To reduce the load on your server made by the sitemap generator, you can add the “sleep” delay after each N (configured) requests to your site for X seconds (configured). Leave blank (“0″) values to crawls the site without delays.
14. Google doesn’t support sitemap files with more than 50,000 pages. That’s why script supports “Sitemap Index” creation for the big sites. So, it will create one sitemap index file and multiple sitemap files with 50 thousand pages each.
    For instance, your website has about 140,000 pages. The XML sitemap generator will create these files:
    • “sitemap.xml” – sitemap index file that includes links to other files (filename depends on what you entered in the “Save sitemap to” field)
    • “sitemap1.xml” – sitemap file (URLs from 1 to 50,000)
    • “sitemap2.xml” – sitemap file (URLs from 50,001 to 100,000)
    • “sitemap3.xml” – sitemap file (URLs from 100,001 to 140,000)

    Please make sure all of these files are writable if your website is large.

15. Enable “Create HTML Sitemap” option to let generator create a sitemap for your visitors. You should also define the “HTML Sitemap filename” where the sitemap will be stored. It is possible to split html sitemap onto multiple files by defining the “Number of links per page in HTML sitemap” option.
    The filenames are like the following:
    • “sitemap.html” – in case when all links fit in one file
      OR
    • “sitemap1.html” – site map file, page 1
    • “sitemap2.html” – site map file, page 2
    • etc

    Same as point above: please make sure all of these files are writable.

    The site map pages layout can be modified to suit to your website in pages/sitemap_tpl.html file.
    Besides modifying the stylesheet for html sitemap, you can change the way it is formatted. The basic template commands are:

    • <TLOOP XX>…</TLOOP> – defines a repeating sequence of code (like page numbers or sitemap links)
    • <TIF XX>…</TIF> – defines a conditional statement that is inserted only when a specific term is met
    • <TVAR XX> – inserts a value of a specified variable

    Please refer to sitemap_tpl.html file for usage example.

16. Enable GZip compression of sitemap files to save on disk space and bandwidth. In this case “.gz” will be added to sitemap filenames (like “sitemap.xml.gz”).
17. “Sitemap URL” is the same file entered in “Save sitemap to” field, but in the URL form. It is required to inform Google about sitemap address.
18. Set “Ping Google” checkbox enabled to let the script inform Google on every sitemap change. In this way you will always let google know about the fresh information on your site.
19. If you want to restrict access to your generator pages, set the login and password here.

Tags:configure, crawlers, Generator, google, sitemap, Standalone, xml

Liked this article? To continue getting our latest free Howtos and Tutorials,
you can also grab the RSS feed or Subscribe to Techgurulive by Email

Not Getting

Related posts

A Brief to Google Crawling and Google Indexing

How to submit a sitemap to Google Webmaster tools