Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func StartCrawling ¶
func StartCrawling(baseURL string, maxDepth int, showTree bool, treeWriter, siteMapWriter io.Writer)
StartCrawling is the main entry point for crawling. It crawls the given URL in depth first search manner.
params:
BaseURL: It is the starting URL for the crawler MaxDepth: It determines the depth of depth-first-search ShowTree: It determines if the tree should be generated for the crawled pages TreeWriter: If showTree is true, the tree is written to treeWriter SiteMapWriter: The xml sitemap is written to the sitemapwriter
Types ¶
type CrawlerState ¶
CrawlerState stores the global state of crawling. It is go rountine safe.
func NewCrawlerState ¶
func NewCrawlerState() *CrawlerState
NewCrawlerState returns a new CrawlerState
func (*CrawlerState) AddURL ¶
func (c *CrawlerState) AddURL(url string) bool
AddURL tries to insert the new url into the global URL cache. Returns false if URL was already present and true if not.
func (*CrawlerState) IncrementCrawledCount ¶
func (c *CrawlerState) IncrementCrawledCount()
IncrementCrawledCount increases the crawled URL count by 1
func (*CrawlerState) WriteSiteMap ¶
func (c *CrawlerState) WriteSiteMap(f io.Writer)
WriteSiteMap generates sitemap from the given list of URLs The sitemap is minimal and contains only the mandatory <loc> field Sample sitemap
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url> <loc>http://foo.com</loc> </url>
</urlset>