crawler

package
v0.0.0-...-cdee56d Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Feb 16, 2019 License: MIT Imports: 8 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func StartCrawling

func StartCrawling(baseURL string, maxDepth int, showTree bool, treeWriter, siteMapWriter io.Writer)

StartCrawling is the main entry point for crawling. It crawls the given URL in depth first search manner.

params:

	BaseURL: 		It is the starting URL for the crawler
	MaxDepth: 		It determines the depth of depth-first-search
	ShowTree: 		It determines if the tree should be generated for the crawled pages
 TreeWriter:		If showTree is true, the tree is written to treeWriter
	SiteMapWriter:	The xml sitemap is written to the sitemapwriter

Types

type CrawlerState

type CrawlerState struct {
	sync.Mutex
	// contains filtered or unexported fields
}

CrawlerState stores the global state of crawling. It is go rountine safe.

func NewCrawlerState

func NewCrawlerState() *CrawlerState

NewCrawlerState returns a new CrawlerState

func (*CrawlerState) AddURL

func (c *CrawlerState) AddURL(url string) bool

AddURL tries to insert the new url into the global URL cache. Returns false if URL was already present and true if not.

func (*CrawlerState) IncrementCrawledCount

func (c *CrawlerState) IncrementCrawledCount()

IncrementCrawledCount increases the crawled URL count by 1

func (*CrawlerState) WriteSiteMap

func (c *CrawlerState) WriteSiteMap(f io.Writer)

WriteSiteMap generates sitemap from the given list of URLs The sitemap is minimal and contains only the mandatory <loc> field Sample sitemap

<?xml version="1.0" encoding="UTF-8"?>

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">

<url>
  <loc>http://foo.com</loc>
</url>

</urlset>

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址