Repository navigation
Item Pipeline
Item Pipeline is one of the Antch core components to process a sequence of input values.
Item is an interface type, so you can passed any value into the item pipeline.
Pipeline Interface:
type PipelineHandler interface {
ServePipeline(Item)
}
type Pipeline func(PipelineHandler) PipelineHandlertype OutputPipeline struct {
next antch.PipelineHandler
}
func (p *OutputPipeline) ServePipeline(v antch.Item) {
fmt.Println(v)
p.next.ServePipeline(v)
}And then register you pipeline into the crawler via UsePipeline().
pipeline := func() antch.Pipeline {
return func(next antch.PipelineHandler) antch.PipelineHandler {
return &OutputPipeline{next}
}
}
crawler := antch.NewCrawler()
crawler.UsePipeline(pipeline())