scrapper

package
v0.0.0-...-4579cc6 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Feb 8, 2026 License: MIT Imports: 23 Imported by: 0

Documentation

Index

Constants

View Source
const (
	FailedToRunScrapper string = "Failed to run scrapper"

	CantDecodeBodyIntoCriteria string = "Can't decode body into a criteria"
	CantReEnqueueFailedMessage string = "Can't re-enqueue failed message"
)

Variables

View Source
var (
	FailedToLogin                          = errors.New("failed to login")
	FailedToUpdateSearchCriteriaExecution  = errors.New("failed to update search criteria execution")
	FailedToParseDatesFromTheGivenCriteria = errors.New("failed to parse dates from the given criteria")

	FailedToDecodeBodyIntoCriteria              = errors.New("failed to decode body into a criteria")
	FailedToRetrieveSearchCriteriaExecutionData = errors.New("failed to retrieve search criteria execution data")
	FailedToReEnqueueFailedMessage              = errors.New("failed to re-enqueue failed message")
	FailedToRunScrapperProcess                  = errors.New("failed to run scrapper")
)

Functions

func ExecuteHandlerV1

func ExecuteHandlerV1(newWebDriverManager webdriver.NewManager, newScrapper New, messageBroker broker.MessageBroker) http.HandlerFunc

ExecuteHandlerV1 HTTP Handler of the endpoint /scrapper/execute/v1

func MakeSearchCriteriaMessageProcessor

func MakeSearchCriteriaMessageProcessor(getSearchCriteriaExecution corpuscreator.GetSearchCriteriaExecution, newWebDriverManager webdriver.NewManager, newScrapper New, messageBroker broker.MessageBroker) broker.ProcessorFunction

MakeSearchCriteriaMessageProcessor creates a new broker.ProcessorFunction

Types

type Execute

type Execute func(ctx context.Context, searchCriteria criteria.Type, executionID int) error

Execute starts the X (formerly Twitter) scrapper

func MakeExecute

func MakeExecute(login auth.Login, updateSearchCriteriaExecution corpuscreator.UpdateSearchCriteriaExecution, insertSearchCriteriaExecutionDay corpuscreator.InsertSearchCriteriaExecutionDay, executeAdvanceSearch search.ExecuteAdvanceSearch, retrieveTweets tweets.RetrieveAll, saveTweets corpuscreator.SaveTweets) Execute

MakeExecute creates a new Execute

func MockExecute

func MockExecute(err error) Execute

MockExecute mocks Execute function

type New

type New func(webDriver selenium.WebDriver) Execute

New initializes all the functions of a scrapper (only Execute for now) It also initializes all the dependencies for its functions

func MakeNew

func MakeNew(httpClient http.Client, localMode bool) New

MakeNew creates a new New

func MockNew

func MockNew(err error) New

MockNew mocks New function

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址