Skip to content

About

A webscraper written in Golang and Selenium

Resources

Stars

0 stars

Watchers

1 watching

Forks

Latest commit

 

History

13 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 

Repository files navigation

go-scraper

This is a Go program that uses the Selenium package to perform web scraping on a specified webpage. It can be used to extract the raw HTML content from a webpage.

Why Selenium ?

Prerequisites

Before running this project, you need to have the following installed on your system:

  1. Go: Make sure you have Go installed on your machine. You can download and install it from https://golang.org/dl/

  2. Ensure you have set the paths corretly refer this blog on how to write Go code

  3. Chrome and ChromeDriver: You need Google Chrome and ChromeDriver installed on your system. You can download Chrome from https://www.google.com/chrome/ and ChromeDriver from https://sites.google.com/a/chromium.org/chromedriver/downloads.

  4. Go Selenium Package: You'll need to install the Selenium package for Go. You can install it using the following command:

    go get github.com/tebeka/selenium
    

How to Run Locally

1. Clone the repository:

git clone git@github.com:neeltom92/go-scraper.git
cd go-scraper

2. Open the main.go file and specify the correct path to your ChromeDriver executable in the code, Initialize a Chrome browser instance on port 4444, Replace "/path/to/chromedriver" with the actual path to your ChromeDriver executable.

service, err := selenium.NewChromeDriverService("/path/to/chromedriver", 4444)

3. Build the Go program:

go build

4. Run the program with the URL you want to scrape as a command-line argument. For example, to scrape "https://example.com," run:

./go-scraper https://google.com

Example:

  1. For example here we are scraping google.com:
Image 1 Image 2 Image 3

About

A webscraper written in Golang and Selenium

Resources

Stars

0 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages

⌂ 换网址