Skip to content

es.scroll() sample #44

Closed
Closed

Description

@nadgowdas

Hi,
I am trying to find a sample code to use scan() or scroll() api in the client, that would fetch paginated results from ES.
Can someone please point me to the reference on how to use that ?

Activity

  1. karmi commented on Apr 17, 2019

    @karmi
    Contributor

    Hello, the esapi.Scroll() API is just a wrapper over the Elasticsearch's Scroll API, so the mechanics of using it are as explained in the docs — start the initial search, getting first batch of results and the scroll ID, and then continue to perform the scroll requests until the hits are empty.

    I was thinking about adding a fully-fleshed example into the _examples folder, similar to the Bulk example, in order to make the mechanics more clear. In the meantime, here's a complex example:

    package main
    
    import (
    	"bytes"
    	"io"
    	"log"
    	"strconv"
    	"strings"
    	"time"
    
    	"github.com/elastic/go-elasticsearch/v8"
    	"github.com/tidwall/gjson"
    )
    
    func main() {
    	log.SetFlags(0)
    
    	var (
    		batchNum int
    		scrollID string
    	)
    
    	es, _ := elasticsearch.NewDefaultClient()
    
    	// Index 100 documents into the "test-scroll" index
    	//
    	log.Println("Indexing the documents...")
    	for i := 1; i <= 100; i++ {
    		res, err := es.Index(
    			"test-scroll",
    			strings.NewReader(`{"title" : "test"}`),
    			es.Index.WithDocumentID(strconv.Itoa(i)),
    		)
    		if err != nil || res.IsError() {
    			log.Fatalf("Error: %s: %s", err, res)
    		}
    	}
    	es.Indices.Refresh(es.Indices.Refresh.WithIndex("test-scroll"))
    
    	// Perform the initial search request to get
    	// the first batch of data and the scroll ID
    	//
    	log.Println("Scrolling the index...")
    	log.Println(strings.Repeat("-", 80))
    	res, _ := es.Search(
    		es.Search.WithIndex("test-scroll"),
    		es.Search.WithSort("_doc"),
    		es.Search.WithSize(10),
    		es.Search.WithScroll(time.Minute),
    	)
    
    	// Handle the first batch of data and extract the scrollID
    	//
    	json := read(res.Body)
    	res.Body.Close()
    
    	scrollID = gjson.Get(json, "_scroll_id").String()
    
    	log.Println("Batch   ", batchNum)
    	log.Println("ScrollID", scrollID)
    	log.Println("IDs     ", gjson.Get(json, "hits.hits.#._id"))
    	log.Println(strings.Repeat("-", 80))
    
    	// Perform the scroll requests in sequence
    	//
    	for {
    		batchNum++
    
    		// Perform the scroll request and pass the scrollID and scroll duration
    		//
    		res, err := es.Scroll(es.Scroll.WithScrollID(scrollID), es.Scroll.WithScroll(time.Minute))
    		if err != nil {
    			log.Fatalf("Error: %s", err)
    		}
    		if res.IsError() {
    			log.Fatalf("Error response: %s", res)
    		}
    
    		json := read(res.Body)
    		res.Body.Close()
    
    		// Extract the scrollID from response
    		//
    		scrollID = gjson.Get(json, "_scroll_id").String()
    
    		// Extract the search results
    		//
    		hits := gjson.Get(json, "hits.hits")
    
    		// Break out of the loop when there are no results
    		//
    		if len(hits.Array()) < 1 {
    			log.Println("Finished scrolling")
    			break
    		} else {
    			log.Println("Batch   ", batchNum)
    			log.Println("ScrollID", scrollID)
    			log.Println("IDs     ", gjson.Get(hits.Raw, "#._id"))
    			log.Println(strings.Repeat("-", 80))
    		}
    	}
    }
    
    func read(r io.Reader) string {
    	var b bytes.Buffer
    	b.ReadFrom(r)
    	return b.String()
    }
  2. nadgowdas commented on Apr 17, 2019

    @nadgowdas
    Author

    Thanks @karmi this was really helpful.

  3. pimousse1099 commented on Oct 2, 2019

    @pimousse1099

    @karmi thanks a lot for that example 👍

    Quick question regarding the multiple defer res.Body.Close().

    shouldn't the res.Body.Close() be called right after each log.Println(strings.Repeat("-", 80)) and before the for loop as we don't need all thos the response body buffers to still be opened ?

  4. karmi commented on Oct 3, 2019

    @karmi
    Contributor

    @rmasclef, you're right, the defer is redundant here, and keeps too many buffers open until the main function exits. I'll update the example so it's not confusing to people coming to this from Google etc.

  5. pimousse1099 commented on Oct 4, 2019

    @pimousse1099

    I suggest, as you say above, to add this snippet in the library examples 👌

  6. arroadie commented on Oct 9, 2019

    @arroadie

    Hi. I'm using the library on v7 and getting an odd error trying to use the scroll option;
    search call being

    resp, err := es.Search(
    		es.Search.WithIndex("indexed-data"),
    		es.Search.WithBody(strings.NewReader(query)),
    		es.Search.WithSize(10),
    		es.Search.WithScroll(time.Minute),
    	)

    and the response

        "error":{
            "root_cause":[
                {
                    "type":"illegal_argument_exception",
                    "reason":"failed to parse [16666h40m0s]"
                }
            ],
            "type":"illegal_argument_exception",
            "reason":"failed to parse [16666h40m0s]",
            "caused_by":{
                "type":"number_format_exception",
                "reason":"For input string: \"16666h40m0\""
            }
        },
        "status":400
    
    }

    do you have any idea why this could be happening?

  7. karmi commented on Oct 9, 2019

    @karmi
    Contributor

    @arroadie , no idea where the 16666h40m0s value would be coming from, when I try locally, scroll=60000ms is being used... Can you enable logging and see what's being sent to ES?

    es, _ = elasticsearch.NewClient(elasticsearch.Config{
    Logger: &estransport.ColorLogger{
    Output: os.Stdout,
    EnableRequestBody: true,
    EnableResponseBody: true,
    },
    })

  8. arroadie commented on Oct 9, 2019

    @arroadie

    disregard my comment. My go.mod was pointing to version 0, which is probably not up to date with the changes you made to fix that.
    fixed the mod file and using the correct client made it work as expected.
    thanks

  9. karmi commented on Oct 9, 2019

    @karmi
    Contributor

    Argh, correct, handling of time.Duration values was buggy on version 0. Glad it works.

  10. NguyenKyThinh94 commented on Dec 29, 2021

    @NguyenKyThinh94

    version 7.3.0

    	result, err := esClient.Search(
    		esClient.Search.WithIndex(payload.Index),
    	        esClient.Search.WithBody(&buf),
    		esClient.Search.WithSize(10),
    		esClient.Search.WithSort("create_date"),
    		esClient.Search.WithScroll(time.Minute),
    	)

    scroll_id in result is too long, it more than 12944 letter
    Then i call scroll again

    		res, err := esClient.Scroll(esClient.Scroll.WithScrollID(scrollID), esClient.Scroll.WithScroll(time.Minute))

    And the error reponce:

    [400 Bad Request] {"error":{"root_cause":[{"type":"too_long_frame_exception","reason":"An HTTP line is larger than 4096 bytes."}],"type":"too_long_frame_exception","reason":"An HTTP line is larger than 4096 bytes."},"status":400}
    

    do you have any idea to help me shorter the scroll id.

  11. Anaethelion commented on Jan 5, 2022

    @Anaethelion
    Contributor

    Hi @NguyenKyThinh94 you get this error because the WithScrollID passes the ID via query string.
    To circumvent this limitation you can send the scroll_id via the WithBody as indicated in the documentation.

  12. woorui commented on Aug 26, 2022

    @woorui

    @NguyenKyThinh94 Hi, I write code below in golang.

    scrollBytes, _ := json.Marshal(&map[string]string{"scroll_id": scrollID})
    
    res, err := es.Scroll(es.Scroll.WithBody(bytes.NewBuffer(scrollBytes)), es.Scroll.WithScroll(time.Minute))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      ⌂ 换网址