You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi,
I am trying to find a sample code to use scan() or scroll() api in the client, that would fetch paginated results from ES.
Can someone please point me to the reference on how to use that ?
Hello, the esapi.Scroll() API is just a wrapper over the Elasticsearch's Scroll API, so the mechanics of using it are as explained in the docs — start the initial search, getting first batch of results and the scroll ID, and then continue to perform the scroll requests until the hits are empty.
I was thinking about adding a fully-fleshed example into the _examples folder, similar to the Bulk example, in order to make the mechanics more clear. In the meantime, here's a complex example:
package main
import (
"bytes""io""log""strconv""strings""time""github.com/elastic/go-elasticsearch/v8""github.com/tidwall/gjson"
)
funcmain() {
log.SetFlags(0)
var (
batchNumintscrollIDstring
)
es, _:=elasticsearch.NewDefaultClient()
// Index 100 documents into the "test-scroll" index//log.Println("Indexing the documents...")
fori:=1; i<=100; i++ {
res, err:=es.Index(
"test-scroll",
strings.NewReader(`{"title" : "test"}`),
es.Index.WithDocumentID(strconv.Itoa(i)),
)
iferr!=nil||res.IsError() {
log.Fatalf("Error: %s: %s", err, res)
}
}
es.Indices.Refresh(es.Indices.Refresh.WithIndex("test-scroll"))
// Perform the initial search request to get// the first batch of data and the scroll ID//log.Println("Scrolling the index...")
log.Println(strings.Repeat("-", 80))
res, _:=es.Search(
es.Search.WithIndex("test-scroll"),
es.Search.WithSort("_doc"),
es.Search.WithSize(10),
es.Search.WithScroll(time.Minute),
)
// Handle the first batch of data and extract the scrollID//json:=read(res.Body)
res.Body.Close()
scrollID=gjson.Get(json, "_scroll_id").String()
log.Println("Batch ", batchNum)
log.Println("ScrollID", scrollID)
log.Println("IDs ", gjson.Get(json, "hits.hits.#._id"))
log.Println(strings.Repeat("-", 80))
// Perform the scroll requests in sequence//for {
batchNum++// Perform the scroll request and pass the scrollID and scroll duration//res, err:=es.Scroll(es.Scroll.WithScrollID(scrollID), es.Scroll.WithScroll(time.Minute))
iferr!=nil {
log.Fatalf("Error: %s", err)
}
ifres.IsError() {
log.Fatalf("Error response: %s", res)
}
json:=read(res.Body)
res.Body.Close()
// Extract the scrollID from response//scrollID=gjson.Get(json, "_scroll_id").String()
// Extract the search results//hits:=gjson.Get(json, "hits.hits")
// Break out of the loop when there are no results//iflen(hits.Array()) <1 {
log.Println("Finished scrolling")
break
} else {
log.Println("Batch ", batchNum)
log.Println("ScrollID", scrollID)
log.Println("IDs ", gjson.Get(hits.Raw, "#._id"))
log.Println(strings.Repeat("-", 80))
}
}
}
funcread(r io.Reader) string {
varb bytes.Bufferb.ReadFrom(r)
returnb.String()
}
Reacted by Filip, tx991020, Pimousse1099, Paritosh Bhatia, b4nks, John Zenk, Ad L'Ecluse, Sun Dawn, Riki Syahputra, Florin Vitan and 11 more
Quick question regarding the multiple defer res.Body.Close().
shouldn't the res.Body.Close() be called right after each log.Println(strings.Repeat("-", 80)) and before the for loop as we don't need all thos the response body buffers to still be opened ?
@rmasclef, you're right, the defer is redundant here, and keeps too many buffers open until the main function exits. I'll update the example so it's not confusing to people coming to this from Google etc.
@arroadie , no idea where the 16666h40m0s value would be coming from, when I try locally, scroll=60000ms is being used... Can you enable logging and see what's being sent to ES?
disregard my comment. My go.mod was pointing to version 0, which is probably not up to date with the changes you made to fix that.
fixed the mod file and using the correct client made it work as expected.
thanks
[400 Bad Request] {"error":{"root_cause":[{"type":"too_long_frame_exception","reason":"An HTTP line is larger than 4096 bytes."}],"type":"too_long_frame_exception","reason":"An HTTP line is larger than 4096 bytes."},"status":400}
do you have any idea to help me shorter the scroll id.
Hi @NguyenKyThinh94 you get this error because the WithScrollID passes the ID via query string.
To circumvent this limitation you can send the scroll_id via the WithBody as indicated in the documentation.
Reacted by Bhargav Saripalli, Anuj Varma and battlecook
Activity
karmi commentedon Apr 17, 2019
Hello, the
esapi.Scroll()API is just a wrapper over the Elasticsearch's Scroll API, so the mechanics of using it are as explained in the docs — start the initial search, getting first batch of results and the scroll ID, and then continue to perform the scroll requests until the hits are empty.I was thinking about adding a fully-fleshed example into the
_examplesfolder, similar to the Bulk example, in order to make the mechanics more clear. In the meantime, here's a complex example:nadgowdas commentedon Apr 17, 2019
Thanks @karmi this was really helpful.
pimousse1099 commentedon Oct 2, 2019
@karmi thanks a lot for that example 👍
Quick question regarding the multiple
defer res.Body.Close().shouldn't the
res.Body.Close()be called right after eachlog.Println(strings.Repeat("-", 80))and before theforloop as we don't need all thos the response body buffers to still be opened ?karmi commentedon Oct 3, 2019
@rmasclef, you're right, the
deferis redundant here, and keeps too many buffers open until the main function exits. I'll update the example so it's not confusing to people coming to this from Google etc.pimousse1099 commentedon Oct 4, 2019
I suggest, as you say above, to add this snippet in the library examples 👌
arroadie commentedon Oct 9, 2019
Hi. I'm using the library on v7 and getting an odd error trying to use the scroll option;
search call being
and the response
do you have any idea why this could be happening?
karmi commentedon Oct 9, 2019
@arroadie , no idea where the
16666h40m0svalue would be coming from, when I try locally,scroll=60000msis being used... Can you enable logging and see what's being sent to ES?go-elasticsearch/_examples/logging/default.go
Lines 48 to 54 in 7f37b7b
arroadie commentedon Oct 9, 2019
disregard my comment. My go.mod was pointing to version 0, which is probably not up to date with the changes you made to fix that.
fixed the mod file and using the correct client made it work as expected.
thanks
karmi commentedon Oct 9, 2019
Argh, correct, handling of
time.Durationvalues was buggy on version 0. Glad it works.NguyenKyThinh94 commentedon Dec 29, 2021
version 7.3.0
scroll_id in result is too long, it more than 12944 letter
Then i call scroll again
And the error reponce:
do you have any idea to help me shorter the scroll id.
Anaethelion commentedon Jan 5, 2022
Hi @NguyenKyThinh94 you get this error because the
WithScrollIDpasses the ID via query string.To circumvent this limitation you can send the
scroll_idvia theWithBodyas indicated in the documentation.woorui commentedon Aug 26, 2022
@NguyenKyThinh94 Hi, I write code below in golang.