Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type TokenCounter ¶
type TokenCounter struct {
// Encoding is the encoding to use for tokenization. Defaults to cl100k_base
Encoding string
}
TokenCounter is an interface for counting tokens in data using tiktoken for tokenization
Source Files
Click to show internal directories.
Click to hide internal directories.