Documentation
¶
Overview ¶
Package textsplitter provides a simple text splitter that can be used to split text into words.
Index ¶
Constants ¶
View Source
const ( // DefaultTokenChunkOverlap specifies the default number of overlapping tokens // between consecutive chunks. DefaultTokenChunkOverlap = 100 // DefaultTokenChunkSize defines the default maximum number of tokens per // chunk. DefaultTokenChunkSize = 4096 // DefaultTokenEncoding specifies the default encoding scheme for // tokenization. DefaultTokenEncoding = "cl100k_base" // DefaultTokenModelName defines the default model name used for // tokenization. DefaultTokenModelName = "gpt-3.5-turbo" )
Default configuration values for token splitting.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type TokenSplitter ¶
type TokenSplitter struct {
// AllowedSpecial defines the list of allowed special tokens.
AllowedSpecial []string
// ChunkOverlap specifies the number of tokens to overlap between
// consecutive chunks.
ChunkOverlap int
// ChunkSize defines the maximum number of tokens per chunk.
ChunkSize int
// DisallowedSpecial defines the list of disallowed special tokens.
DisallowedSpecial []string
// EncodingName specifies the encoding scheme to use for tokenization.
EncodingName string
// ModelName defines the model to determine token encoding.
ModelName string
}
TokenSplitter implements text splitting functionality based on token count. It provides configuration options for chunk size, overlap, model selection, and special token handling.
func NewTokenSplitter ¶
func NewTokenSplitter(chunkThreshold int) TokenSplitter
NewTokenSplitter creates a new TokenSplitter with default configuration. It allows overriding the default chunk size through the chunkThreshold parameter.
Source Files
Click to show internal directories.
Click to hide internal directories.