Zum Hauptinhalt springen

Class: SentenceSplitter

SentenceSplitter is our default text splitter that supports splitting into sentences, paragraphs, or fixed length chunks with overlap.

One of the advantages of SentenceSplitter is that even in the fixed length chunks it will try to keep sentences together.

Constructors

new SentenceSplitter()

new SentenceSplitter(options?): SentenceSplitter

Parameters

options?

options.chunkOverlap?: number

options.chunkSize?: number

options.chunkingTokenizerFn?

options.paragraphSeparator?: string

options.splitLongSentences?: boolean

options.tokenizer?: Tokenizer

Returns

SentenceSplitter

Source

packages/core/src/TextSplitter.ts:76

Properties

chunkOverlap

chunkOverlap: number

Source

packages/core/src/TextSplitter.ts:69


chunkSize

chunkSize: number

Source

packages/core/src/TextSplitter.ts:68


chunkingTokenizerFn()

private chunkingTokenizerFn: (text) => string[]

Parameters

text: string

Returns

string[]

Source

packages/core/src/TextSplitter.ts:73


paragraphSeparator

private paragraphSeparator: string

Source

packages/core/src/TextSplitter.ts:72


splitLongSentences

private splitLongSentences: boolean

Source

packages/core/src/TextSplitter.ts:74


tokenizer

private tokenizer: Tokenizer

Source

packages/core/src/TextSplitter.ts:71

Methods

combineTextSplits()

combineTextSplits(newSentenceSplits, effectiveChunkSize): TextSplit[]

Parameters

newSentenceSplits: SplitRep[]

effectiveChunkSize: number

Returns

TextSplit[]

Source

packages/core/src/TextSplitter.ts:210


getEffectiveChunkSize()

private getEffectiveChunkSize(extraInfoStr?): number

Parameters

extraInfoStr?: string

Returns

number

Source

packages/core/src/TextSplitter.ts:108


getParagraphSplits()

getParagraphSplits(text, effectiveChunkSize?): string[]

Parameters

text: string

effectiveChunkSize?: number

Returns

string[]

Source

packages/core/src/TextSplitter.ts:126


getSentenceSplits()

getSentenceSplits(text, effectiveChunkSize?): string[]

Parameters

text: string

effectiveChunkSize?: number

Returns

string[]

Source

packages/core/src/TextSplitter.ts:152


processSentenceSplits()

private processSentenceSplits(sentenceSplits, effectiveChunkSize): SplitRep[]

Splits sentences into chunks if necessary.

This isn't great behavior because it can split down the middle of a word or in non-English split down the middle of a Unicode codepoint so the splitting is turned off by default. If you need it, please set the splitLongSentences option to true.

Parameters

sentenceSplits: string[]

effectiveChunkSize: number

Returns

SplitRep[]

Source

packages/core/src/TextSplitter.ts:181


splitText()

splitText(text, extraInfoStr?): string[]

Parameters

text: string

extraInfoStr?: string

Returns

string[]

Source

packages/core/src/TextSplitter.ts:304


splitTextWithOverlaps()

splitTextWithOverlaps(text, extraInfoStr?): TextSplit[]

Parameters

text: string

extraInfoStr?: string

Returns

TextSplit[]

Source

packages/core/src/TextSplitter.ts:276