Pular para o conteúdo principal

Class: SentenceSplitter

SentenceSplitter is our default text splitter that supports splitting into sentences, paragraphs, or fixed length chunks with overlap.

One of the advantages of SentenceSplitter is that even in the fixed length chunks it will try to keep sentences together.

Constructors​

new SentenceSplitter()​

new SentenceSplitter(options?): SentenceSplitter

Parameters​

• options?

• options.chunkOverlap?: number

• options.chunkSize?: number

• options.chunkingTokenizerFn?

• options.paragraphSeparator?: string

• options.splitLongSentences?: boolean

• options.tokenizer?: Tokenizer

Returns​

SentenceSplitter

Source​

packages/core/src/TextSplitter.ts:76

Properties​

chunkOverlap​

chunkOverlap: number

Source​

packages/core/src/TextSplitter.ts:69

chunkSize​

chunkSize: number

Source​

packages/core/src/TextSplitter.ts:68

chunkingTokenizerFn()​

private chunkingTokenizerFn: (text) => string[]

Parameters​

• text: string

Returns​

string[]

Source​

packages/core/src/TextSplitter.ts:73

paragraphSeparator​

private paragraphSeparator: string

Source​

packages/core/src/TextSplitter.ts:72

splitLongSentences​

private splitLongSentences: boolean

Source​

packages/core/src/TextSplitter.ts:74

tokenizer​

private tokenizer: Tokenizer

Source​

packages/core/src/TextSplitter.ts:71

Methods​

combineTextSplits()​

combineTextSplits(newSentenceSplits, effectiveChunkSize): TextSplit[]

Parameters​

• newSentenceSplits: SplitRep[]

• effectiveChunkSize: number

Returns​

TextSplit[]

Source​

packages/core/src/TextSplitter.ts:210

getEffectiveChunkSize()​

private getEffectiveChunkSize(extraInfoStr?): number

Parameters​

• extraInfoStr?: string

Returns​

number

Source​

packages/core/src/TextSplitter.ts:108

getParagraphSplits()​

getParagraphSplits(text, effectiveChunkSize?): string[]

Parameters​

• text: string

• effectiveChunkSize?: number

Returns​

string[]

Source​

packages/core/src/TextSplitter.ts:126

getSentenceSplits()​

getSentenceSplits(text, effectiveChunkSize?): string[]

Parameters​

• text: string

• effectiveChunkSize?: number

Returns​

string[]

Source​

packages/core/src/TextSplitter.ts:152

processSentenceSplits()​

private processSentenceSplits(sentenceSplits, effectiveChunkSize): SplitRep[]

Splits sentences into chunks if necessary.

This isn't great behavior because it can split down the middle of a word or in non-English split down the middle of a Unicode codepoint so the splitting is turned off by default. If you need it, please set the splitLongSentences option to true.

Parameters​

• sentenceSplits: string[]

• effectiveChunkSize: number

Returns​

SplitRep[]

Source​

packages/core/src/TextSplitter.ts:181

splitText()​

splitText(text, extraInfoStr?): string[]

Parameters​

• text: string

• extraInfoStr?: string

Returns​

string[]

Source​

packages/core/src/TextSplitter.ts:304

splitTextWithOverlaps()​

splitTextWithOverlaps(text, extraInfoStr?): TextSplit[]

Parameters​

• text: string

• extraInfoStr?: string

Returns​

TextSplit[]

Source​

packages/core/src/TextSplitter.ts:276