[Question]: Why is the entire parsed md file divided into one chunk?

[Negai-98]

Describe your problem

The default settings are used. The chunk length of the md file is obviously longer than the setting, which should be due to the absence of a delimiter. The delimiter includes \n but cannot recognize the line break in md.

[Negai-98]

Platform: Windows WSL
Version: 1.14.1-dev

[Snify89]

Carriage Return / Line Feed delimiter issue?

[Negai-98]

It is supposed to be like that, but I am not sure why it fails to recognize the line breaks in the md file. I use ragflow/readme.md as the document.

[Snify89]

It is supposed to be like that, but I am not sure why it fails to recognize the line breaks in the md file. I use ragflow/readme.md as the document.

Could you please check and report back here, what kind of BOM (Byte Order Mark) the file has? This might be an en-/decoding issue.

[Negai-98]

utf-8 @Snify89

[Negai-98]

Manual copy the content from the md file directly into a docx file, which can work properly, but it is best to be able to parse the md file directly.

[Snify89]

Maybe it's a HTML parsing issue?! Haven't reproduced yet, Def, worth a look. Thanks for reporting.

[KevinHuSh]

What about turning down the chunk token number?
Or, set the delimitor: `##`\n\r

[Negai-98]

@KevinHuSh Changing the delimiter or down the chunk token number for splitting does not seem to have an effect on md files; it still combines full md document into one single chunk.

[Negai-98]

I tried reinstalling the latest version of Nightly, but I am still getting the same result.

[Snify89]

Do you get more than 1 chunk, if you use other chunk methods? Maybe the chunk method decided to use just one chunk?

Edit: It's weird tho, that the docx chunks better?!

[Negai-98]

Using the QA mode can chunk normaly, it seems there is a bug in the general mode regarding MD chunking.
@Snify89

[Negai-98]

Additionaly, here I convert the readme.md to docx, and then chunk using general method. It work properly.