fix: Bring in fix from custom nodes

[sjrl]

Related Issues

Brings in the fix first introduced here https://github.com/deepset-ai/deepset-cloud-custom-nodes/pull/367

Proposed Changes:

• To avoid fully subsuming the previous chunk, we ignore the first sentence from that chunk when calculating sentence overlap. i.e. we want to avoid cases of: Doc1 = [s1, s2], Doc2 = [s1, s2, s3].
• Only applies when splitting by word and respecting sentence boundary.
• This mirrors the implementation in v1's PreProcessor.

Since this component was a port of the functionality of Haystack v1 I view this as a bug fix.

A few other changes:

• While I was here I also finished adding function support for this component by updating the _split_into_units function and added the splitting_function init parameter.
• Also fixed another bug which was caused by the introduction of the to_dict method of the underlying DocumentSplitter when adding function support. This means without defining a new to_dict method for this component, a number of parameters were being ignored when saving this component to yaml. So I added a to_dict method for this component and added a corresponding test.

How did you test it?

Copied over tests

Notes for the reviewer

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.
• I documented my code
• I ran pre-commit hooks and fixed any issue

[coveralls]

Pull Request Test Coverage Report for Build 11812877811

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 11 unchanged lines in 1 file lost coverage.
• Overall coverage decreased (-0.02%) to 90.159%

---

Files with Coverage Reduction	New Missed Lines	%
components/preprocessors/nltk_document_splitter.py	11	94.3%

Totals
Change from base Build 11799929241:	-0.02%
Covered Lines:	7787
Relevant Lines:	8637

---

💛 - Coveralls