Text Segmentation by TextTiling with Modern Neural Retrieval Models

Text segmentation is a foundational task in natural language processing, supporting a wide range of applications such as document summarization, topic modeling, media content repurposing, and more recently, Retrieval-Augmented Generation (RAG) systems ( 2503.09600 ). Effective segmentation enables better contextualization of content, allowing downstream models to focus on coherent blocks of information rather than arbitrary fixed-length inputs. Classical approaches like TextTiling segment texts into semantically coherent passages based on lexical cohesion, typically using text similarity measures. Subsequent approaches have revisited the TextTiling idea leveraging representations obtained from encoder models such as BERT ( PeerJ-CS , GitHub - DeepTiling ). More recently, with the advent of neural retrieval and large language models (LLMs), there is a renewed interest in developing segmentation strategies that can more precisely model topical boundaries and improve the efficiency and quality of retrieval-based systems. Recent approaches have even proposed the use of LLMs to perform text segmentation ( 2406.17526 ), although computational efficiency is a concern with these methods.

In the context of this M.Sc. research project, the candidate will revisit the original TextTiling method and augment it by integrating modern neural sentence representations, particularly those derived from models pre-trained for dense retrieval tasks (e.g., sentence-transformers or embedding models based on larger Transformer decoders fine-tuned for semantic search). By replacing TF-IDF vectors with dense embeddings, the method aims to improve the detection of semantic shifts between text segments, while maintaining the interpretability and unsupervised nature of the original algorithm. In parallel, we will compare this enhanced TextTiling method with LLM-based segmentation approaches such as LumberChunker, which dynamically determine chunk boundaries through generative prompting. The goal is to assess the trade-offs between lightweight embedding-based segmentation, and more computationally intensive instruction-tuned models in real-world scenarios.

Evaluation will be performed using a mixture of segmentation benchmarks, perhaps also considering downstream task performance within RAG pipelines. Benchmarks such as Wiki-727K, or the GutenQA benchmark used in LumberChunker, will be employed to quantify segmentation quality using established metrics. A technical report will document the results, envisioning the submission to a leading conference in natural language processing or information retrieval.