Show HN: Chonky – a neural approach for text semantic chunking
a year ago
- #Text Processing
- #Python
- #NLP
- Chonky is a Python library for semantic text segmentation using a transformer model.
- It is useful in RAG (Retrieval-Augmented Generation) systems.
- Installation is simple via pip: `pip install chonky`.
- Usage involves initializing a `TextSplitter` and processing text into meaningful chunks.
- The library downloads the transformer model on first run.
- Example provided demonstrates splitting a sample text into chunks separated by '--'.