support for processing large unstructured text files (.txt) using automatic semantic section detection#96

Open

SushantGautam wants to merge 16 commits intoVectifyAI:mainfrom

SushantGautam:main

Commits on Feb 2, 2026

Update utils.py
SushantGautam
authored
Initial plan
Copilot
committed
Add max-input-tokens parameter to limit LLM input

Copilot
and
SushantGautam
committed
Address code review feedback - add documentation and logging

Copilot
and
SushantGautam
committed
Replace truncation with chunking for better handling of large inputs

Copilot
and
SushantGautam
committed
Merge pull request #1 from SushantGautam/copilot/limit-input-to-llm
SushantGautam
authored
Initial plan
Copilot
committed
Fix code review issues: validation, async concurrency, spelling, and documentation

Copilot
and
SushantGautam
committed
Merge pull request #2 from SushantGautam/copilot/fix-chunk-processing-issues
SushantGautam
authored
Initial plan
Copilot
committed
Add .txt file support with windowing and semantic heading detection

Copilot
and
SushantGautam
committed
Update README with text file support documentation

Copilot
and
SushantGautam
committed
Fix merge function to properly handle hierarchical sections

Copilot
and
SushantGautam
committed
Add example script and documentation for .txt file processing

Copilot
and
SushantGautam
committed
Fix character position tracking in semantic section detection

Copilot
and
SushantGautam
committed
Merge pull request #3 from SushantGautam/copilot/extend-framework-to-support-long-txt
SushantGautam
authored