AIAC IndicNLP Tokenizer Baseline & Pre-processing Utilities

AIAC Lab Contributors

GitHub Open Source Release · 2026-01-20

Abstract

Modular Python utility package providing standardized Unicode normalization, sentence splitting, and whitespace sanitization for regional text datasets. Published with complete unit tests and typing stubs.

Cite this Work

@techreport{aiac_osr_2026_01,
  title = {AIAC IndicNLP Tokenizer Baseline & Pre-processing Utilities},
  author = {AIAC Lab Contributors},
  institution = {AI & Analytics Club Research Division},
  year = {2026},
  url = {https://aiac.siddharthgupta.com.np/outputs/osr-2026-01}
}

License & Open Access

Distribution: This work is released under Apache-2.0. You are free to share and adapt the material provided appropriate attribution is credited to AIAC and original student authors.