This is an old revision of the document!

---

Cutter - a Universal Multilingual Tokenizer

History

• end of 2015 – concept and first development version (PHP)
• April 2016 – first release
• until May 2017 – continued development for up to 17 languages
• from January 2018 – reimplementation in Python
• June 2018 – released as Python module

Demos

• version 1.0 (Apr. 2016)
• version 1.2 (Jul. 2016)
• version 1.4 (Feb. 2017)
• version 1.6 (May 2017)
• version 2.0 (June 2018)
• version 2.1 (August 2018)
• version 2.2 (September 2018)

Source

• graen/cutter (version 1.x – PHP)
• graen/cutter-ng and https://github.com/j0hannes/cutter-ng (version 2.x – Python3)

Contributors

• Johannes Graën
• Martin Volk
• Mara Bertamini
• Chantal Amrhein
• Phillip Ströbel
• Anne Göhring
• Natalia Korchagina
• Simon Clematide
• Daniel Wüest
• Alex Flückiger

Citation

cutter.bib

@inproceedings{GraenBertaminiVolk2018,
          number = {2226},
           month = {June},
          author = {Johannes Gra{\"e}n and Mara Bertamini and Martin Volk},
          series = {CEUR Workshop Proceedings},
       booktitle = {Swiss Text Analytics Conference},
          editor = {Mark Cieliebak and Don Tuggener and Fernando Benites},
           title = {Cutter -- a Universal Multilingual Tokenizer},
       publisher = {CEUR-WS},
            year = {2018},
           pages = {75--81},
             url = {https://doi.org/10.5167/uzh-157243},
            issn = {1613-0073},
}