Natural language processing tools

Lê Đức Trọng 1

Crawler and Parser tools

• • Crawler tools: • Crawler 4j: http://code.google.com/p/crawler4j/ • httpClient: http://hc.apache.org/httpclient-3.x/ Parser tools: • htmlParser: http://htmlparser.sourceforge.net/ • Jsoup html parser: http://jsoup.org/ • Neko html parser: http://nekohtml.sourceforge.net/ 2

Vietnamese NLP – Tools

• • JVnTextPro: http://sourceforge.net/projects/jvntextpro/ • Sentence Segmentation, Sentence Tokenization, Word Segmentation, POS-Tagging VnToolkit: http://www.loria.fr/~lehong/softwares.php

• • • An automatic tagger for Vietnamese texts A tokenize for automatic word segmentation of Vietnamese texts A sentence detector for automatic detecting sentences of Vietnamese texts • VLSP Tools: http://vlsp.vietlp.org:8080/demo/?page=resources • Vietnamese Chunking 3

NLP Toolkits

• • • • • LingPipe: http://alias-i.com/lingpipe/ • Find the names of people, organizations or locations in news • Automatically classify Twitter search results into categories • Suggest correct spellings of queries Mallet - Machine Learning for Language Toolkit: http://mallet.cs.umass.edu/ • Statistic, document classification, clustering, topic modeling, information extraction Stanford NLP softwares: http://www-nlp.stanford.edu/software/ • Word segmentation, part-of-speech tagging, named entity recognition, chunking, parsing, classification and coreference resolution NLTK: http://www.nltk.org/ • Open source Python modules, linguistic data and documentation for research and development in natural language processing and text analytics.

OpenNLP: http://opennlp.apache.org/ • Tokenization, sentence segmentation, part-of-speech tagging, named entity extraction, chunking, parsing, and coreference resolution 4

Machine learning libraries

• • • Conditional random fields (CRF) • CRF: http://crf.sourceforge.net/ Maximum entropy (Maxent) • OpenNLP, Mallet Support vector machine (SVM) • • libSVM: http://www.csie.ntu.edu.tw/~cjlin/libsvm/ svmLight: http://svmlight.joachims.org/ 5