ASV Toolbox

This is the homepage for the ASV Toolbox project. Here you can download the most recent version of the ASV Toolbox(complete) or the modules of the ASV Toolbox. It is written in JAVA (compiled with version 1.5).
Current Version is 1.0.

Introduction

ASV Toolbox is a modular collection of tools for the exploration of written language data. They work either on word lists or text and solve several linguistic classification and clustering tasks. The topics covered contain language detection, POS-tagging, base form reduction, named entity recognition, and terminology extraction. On a more abstract level, the algorithms deal with various kinds of word similarity, using pattern based and statistical approaches. The collection can be used to work on large real world data sets as well as for studying the underlying algorithms. The ASV Toolbox can work on plain text files and connect to a MySQL database. While it is especially designed to work with corpora of the Leipzig Corpora Collection, it can easily be adapted to other sources.

Installation

ASV Toolbox complete anf ASV Toolbox framework

Download the zip file and unzip it into a directory of your choice.

ASV Toolbox modules and modules resources (examples, documentation, languages, ...)

Download the zip file. Unzip the zip file to the directory containing the ASV Toolbox home.
Windows users might simply use "extract here", UNIX users should use "unzip -o <filename>.zip"

If you download a module you have edit the file toolbox.start which you will find in config folder in your ASV toolbox home. Every module has a copy of this file named toolbox.start.modulename. After unzipping the module, this file is located in the config folder. Copy the line into the toolbox.start file (use a new line). Example: if you want to include Genetomorph and ViterbiTagger, your toolbox.start file should look like this:

de.uni_leipzig.asv.toolbox.genetoMorph.GenetoMorph

de.uni_leipzig.asv.toolbox.viterbitagger.gui.ViterbiTagger

ASV Toolbox complete

The complete ASV Toolbox package contains the following modules:
Chinese Whispers: graph clustering tool
Levenshtein: spell checking tool
Baseforms: baseform reduction and splitting compound nouns tool
Pretree: training tool for pretrees and classify tool
TE: terminology extraction tool
Pendulum: gazetteer bootstrapping tool (for Named Entity Recognition)
Namerec: Named Entity Recognition system
JLanI: language identification tool
Viterbitagger: POS tagging tool
Zipfel: tool for Zipf's law
AHC: agglomerative hierarchical clustering tool
Genetomorph: finding morphological structure with a genetic algorithm
Your Tool: template tool for your program

Version: 1.0
file format: zip
file size: 258MB
file link: ASV Toolbox.zip

ASV Toolbox - Documentation

Here you can download the complete documentation for the ASV Toolbox.

Version: 1.0
file format: zip
file size: 7.46MB
file link: ASV Toolbox_Docu.zip

ASV Toolbox - framework

The framework does NOT contain module but is needed to use a module. It contains the libraries and utilities of the ASV Toolbox.

Version: 1.0
file format: zip
file size: 197KB
file link: ASV Toolbox_framework.zip

ASV Toolbox - Chinese Whispers

This very efficient graph clustering algorithm  has been used for language separation, unsupervised POS tagging and word sense induction. Its application is not bound to language data; the program can partition arbitrary undirected, weighted graphs of arbitrary sizes. Best results are obtained on graphs with small world structure.

Version: 1.0
file format: zip
file size: 4.66MB
file link: ASV Toolbox_CW.zip
documentation: Documentation

ASV Toolbox - Levenshtein

Based on a Directed Acyclic Word Graph implementation, this tool allows efficient basic spell checking by offering words from a given word list with Levenshtein edit distances. As resources, we provide the top frequent 50.000 words for currently 15 languages. Training of custom word lists is possible. For example, the Italian wordlist returns for the misspelled input word “spagetti” the correct spelling “spaghetti” with distance 1 and offers “spetti”, “soggetti”, “panetti” and “paletti” with distance 2.

Version: 1.0
file format: zip
file size: 8.20MB
file link: ASV Toolbox_Levenshtein.zip
documentation: Documentation

ASV Toolbox - Baseforms

Version: 1.0
file format: zip
file size: 3.08MB
file link: ASV Toolbox_Baseforms.zip
documentation: Documentation

ASV Toolbox - Pretree

This implementation of Compact Patricia Tree classifiers proved to be useful for morphology-related tasks and is used by various other tools in ASV Toolbox. The tool provides possibilities to train and evaluate classifiers that use beginnings or endings of strings as features. An important property of this classifier is that it reproduces the training set classification to 100%. Therefore, pretrees are capable of storing an exception list, while generalizing on unseen examples.
 
Version: 1.0
file format: zip
file size: 22.3MB
file link: ASV Toolbox_Pretree.zip
documentation: Documentation

ASV Toolbox - TE

TE is the abbreviation for Terminolgy Extraction.

This tool extracts terminologically relevant terms and phrases from short documents by comparing them to a large background corpus. Currently, it is available for English, Finnish and German.

Version: 1.0
file format: zip
file size: 32MB
file link: ASV_Toolbox_TE_v1.0.zip (incl. language English)
file link: ASV_Toolbox_TE_v1.1.zip (incl. language English)
documentation: Documentation

language	download link	file size
German	ASV_Toolbox_TE_DE.zip	73MB
Finnish	ASV_Toolbox_TE_FI.zip	105MB

ASV Toolbox - Pendulum

For building gazetteers for generalized Named Entity Recognition, this tool provides a bootstrapping framework that grows small initial gazetteers using a set of rules and a customizable regular expression tagging. In the case of person names, this search-and verification methodology is able to extract e.g. some 40,000 names starting from a list of 20 with high precision from large plain text corpora.

Version: 1.0
file format: zip
file size: 2.83MB
file link: ASV Toolbox_Pendulum.zip
documentation: Documentation

ASV Toolbox - Namerec

Namerec is a gazetteer- and rule-based Namend Entity Recognition tool.

Here you can specify Named Entity Extraction rules that make heavy use of gazetteers (e.g. built by the Pendulum as described in the previous section). Extraction patterns are freely configurable; the tool marks plain text with NER markup. Resources are available for German for person names with professions; a sample markup is given here:

 <person pattern="TIT PU VN NN">Dr . Angela Merkel</person> hat <person pattern="VN NN">Gerhard Schröder</person> im Amt abgelöst .

Version: 1.0
file format: zip
file size: 2.65MB
file link: ASV Toolbox_Namerec.zip
documentation: Documentation

ASV Toolbox - JLanI

JLanI identifies the language of sentences. This state-of-the-art word-based language identification program allows identifying the language at sentence level. It can be used to identify foreign language inserts in a corpus. At the moment, 25 languages are supported; the number of languages is easily extendible by providing frequency lists.

Version: 1.0
file format: zip
file size: 2.59MB
file link: ASV Toolbox_JLanI.zip
documentation: Documentation

ASV Toolbox - Viterbitagger

Train, use and evaluate a simple POS tagger.

This simple tagger implementation is based on tag trigrams and tag distributions for words. Not as powerful as a full HMM implementation, it comes with a morphological back-off component (realized with Pretree) and is capable of training tagger models on very large annotated texts in flexible formats. Further, it allows tagging previously tagged text with a second (e.g. semantic) tag. The format of the tagger model is readable as plain text, which could prove useful for educational purposes. An evaluation framework is included that also deals with evaluating on different tag sets for Gold standard and test. Supervised models are provided for English, German and Finnish, unsupervised tagger models for resource-scarce languages or domain-specific applications are available at http://wortschatz.uni-leipzig.de/~cbiemann/software/unsupos.html
 
Version: 1.0
file format: zip
file size: 4.82MB
file link: ASV Toolbox_Viterbitagger.zip
documentation: Documentation

tagger model	download link	file size
German	ASV Toolbox_VT_DE.zip	327KB
English	ASV Toolbox_VT_EN.zip	2.54MB
Finnish	ASV Toolbox_VT_FI.zip	71KB

ASV Toolbox - Zipfel

For an introduction to quantitative linguistics, this tool enables to visualize Zipfian distributions for word frequency lists and documents. Various parameters are computed, rank-frequency lists are browseable and the plot can be exported in various formats.

Version: 1.0
file format: zip
file size: 1.11MB
file link: ASV Toolbox_Zipfel.zip
documentation: Documentation

ASV Toolbox - HAC

This basic implementation of HA-clustering allows clustering elements represented as vectors with various norms and distance measures. As example configuration, words can be clustered by their common significant co-occurrences (available from LCC in 15 languages). The result can be exported as XML file or in dendrogram picture format.

Version: 1.0
file format: zip
file size: 1.25MB
file link: ASV Toolbox_HAC.zip
documentation: Documentation

ASV Toolbox - Genetomorph

A Genetic Algorithm is used to detect morphological regularities in word lists. A fitness function that minimizes the cost of describing morphological rules is optimized, individual solutions can be browsed and the progress until convergence is visualized in a plot. Sample data is available for German nouns and adjectives.

Version: 1.0
file format: zip
file size: 4.42MB
file link: ASV Toolbox_GenetoMorph.zip
documentation: Documentation

ASV Toolbox - Your Tool

Template tool to integrate your own programm as module in the ASV Toolbox.

Version: 1.0
file format: zip
file size: 626KB
file link: ASV Toolbox_YT.zip
documentation: Documentation