👄 The most accurate natural language detection library in the Go ecosystem, suitable for long and short text alike

Overview

lingua


Build Status codecov supported languages Go Reference Go Report Card license

Table of Contents

  1. What does this library do?
  2. Why does this library exist?
  3. Which languages are supported?
  4. How good is it?
  5. Why is it better than other libraries?
  6. Test report generation
  7. How to add it to your project?
  8. How to build?
  9. How to use?
  10. What's next for version 1.1.0?
  11. Contributions

1. What does this library do? Top ▲

Its task is simple: It tells you which language some provided textual data is written in. This is very useful as a preprocessing step for linguistic data in natural language processing applications such as text classification and spell checking. Other use cases, for instance, might include routing e-mails to the right geographically located customer service department, based on the e-mails' languages.

2. Why does this library exist? Top ▲

Language detection is often done as part of large machine learning frameworks or natural language processing applications. In cases where you don't need the full-fledged functionality of those systems or don't want to learn the ropes of those, a small flexible library comes in handy.

So far, the only other comprehensive open source library in the Go ecosystem for this task is Whatlanggo. Unfortunately, it has two major drawbacks:

  1. Detection only works with quite lengthy text fragments. For very short text snippets such as Twitter messages, it does not provide adequate results.
  2. The more languages take part in the decision process, the less accurate are the detection results.

Lingua aims at eliminating these problems. It nearly does not need any configuration and yields pretty accurate results on both long and short text, even on single words and phrases. It draws on both rule-based and statistical methods but does not use any dictionaries of words. It does not need a connection to any external API or service either. Once the library has been downloaded, it can be used completely offline.

3. Which languages are supported? Top ▲

Compared to other language detection libraries, Lingua's focus is on quality over quantity, that is, getting detection right for a small set of languages first before adding new ones. Currently, the following 75 languages are supported:

  • A
    • Afrikaans
    • Albanian
    • Arabic
    • Armenian
    • Azerbaijani
  • B
    • Basque
    • Belarusian
    • Bengali
    • Norwegian Bokmal
    • Bosnian
    • Bulgarian
  • C
    • Catalan
    • Chinese
    • Croatian
    • Czech
  • D
    • Danish
    • Dutch
  • E
    • English
    • Esperanto
    • Estonian
  • F
    • Finnish
    • French
  • G
    • Ganda
    • Georgian
    • German
    • Greek
    • Gujarati
  • H
    • Hebrew
    • Hindi
    • Hungarian
  • I
    • Icelandic
    • Indonesian
    • Irish
    • Italian
  • J
    • Japanese
  • K
    • Kazakh
    • Korean
  • L
    • Latin
    • Latvian
    • Lithuanian
  • M
    • Macedonian
    • Malay
    • Maori
    • Marathi
    • Mongolian
  • N
    • Norwegian Nynorsk
  • P
    • Persian
    • Polish
    • Portuguese
    • Punjabi
  • R
    • Romanian
    • Russian
  • S
    • Serbian
    • Shona
    • Slovak
    • Slovene
    • Somali
    • Sotho
    • Spanish
    • Swahili
    • Swedish
  • T
    • Tagalog
    • Tamil
    • Telugu
    • Thai
    • Tsonga
    • Tswana
    • Turkish
  • U
    • Ukrainian
    • Urdu
  • V
    • Vietnamese
  • W
    • Welsh
  • X
    • Xhosa
  • Y
    • Yoruba
  • Z
    • Zulu

4. How good is it? Top ▲

Lingua is able to report accuracy statistics for some bundled test data available for each supported language. The test data for each language is split into three parts:

  1. a list of single words with a minimum length of 5 characters
  2. a list of word pairs with a minimum length of 10 characters
  3. a list of complete grammatical sentences of various lengths

Both the language models and the test data have been created from separate documents of the Wortschatz corpora offered by Leipzig University, Germany. Data crawled from various news websites have been used for training, each corpus comprising one million sentences. For testing, corpora made of arbitrarily chosen websites have been used, each comprising ten thousand sentences. From each test corpus, a random unsorted subset of 1000 single words, 1000 word pairs and 1000 sentences has been extracted, respectively.

Given the generated test data, I have compared the detection results of Lingua and Whatlanggo running over the data of Lingua's supported 75 languages. Additionally, I have added Google's CLD3 to the comparison with the help of the gocld3 bindings. Languages that are not supported by CLD3 or Whatlanggo are simply ignored during the detection process.

The box plot below shows the distribution of the averaged accuracy values for all three performed tasks: Single word detection, word pair detection and sentence detection. Lingua clearly outperforms its contenders. Bar plots for each language and further box plots for the separate detection tasks can be found in the file ACCURACY_PLOTS.md. Detailed statistics including mean, median and standard deviation values for each language and classifier are available in the file ACCURACY_TABLE.md.

Average Detection Performance

5. Why is it better than other libraries? Top ▲

Every language detector uses a probabilistic n-gram model trained on the character distribution in some training corpus. Most libraries only use n-grams of size 3 (trigrams) which is satisfactory for detecting the language of longer text fragments consisting of multiple sentences. For short phrases or single words, however, trigrams are not enough. The shorter the input text is, the less n-grams are available. The probabilities estimated from such few n-grams are not reliable. This is why Lingua makes use of n-grams of sizes 1 up to 5 which results in much more accurate prediction of the correct language.

A second important difference is that Lingua does not only use such a statistical model, but also a rule-based engine. This engine first determines the alphabet of the input text and searches for characters which are unique in one or more languages. If exactly one language can be reliably chosen this way, the statistical model is not necessary anymore. In any case, the rule-based engine filters out languages that do not satisfy the conditions of the input text. Only then, in a second step, the probabilistic n-gram model is taken into consideration. This makes sense because loading less language models means less memory consumption and better runtime performance.

In general, it is always a good idea to restrict the set of languages to be considered in the classification process using the respective api methods. If you know beforehand that certain languages are never to occur in an input text, do not let those take part in the classifcation process. The filtering mechanism of the rule-based engine is quite good, however, filtering based on your own knowledge of the input text is always preferable.

6. Test report generation Top ▲

If you want to reproduce the accuracy results above, you can generate the test reports yourself for both classifiers and all languages by doing:

go run cmd/accuracy_reporter.go

For each detector and language, a test report file is then written into /accuracy-reports. As an example, here is the current output of the Lingua German report:

##### German #####

>>> Accuracy on average: 89.13%

>> Detection of 1000 single words (average length: 9 chars)
Accuracy: 73.90%
Erroneously classified as Dutch: 2.30%, Danish: 2.10%, English: 2.00%, Latin: 1.90%, Bokmal: 1.60%, Basque: 1.20%, French: 1.20%, Italian: 1.20%, Esperanto: 1.10%, Swedish: 1.00%, Afrikaans: 0.80%, Tsonga: 0.70%, Nynorsk: 0.60%, Portuguese: 0.60%, Yoruba: 0.60%, Finnish: 0.50%, Sotho: 0.50%, Welsh: 0.50%, Estonian: 0.40%, Irish: 0.40%, Polish: 0.40%, Spanish: 0.40%, Swahili: 0.40%, Tswana: 0.40%, Bosnian: 0.30%, Icelandic: 0.30%, Tagalog: 0.30%, Albanian: 0.20%, Catalan: 0.20%, Croatian: 0.20%, Indonesian: 0.20%, Lithuanian: 0.20%, Maori: 0.20%, Romanian: 0.20%, Xhosa: 0.20%, Zulu: 0.20%, Latvian: 0.10%, Malay: 0.10%, Slovak: 0.10%, Slovene: 0.10%, Somali: 0.10%, Turkish: 0.10%

>> Detection of 1000 word pairs (average length: 18 chars)
Accuracy: 93.80%
Erroneously classified as Dutch: 0.90%, Latin: 0.80%, English: 0.70%, Swedish: 0.60%, Danish: 0.50%, French: 0.40%, Bokmal: 0.30%, Irish: 0.20%, Tagalog: 0.20%, Tsonga: 0.20%, Afrikaans: 0.10%, Esperanto: 0.10%, Estonian: 0.10%, Finnish: 0.10%, Italian: 0.10%, Maori: 0.10%, Nynorsk: 0.10%, Portuguese: 0.10%, Somali: 0.10%, Swahili: 0.10%, Turkish: 0.10%, Welsh: 0.10%, Xhosa: 0.10%, Zulu: 0.10%

>> Detection of 1000 sentences (average length: 111 chars)
Accuracy: 99.70%
Erroneously classified as Dutch: 0.20%, Latin: 0.10%

7. How to add it to your project? Top ▲

go get github.com/pemistahl/[email protected]

8. How to build? Top ▲

Lingua requires Go version 1.16.

git clone https://github.com/pemistahl/lingua-go.git
cd lingua-go
go build

The source code is accompanied by an extensive unit test suite. To run them, simply say:

go test

9. How to use? Top ▲

9.1 Basic usage

package main

import (
    "fmt"
    "github.com/pemistahl/lingua-go"
)

func main() {
    languages := []lingua.Language{
        lingua.English,
        lingua.French,
        lingua.German,
        lingua.Spanish,
    }

    detector := lingua.NewLanguageDetectorBuilder().
        FromLanguages(languages...).
        Build()

    if language, exists := detector.DetectLanguageOf("languages are awesome"); exists {
        fmt.Println(language)
    }

    // Output: English
}

9.2 Minimum relative distance

By default, Lingua returns the most likely language for a given input text. However, there are certain words that are spelled the same in more than one language. The word prologue, for instance, is both a valid English and French word. Lingua would output either English or French which might be wrong in the given context. For cases like that, it is possible to specify a minimum relative distance that the logarithmized and summed up probabilities for each possible language have to satisfy. It can be stated in the following way:

package main

import (
    "fmt"
    "github.com/pemistahl/lingua-go"
)

func main() {
    languages := []lingua.Language{
        lingua.English,
        lingua.French,
        lingua.German,
        lingua.Spanish,
    }

    detector := lingua.NewLanguageDetectorBuilder().
        FromLanguages(languages...).
        WithMinimumRelativeDistance(0.25).
        Build()

    language, exists := detector.DetectLanguageOf("languages are awesome")

    fmt.Println(language)
    fmt.Println(exists)

    // Output:
    // Unknown
    // false
}

Be aware that the distance between the language probabilities is dependent on the length of the input text. The longer the input text, the larger the distance between the languages. So if you want to classify very short text phrases, do not set the minimum relative distance too high. Otherwise Unknown will be returned most of the time as in the example above. This is the return value for cases where language detection is not reliably possible.

9.3 Confidence values

Knowing about the most likely language is nice but how reliable is the computed likelihood? And how less likely are the other examined languages in comparison to the most likely one? These questions can be answered as well:

package main

import (
    "fmt"
    "github.com/pemistahl/lingua-go"
)

func main() {
    languages := []lingua.Language{
        lingua.English,
        lingua.French,
        lingua.German,
        lingua.Spanish,
    }

    detector := lingua.NewLanguageDetectorBuilder().
        FromLanguages(languages...).
        Build()

    confidenceValues := detector.ComputeLanguageConfidenceValues("languages are awesome")

    for _, elem := range confidenceValues {
        fmt.Printf("%s: %.2f\n", elem.Language(), elem.Value())
    }

    // Output:
    // English: 1.00
    // French: 0.79
    // German: 0.75
    // Spanish: 0.72
}

In the example above, a slice of ConfidenceValue is returned containing all possible languages sorted by their confidence value in descending order. The values that this method computes are part of a relative confidence metric, not of an absolute one. Each value is a number between 0.0 and 1.0. The most likely language is always returned with value 1.0. All other languages get values assigned which are lower than 1.0, denoting how less likely those languages are in comparison to the most likely language.

The slice returned by this method does not necessarily contain all languages which the calling instance of LanguageDetector was built from. If the rule-based engine decides that a specific language is truly impossible, then it will not be part of the returned slice. Likewise, if no ngram probabilities can be found within the detector's languages for the given input text, the returned slice will be empty. The confidence value for each language not being part of the returned slice is assumed to be 0.0.

9.4 Eager loading versus lazy loading

By default, Lingua uses lazy-loading to load only those language models on demand which are considered relevant by the rule-based filter engine. For web services, for instance, it is rather beneficial to preload all language models into memory to avoid unexpected latency while waiting for the service response. If you want to enable the eager-loading mode, you can do it like this:

lingua.NewLanguageDetectorBuilder().
    FromAllLanguages().
    WithPreloadedLanguageModels().
    Build()

Multiple instances of LanguageDetector share the same language models in memory which are accessed asynchronously by the instances.

9.5 Methods to build the LanguageDetector

There might be classification tasks where you know beforehand that your language data is definitely not written in Latin, for instance (what a surprise :-). The detection accuracy can become better in such cases if you exclude certain languages from the decision process or just explicitly include relevant languages:

// Including all languages available in the library
// consumes at least 2GB of memory and might
// lead to slow runtime performance.
lingua.NewLanguageDetectorBuilder().FromAllLanguages()

// Include only languages that are not yet extinct
// (= currently excludes Latin).
lingua.NewLanguageDetectorBuilder().FromAllSpokenLanguages()

// Include only languages written with Cyrillic script.
lingua.NewLanguageDetectorBuilder().FromAllLanguagesWithCyrillicScript()

// Exclude only the Spanish language from the decision algorithm.
lingua.NewLanguageDetectorBuilder().FromAllLanguagesWithout(lingua.Spanish)

// Only decide between English and German.
lingua.NewLanguageDetectorBuilder().FromLanguages(lingua.English, lingua.German)

// Select languages by ISO 639-1 code.
lingua.NewLanguageDetectorBuilder().FromIsoCodes639_1(lingua.EN, lingua.DE)

// Select languages by ISO 639-3 code.
lingua.NewLanguageDetectorBuilder().FromIsoCodes639_3(lingua.ENG, lingua.DEU)

10. What's next for version 1.1.0? Top ▲

Take a look at the planned issues.

11. Contributions Top ▲

In case you want to contribute something to Lingua, please take a look at the file CONTRIBUTING.md.

Comments
  • Panics at loadJson

    Panics at loadJson

    Code to reproduce:

    package main
    
    import (
        "fmt"
        "github.com/pemistahl/lingua-go"
    )
    
    func main() {
        languages := []lingua.Language{
            lingua.English,
            lingua.French,
            lingua.German,
            lingua.Spanish,
        }
    
        detector := lingua.NewLanguageDetectorBuilder().
            FromLanguages(languages...).
            Build()
    
        confidenceValues := detector.ComputeLanguageConfidenceValues("languages are awesome")
    
        for _, elem := range confidenceValues {
            fmt.Printf("%s: %.2f\n", elem.Language(), elem.Value())
        }
    
        // Output:
        // English: 1.00
        // French: 0.79
        // German: 0.75
        // Spanish: 0.72
    }
    

    go.mod

    module lingua
    
    go 1.16
    
    require github.com/pemistahl/lingua-go v1.0.0
    

    go env:

    ❯ go env
    GO111MODULE="on"
    GOARCH="amd64"
    GOBIN=""
    GOCACHE="/Users/dmitriysmotrov/Library/Caches/go-build"
    GOENV="/Users/dmitriysmotrov/Library/Application Support/go/env"
    GOEXE=""
    GOFLAGS=""
    GOHOSTARCH="amd64"
    GOHOSTOS="darwin"
    GOINSECURE=""
    GOMODCACHE="/Users/dmitriysmotrov/.gvm/pkgsets/go1.16.5/global/pkg/mod"
    GONOPROXY=""
    GONOSUMDB=""
    GOOS="darwin"
    GOPATH="/Users/dmitriysmotrov/.gvm/pkgsets/go1.16.5/global"
    GOPRIVATE=""
    GOPROXY="https://proxy.golang.org,direct"
    GOROOT="/Users/dmitriysmotrov/.gvm/gos/go1.16.5"
    GOSUMDB="sum.golang.org"
    GOTMPDIR=""
    GOTOOLDIR="/Users/dmitriysmotrov/.gvm/gos/go1.16.5/pkg/tool/darwin_amd64"
    GOVCS=""
    GOVERSION="go1.16.5"
    GCCGO="gccgo"
    AR="ar"
    CC="clang"
    CXX="clang++"
    CGO_ENABLED="1"
    GOMOD="/Users/dmitriysmotrov/space/dsxack/lingua/go.mod"
    CGO_CFLAGS="-g -O2"
    CGO_CPPFLAGS=""
    CGO_CXXFLAGS="-g -O2"
    CGO_FFLAGS="-g -O2"
    CGO_LDFLAGS="-g -O2"
    PKG_CONFIG="pkg-config"
    GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/z5/8ts06jv92yjc5sp5mdsdzr2h0000gn/T/go-build2817996487=/tmp/go-build -gno-record-gcc-switches -fno-common"
    

    Expect: no panics

    Actual:

    panic: runtime error: invalid memory address or nil pointer dereference
    	panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x10e9c82]
    
    goroutine 22 [running]:
    archive/zip.(*ReadCloser).Close(0x0, 0x0, 0x0)
    	/Users/dmitriysmotrov/.gvm/gos/go1.16.5/src/archive/zip/reader.go:161 +0x22
    panic(0x11841e0, 0x12c0160)
    	/Users/dmitriysmotrov/.gvm/gos/go1.16.5/src/runtime/panic.go:965 +0x1b9
    github.com/pemistahl/lingua-go.loadJson(0x18, 0x5, 0x0, 0x0, 0x0)
    	/Users/dmitriysmotrov/.gvm/pkgsets/go1.16rc1/global/pkg/mod/github.com/pemistahl/[email protected]/json.go:32 +0x18e
    github.com/pemistahl/lingua-go.loadFivegrams(...)
    	/Users/dmitriysmotrov/.gvm/pkgsets/go1.16rc1/global/pkg/mod/github.com/pemistahl/[email protected]/fivegrams.go:925
    github.com/pemistahl/lingua-go.germanFivegramModel.func1.1()
    	/Users/dmitriysmotrov/.gvm/pkgsets/go1.16rc1/global/pkg/mod/github.com/pemistahl/[email protected]/fivegrams.go:368 +0x45
    sync.(*Once).doSlow(0xc0000b0f30, 0xc000094b68)
    	/Users/dmitriysmotrov/.gvm/gos/go1.16.5/src/sync/once.go:68 +0xec
    sync.(*Once).Do(...)
    	/Users/dmitriysmotrov/.gvm/gos/go1.16.5/src/sync/once.go:59
    github.com/pemistahl/lingua-go.germanFivegramModel.func1(0x11831c0, 0xc00009afc0)
    	/Users/dmitriysmotrov/.gvm/pkgsets/go1.16rc1/global/pkg/mod/github.com/pemistahl/[email protected]/fivegrams.go:367 +0xbb
    github.com/pemistahl/lingua-go.languageDetector.lookUpNgramProbability(0xc0000d49c0, 0x4, 0x4, 0x0, 0xc00012c080, 0x2, 0x2, 0xc00009b0b0, 0xc00009b050, 0xc00009af90, ...)
    	/Users/dmitriysmotrov/.gvm/pkgsets/go1.16rc1/global/pkg/mod/github.com/pemistahl/[email protected]/detector.go:530 +0x1cb
    github.com/pemistahl/lingua-go.languageDetector.computeSumOfNgramProbabilities(0xc0000d49c0, 0x4, 0x4, 0x0, 0xc00012c080, 0x2, 0x2, 0xc00009b0b0, 0xc00009b050, 0xc00009af90, ...)
    	/Users/dmitriysmotrov/.gvm/pkgsets/go1.16rc1/global/pkg/mod/github.com/pemistahl/[email protected]/detector.go:516 +0xf7
    github.com/pemistahl/lingua-go.languageDetector.computeLanguageProbabilities(0xc0000d49c0, 0x4, 0x4, 0x0, 0xc00012c080, 0x2, 0x2, 0xc00009b0b0, 0xc00009b050, 0xc00009af90, ...)
    	/Users/dmitriysmotrov/.gvm/pkgsets/go1.16rc1/global/pkg/mod/github.com/pemistahl/[email protected]/detector.go:474 +0xca
    github.com/pemistahl/lingua-go.languageDetector.lookUpLanguageModels(0xc0000d49c0, 0x4, 0x4, 0x0, 0xc00012c080, 0x2, 0x2, 0xc00009b0b0, 0xc00009b050, 0xc00009af90, ...)
    	/Users/dmitriysmotrov/.gvm/pkgsets/go1.16rc1/global/pkg/mod/github.com/pemistahl/[email protected]/detector.go:442 +0xca
    created by github.com/pemistahl/lingua-go.languageDetector.ComputeLanguageConfidenceValues
    	/Users/dmitriysmotrov/.gvm/pkgsets/go1.16rc1/global/pkg/mod/github.com/pemistahl/[email protected]/detector.go:170 +0x525
    panic: runtime error: invalid memory address or nil pointer dereference
    	panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x10e9c82]
    
    goroutine 18 [running]:
    archive/zip.(*ReadCloser).Close(0x0, 0x0, 0x0)
    	/Users/dmitriysmotrov/.gvm/gos/go1.16.5/src/archive/zip/reader.go:161 +0x22
    panic(0x11841e0, 0x12c0160)
    	/Users/dmitriysmotrov/.gvm/gos/go1.16.5/src/runtime/panic.go:965 +0x1b9
    github.com/pemistahl/lingua-go.loadJson(0x18, 0x1, 0x0, 0x0, 0x0)
    	/Users/dmitriysmotrov/.gvm/pkgsets/go1.16rc1/global/pkg/mod/github.com/pemistahl/[email protected]/json.go:32 +0x18e
    github.com/pemistahl/lingua-go.loadUnigrams(...)
    	/Users/dmitriysmotrov/.gvm/pkgsets/go1.16rc1/global/pkg/mod/github.com/pemistahl/[email protected]/unigrams.go:925
    github.com/pemistahl/lingua-go.germanUnigramModel.func1.1()
    	/Users/dmitriysmotrov/.gvm/pkgsets/go1.16rc1/global/pkg/mod/github.com/pemistahl/[email protected]/unigrams.go:368 +0x45
    sync.(*Once).doSlow(0xc0000b1d40, 0xc000064b68)
    	/Users/dmitriysmotrov/.gvm/gos/go1.16.5/src/sync/once.go:68 +0xec
    sync.(*Once).Do(...)
    	/Users/dmitriysmotrov/.gvm/gos/go1.16.5/src/sync/once.go:59
    github.com/pemistahl/lingua-go.germanUnigramModel.func1(0x11831c0, 0xc00009b050)
    	/Users/dmitriysmotrov/.gvm/pkgsets/go1.16rc1/global/pkg/mod/github.com/pemistahl/[email protected]/unigrams.go:367 +0xbb
    github.com/pemistahl/lingua-go.languageDetector.lookUpNgramProbability(0xc0000d49c0, 0x4, 0x4, 0x0, 0xc00012c080, 0x2, 0x2, 0xc00009b0b0, 0xc00009b050, 0xc00009af90, ...)
    	/Users/dmitriysmotrov/.gvm/pkgsets/go1.16rc1/global/pkg/mod/github.com/pemistahl/[email protected]/detector.go:538 +0x128
    github.com/pemistahl/lingua-go.languageDetector.computeSumOfNgramProbabilities(0xc0000d49c0, 0x4, 0x4, 0x0, 0xc00012c080, 0x2, 0x2, 0xc00009b0b0, 0xc00009b050, 0xc00009af90, ...)
    	/Users/dmitriysmotrov/.gvm/pkgsets/go1.16rc1/global/pkg/mod/github.com/pemistahl/[email protected]/detector.go:516 +0xf7
    github.com/pemistahl/lingua-go.languageDetector.computeLanguageProbabilities(0xc0000d49c0, 0x4, 0x4, 0x0, 0xc00012c080, 0x2, 0x2, 0xc00009b0b0, 0xc00009b050, 0xc00009af90, ...)
    	/Users/dmitriysmotrov/.gvm/pkgsets/go1.16rc1/global/pkg/mod/github.com/pemistahl/[email protected]/detector.go:474 +0xca
    github.com/pemistahl/lingua-go.languageDetector.lookUpLanguageModels(0xc0000d49c0, 0x4, 0x4, 0x0, 0xc00012c080, 0x2, 0x2, 0xc00009b0b0, 0xc00009b050, 0xc00009af90, ...)
    	/Users/dmitriysmotrov/.gvm/pkgsets/go1.16rc1/global/pkg/mod/github.com/pemistahl/[email protected]/detector.go:442 +0xca
    created by github.com/pemistahl/lingua-go.languageDetector.ComputeLanguageConfidenceValues
    	/Users/dmitriysmotrov/.gvm/pkgsets/go1.16rc1/global/pkg/mod/github.com/pemistahl/[email protected]/detector.go:170 +0x525
    
    opened by dsxack 7
  • `lingua.Unknown` is not handled appropriately if included in the set of input languages

    `lingua.Unknown` is not handled appropriately if included in the set of input languages

    I have started testing the library a few days ago and just saw a first nil pointer panic like this:

    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x4314fd8]
    
    goroutine 22414 [running]:
    github.com/pemistahl/lingua-go.loadJson(0xa834340, 0x45537a0)
    	/Users/marian/go/pkg/mod/github.com/pemistahl/[email protected]/json.go:37 +0x178
    github.com/pemistahl/lingua-go.languageDetector.loadLanguageModels({{0xc01622a700, 0x1f, 0x1f}, 0x0, {0xc017283f80, 0xc, 0x10}, 0xc01312f320, 0xa834340, 0xa834240, ...}, ...)
    	/Users/marian/go/pkg/mod/github.com/pemistahl/[email protected]/detector.go:612 +0x8d
    github.com/pemistahl/lingua-go.languageDetector.lookUpNgramProbability({{0xc01622a700, 0x1f, 0x1f}, 0x0, {0xc017283f80, 0xc, 0x10}, 0xc01312f320, 0xa834340, 0xa834240, ...}, ...)
    	/Users/marian/go/pkg/mod/github.com/pemistahl/[email protected]/detector.go:552 +0x146
    github.com/pemistahl/lingua-go.languageDetector.computeSumOfNgramProbabilities({{0xc01622a700, 0x1f, 0x1f}, 0x0, {0xc017283f80, 0xc, 0x10}, 0xc01312f320, 0xa834340, 0xa834240, ...}, ...)
    	/Users/marian/go/pkg/mod/github.com/pemistahl/[email protected]/detector.go:526 +0x145
    github.com/pemistahl/lingua-go.languageDetector.computeLanguageProbabilities({{0xc01622a700, 0x1f, 0x1f}, 0x0, {0xc017283f80, 0xc, 0x10}, 0xc01312f320, 0xa834340, 0xa834240, ...}, ...)
    	/Users/marian/go/pkg/mod/github.com/pemistahl/[email protected]/detector.go:484 +0xcb
    github.com/pemistahl/lingua-go.languageDetector.lookUpLanguageModels({{0xc01622a700, 0x1f, 0x1f}, 0x0, {0xc017283f80, 0xc, 0x10}, 0xc01312f320, 0xa834340, 0xa834240, ...}, ...)
    	/Users/marian/go/pkg/mod/github.com/pemistahl/[email protected]/detector.go:452 +0xb7
    created by github.com/pemistahl/lingua-go.languageDetector.ComputeLanguageConfidenceValues
    	/Users/marian/go/pkg/mod/github.com/pemistahl/[email protected]/detector.go:176 +0x455
    

    Seems as if even reading from an embed File can fail at some point.

    I'm using go version go1.17.2 darwin/amd64.

    bug 
    opened by marians 5
  • Add absolute confidence metric

    Add absolute confidence metric

    I do see that the README has this example:

    package main
    
    import (
        "fmt"
        "github.com/pemistahl/lingua-go"
    )
    
    func main() {
        languages := []lingua.Language{
            lingua.English,
            lingua.French,
            lingua.German,
            lingua.Spanish,
        }
    
        detector := lingua.NewLanguageDetectorBuilder().
            FromLanguages(languages...).
            Build()
    
        confidenceValues := detector.ComputeLanguageConfidenceValues("languages are awesome")
    
        for _, elem := range confidenceValues {
            fmt.Printf("%s: %.2f\n", elem.Language(), elem.Value())
        }
    
        // Output:
        // English: 1.00
        // French: 0.79
        // German: 0.75
        // Spanish: 0.72
    }
    

    But if I do detector.ComputeLanguageConfidenceValues("yo bebo ein large quantity of tasty leche"), English is still going to result in 1.0. How do I get something like a certainty / probability that the text is English? Because 1.0 doesn't seem so helpful in that case. It might just be my lack of math experience, I'm assuming this is possible with the values above in the example, but I don't exactly see how.

    new feature 
    opened by BenStigsen 4
  • Put resources into compiled binary

    Put resources into compiled binary

    Resources not be available from compiled binary and this does not allow using the module without put language-models near compiled binary.

    It fixes https://github.com/pemistahl/lingua-go/issues/2

    opened by dsxack 3
  • Detection results differ between JVM and Go edition of Lingua

    Detection results differ between JVM and Go edition of Lingua

    For some text samples, Lingua-go detected the wrong language, but Lingua's results were correct.

    Examples:

    texts | lingua-go|lingua -- | -- | --   Chắc tại có +18:)?| Malay  | VIETNAMESE

    bug 
    opened by hwdlei 2
  • Reduce

    Reduce "bloat"

    Hi,

    Thanks for the excellent work first and foremost, but may I suggest keeping metadata (e.g. https://github.com/pemistahl/lingua-go/commit/61c7054b215d4252f65ae35959b8587acab9927a) separately, outside this repository. You could create another repo, e.g. github.com/pemistahl/lingua-go-accuracy-reports or similar.

    The comparisons are useful but also currently bloat the repository, plus they introduce quite a few extra dependencies, i.e. https://github.com/pemistahl/lingua-go/blob/main/go.sum.

    What do you think?

    enhancement 
    opened by dim 2
  • Language detection is sometimes non-deterministic

    Language detection is sometimes non-deterministic

    To reproduce the issue:

    package main
    
    import (
    	"github.com/pemistahl/lingua-go"
    	"log"
    )
    
    func main() {
    	detectorAll := lingua.NewLanguageDetectorBuilder().FromAllLanguages().WithPreloadedLanguageModels().Build()
    	for i := 0; i < 1000 ; i++ {
    		lang,_:=detectorAll.DetectLanguageOf("Az elmúlt hétvégén 12-re emelkedett az elhunyt koronavírus-fertőzöttek száma Szlovákiában. Mindegyik szociális otthon dolgozóját letesztelik, Matovič szerint az ingázóknak még várniuk kellene a teszteléssel")
    		log.Println(lang.IsoCode639_1().String())
    
    	}
    }
    

    Thank you for amazing work!

    bug 
    opened by FilipAlexander 2
  • Strange matching for Spanish phrase detected as Finnish

    Strange matching for Spanish phrase detected as Finnish

    Hey! I've been messing with this library, most of it seems great! There is one issue I've ran into with a spanish phrase being detected as Finnish, as it has a confidence level of 1, I'm unsure if this is intended.

    Phrase: ¿les gustan los pokemon?

    With the following code:

    package main
    
    import (
    	"log"
    
    	"github.com/pemistahl/lingua-go"
    )
    
    func main() {
    	detector := lingua.
    		NewLanguageDetectorBuilder().
    		FromAllSpokenLanguages().
    		WithPreloadedLanguageModels().
    		Build()
    
    	content := "¿les gustan los pokemon?"
    	lang, reliable := detector.DetectLanguageOf(content)
    	log.Println(lang.String(), reliable)
    
    	log.Println(" --- ")
    
    	confidences := detector.ComputeLanguageConfidenceValues(content)
    	for _, langConf := range confidences {
    		log.Println(langConf.Language().String(), langConf.Value())
    	}
    }
    

    The following output is produced:

    2022/04/20 00:22:43 Finnish true
    2022/04/20 00:22:43  --- 
    2022/04/20 00:22:43 Finnish 1
    2022/04/20 00:22:43 English 0.9883978684270469
    2022/04/20 00:22:43 Indonesian 0.978563900119626
    2022/04/20 00:22:43 Spanish 0.9747851212151981
    2022/04/20 00:22:43 Croatian 0.9724182360849759
    2022/04/20 00:22:43 Lithuanian 0.9647225277871057
    2022/04/20 00:22:43 Estonian 0.9641581778214242
    2022/04/20 00:22:43 Esperanto 0.9606587809451471
    2022/04/20 00:22:43 Polish 0.9594230676987932
    2022/04/20 00:22:43 Slovene 0.9546050214213473
    2022/04/20 00:22:43 Malay 0.9541465232681227
    2022/04/20 00:22:43 Albanian 0.9524198444722406
    2022/04/20 00:22:43 Italian 0.9486618781887298
    2022/04/20 00:22:43 Catalan 0.946963416607054
    2022/04/20 00:22:43 Danish 0.9403916449998727
    2022/04/20 00:22:43 Bosnian 0.9269675882527444
    2022/04/20 00:22:43 Portuguese 0.9261989417434195
    2022/04/20 00:22:43 German 0.919921338933763
    2022/04/20 00:22:43 Sotho 0.9152876229202939
    2022/04/20 00:22:43 Dutch 0.9145928120132025
    2022/04/20 00:22:43 French 0.9140644855054184
    2022/04/20 00:22:43 Slovak 0.9125324543349711
    2022/04/20 00:22:43 Latvian 0.9119548274103094
    2022/04/20 00:22:43 Tswana 0.9030296447404719
    2022/04/20 00:22:43 Romanian 0.8980252449808623
    2022/04/20 00:22:43 Nynorsk 0.8962667914904449
    2022/04/20 00:22:43 Tagalog 0.8961041054613276
    2022/04/20 00:22:43 Swedish 0.8861739698250194
    2022/04/20 00:22:43 Hungarian 0.8860583424196719
    2022/04/20 00:22:43 Bokmal 0.8860501842325473
    2022/04/20 00:22:43 Swahili 0.8855438630695021
    2022/04/20 00:22:43 Czech 0.877987508198549
    2022/04/20 00:22:43 Welsh 0.8706583132077192
    2022/04/20 00:22:43 Turkish 0.8635506224236865
    2022/04/20 00:22:43 Yoruba 0.8618678522282041
    2022/04/20 00:22:43 Basque 0.8587542505212317
    2022/04/20 00:22:43 Afrikaans 0.8435800177987139
    2022/04/20 00:22:43 Maori 0.8429171795365868
    2022/04/20 00:22:43 Ganda 0.8407646218672701
    2022/04/20 00:22:43 Icelandic 0.8248853640378799
    2022/04/20 00:22:43 Tsonga 0.8245248538291974
    2022/04/20 00:22:43 Irish 0.817982923494266
    2022/04/20 00:22:43 Zulu 0.8175325635441859
    2022/04/20 00:22:43 Shona 0.8008811823165958
    2022/04/20 00:22:43 Xhosa 0.7829601259301775
    2022/04/20 00:22:43 Vietnamese 0.774240344355879
    2022/04/20 00:22:43 Azerbaijani 0.7541427903961347
    2022/04/20 00:22:43 Somali 0.7538078988192347
    

    I'm not sure why it ranked Spanish as 4th. Is there a good method to get around this? Unfortunately given my use case I need to detect from a wide range of languages like this.

    This library is overall awesome, I'm using the latest stable release, thank you for this!

    opened by aidenwallis 1
  • build to wasm, run slowly.

    build to wasm, run slowly.

    I build the simple example to detect the word 'app', it work. but when I build it to wasm.it cost about 6 second to done.Is there any way to solve this problem?

    opened by Kobshobe 1
  • Find more efficient serialization format for language models

    Find more efficient serialization format for language models

    Currently, the language models are stored in json files. Perhaps it is possible to store them in some kind of binary format which can be loaded faster than the json files.

    One promising candidate could be protocol buffers.

    enhancement 
    opened by pemistahl 0
  • Add low accuracy mode

    Add low accuracy mode

    Lingua's high detection accuracy comes at the cost of being noticeably slower than other language detectors. The large language models also consume significant amounts of memory. These requirements might not be feasible for systems running low on resources.

    For users who want to classify mostly long texts or need to save resources, a so-called low accuracy mode will be implemented that loads only a small subset of the language models into memory. The API will be as follows:

    lingua.NewLanguageDetectorBuilder().FromAllLanguages().WithLowAccuracyMode().Build()
    

    The downside of this approach is that detection accuracy for short texts consisting of less than 120 characters will drop significantly. However, detection accuracy for texts which are longer than 120 characters will remain mostly unaffected.

    new feature 
    opened by pemistahl 0
  • Find more memory-efficient data structure for language models

    Find more memory-efficient data structure for language models

    Currently, the language models are loaded into simple maps at runtime. Even though accessing the maps is pretty fast, they consume a significant amount of memory. The goal is to investigate whether there are more suitable data structures available that require less storage space in memory, something like NumPy for Python.

    One promising candidate could be Gonum.

    enhancement 
    opened by pemistahl 0
  • Detect multiple languages in mixed-language text

    Detect multiple languages in mixed-language text

    Currently, for a given input string, only the most likely language is returned. However, if the input contains contiguous sections of multiple languages, it will be desirable to detect all of them and return an ordered sequence of items, where each item consists of a start index, an end index and the detected language.

    Input: He turned around and asked: "Entschuldigen Sie, sprechen Sie Deutsch?"

    Output:

    [
      {"start": 0, "end": 27, "language": ENGLISH}, 
      {"start": 28, "end": 69, "language": GERMAN}
    ]
    
    new feature 
    opened by pemistahl 1
Releases(v1.1.1)
  • v1.1.1(Nov 22, 2022)

  • v1.1.0(Nov 21, 2022)

    Features

    • The new method LanguageDetectorBuilder.WithLowAccuracyMode() has been introduced. By activating it, detection accuracy for short text is reduced in favor of a smaller memory footprint and faster detection performance. (#17)

    • The new method LanguageDetector.ComputeLanguageConfidence() has been introduced. It allows to retrieve the confidence value for one specific language only, given the input text. (#19)

    Improvements

    • The computation of the confidence values has been revised and the min-max normalization algorithm is now applied to the values, making them better comparable by behaving more like real probabilities. (#16)

    • The language models are now serialized as protocol buffers instead of json. Thanks to this change, they are now loaded into memory twice as fast as before. (#22)

    Bug Fixes

    • The unigram counts in the statistics engine were not retrieved correctly. This has been fixed, producing more correct detection results. (#14)

    Compatibility

    • The lowest supported Go version is 1.18 now. Older versions are no longer compatible with this library.

    Miscellaneous

    • The library now has a fresh and colorful new logo. Why? Well, why not? (-:
    Source code(tar.gz)
    Source code(zip)
  • v1.0.5(Dec 25, 2021)

    Bug Fixes

    • The character â was erroneously not treated as a possible indicator for French.

    Improvements

    • The dependencies to the other language detectors which are used for the accuracy comparisons were always downloaded together with the main library. They are only needed when you want to update the accuracy reports, therefore the cmd/ subdirectory now contains its own Go module that defines those dependencies. They have now been removed from the main library. Thanks to @dim and @BoeingX for identifying this problem. (#8)
    Source code(tar.gz)
    Source code(zip)
  • v1.0.4(Nov 28, 2021)

    Bug Fixes

    • It was possible to include lingua.Unknown in the set of input languages for building the language detector. It is only meant as a return value, so it is now automatically removed from the set of input languages. Thanks to @marians for identifying this problem. (#7)
    Source code(tar.gz)
    Source code(zip)
  • v1.0.3(Oct 20, 2021)

  • v1.0.2(Oct 13, 2021)

    Bug Fixes

    • In very rare cases, the language returned by the detector was non-deterministic. This has been fixed. Big thanks to @FilipAlexander for identifying this problem. (#6)
    Source code(tar.gz)
    Source code(zip)
  • v1.0.1(Jun 27, 2021)

    Bug Fixes

    • The language models were not embedded into the compiled binary. This resulted in problems when trying to use Lingua within a Docker container, for instance. Big thanks to @dsxack for identifying this problem and providing a fix. (#2 #3)
    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(Jun 21, 2021)

Owner
Peter M. Stahl
Computational linguist, Rust enthusiast, green IT advocate
Peter M. Stahl
ByNom is a Go package for parsing byte sequences, suitable for parsing text and binary data

ByNom is a Go package for parsing byte sequences. Its goal is to provide tools to build safe byte parsers without compromising the speed or memo

Andrew Bashkatov 4 May 5, 2021
Zero-width character detection and removal for Go

go-zero-width Zero-width character detection and removal for Go. Inspired by this Medium article. Installation go get github.com/trubitsyn/go-zero-wid

Nikola Trubitsyn 106 Sep 27, 2022
Unified text diffing in Go (copy of the internal diffing packages the officlal Go language server uses)

gotextdiff - unified text diffing in Go This is a copy of the Go text diffing packages that the official Go language server gopls uses internally to g

Hexops 96 Nov 16, 2022
This package provides Go (golang) types and helper functions to do some basic but useful things with mxGraph diagrams in XML, which is most famously used by app.diagrams.net, the new name of draw.io.

Go Draw - Golang MX This package provides types and helper functions to do some basic but useful things with mxGraph diagrams in XML, which is most fa

null 2 Aug 30, 2022
A general purpose application and library for aligning text.

align A general purpose application that aligns text The focus of this application is to provide a fast, efficient, and useful tool for aligning text.

John Moore 78 Sep 27, 2022
omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.

omniparser Omniparser is a native Golang ETL parser that ingests input data of various formats (CSV, txt, fixed length/width, XML, EDI/X12/EDIFACT, JS

JF Technology 518 Nov 18, 2022
:book: A Golang library for text processing, including tokenization, part-of-speech tagging, and named-entity extraction.

prose prose is a natural language processing library (English only, at the moment) in pure Go. It supports tokenization, segmentation, part-of-speech

Joseph Kato 3k Nov 20, 2022
A modern text indexing library for go

bleve modern text indexing in go - blevesearch.com Features Index any go data structure (including JSON) Intelligent defaults backed up by powerful co

bleve 8.7k Nov 22, 2022
Fonetic is a library to assess pronounceablility of a given text

fonetic-go assess pronounciblity of text Introduction Fonetic is a library to assess pronounceablility of a given text. For more information, check ou

Somdev Sangwan 35 Oct 21, 2022
Parse placeholder and wildcard text commands

allot allot is a small Golang library to match and parse commands with pre-defined strings. For example use allot to define a list of commands your CL

Sebastian Müller 55 Nov 24, 2022
Produces a set of tags from given source. Source can be either an HTML page, Markdown document or a plain text. Supports English, Russian, Chinese, Hindi, Spanish, Arabic, Japanese, German, Hebrew, French and Korean languages.

Tagify Gets STDIN, file or HTTP address as an input and returns a list of most popular words ordered by popularity as an output. More info about what

ZoomIO 24 Sep 27, 2022
Templating system for HTML and other text documents - go implementation

FAQ What is Kasia.go? Kasia.go is a Go implementation of the Kasia templating system. Kasia is primarily designed for HTML, but you can use it for any

Michał Derkacz 74 Mar 15, 2022
Small and fast FTS (full text search)

Microfts A small full text indexing and search tool focusing on speed and space. Initial tests seem to indicate that the database takes about twice as

Bill Burdick 27 Jul 30, 2022
Diff, match and patch text in Go

go-diff go-diff offers algorithms to perform operations required for synchronizing plain text: Compare two texts and return their differences. Perform

Sergi Mansilla 1.4k Dec 2, 2022
PipeIt is a text transformation, conversion, cleansing and extraction tool.

PipeIt PipeIt is a text transformation, conversion, cleansing and extraction tool. Features Split - split text to text array by given separator. Regex

Allen Dang 73 Aug 15, 2022
Fast and secure steganography CLI for hiding text/files in images.

indie CLI This complete README is hidden in the target.png file below without the original readme.png this could have also been a lie as none could ev

BoB 4 Mar 20, 2022
a simple and lightweight terminal text editor written in Go

Simple Text editor written in Golang build go build main.go

buzz 3 Oct 4, 2021
AppGo is an application that is intended to read a plain text log file and deliver an encoded polyline

AppGo AppGo is an application that is intended to read a plain text log file and deliver an encoded polyline. Installation To run AppGo it is necessar

Wendy Conde 0 Oct 23, 2021
A UTF-8 and internationalisation testing utility for text rendering.

ɱéťàł "English, but metal" Metal is a tool that converts English text into a legible, Zalgo-like character swap for the purposes of testing localisati

Harley 0 Jan 14, 2022