Self-contained Machine Learning and Natural Language Processing library in Go

Overview

alt text

Mentioned in Awesome Go Go Reference Go Go Report Card Maintainability codecov License Unstable PRs Welcome

If you like the project, please ★ star this repository to show your support! 🤩

A Machine Learning library written in pure Go designed to support relevant neural architectures in Natural Language Processing.

spaGO is self-contained, in that it uses its own lightweight computational graph framework for both training and inference, easy to understand from start to finish.

Usage

Requirements:

Clone this repo or get the library:

go get -u github.com/nlpodyssey/spago

spaGO supports two main use cases, which are explained more in detail in the following.

CLI mode

Several programs can be leveraged to tour the current NLP capabilities in spaGO. A list of the demos now follows.

The Docker image can be built like this.

docker build -t spago:main . -f Dockerfile

Library mode

You can access the core functionality of spaGO, i.e. optimizing mathematical expressions by back-propagating gradients through a computational graph, in your own code by using spaGO in library mode.

At a high level, it comprises four main modules:

  1. Matrix
  2. Graph
  3. Model
  4. Optimizer

To get started, look at the implementation of built-in neural models, such as the LSTM. Don't be afraid, it is straightforward Go code. The idea is that you could have written spaGO :)

You may find a Feature Source Tree useful for a quick overview of the library's package organization.

There is also a repo with handy examples, such as MNIST classification.

Features

Automatic differentiation

  • You write the forward(), it does all backward() derivatives for you:
    • Define-by-Run (default, just like PyTorch does)
    • Define-and-Run (similar to the static graph of TensorFlow)

Optimization methods

  • Gradient descent:
    • Adam, RAdam, RMS-Prop, AdaGrad, SGD
  • Differential Evolution

Neural networks

  • Feed-forward models (Linear, Highway, Convolution, ...)
  • Recurrent models (LSTM, GRU, BiLSTM...)
  • Attention mechanisms (Self-Attention, Multi-Head Attention, ...)
  • Recursive auto-encoders

Natural Language Processing

  • Memory-efficient Word Embeddings (with badger key–value store)
  • Character Language Models
  • Recurrent Sequence Labeler with CRF on top (e.g. Named Entities Recognition)
  • Transformer models (BERT-like)
    • Masked language model
    • Next sentence prediction
    • Tokens Classification
    • Text Classification (e.g. Sentiment Analysis)
    • Question Answering
    • Textual Entailment
    • Text Similarity

Compatible with pre-trained state-of-the-art neural models:

Current Status

We're not at a v1.0.0 yet, so spaGO is currently work-in-progress.

However, it has been running smoothly for a quite a few months now in a system that analyzes thousands of news items a day!

Besides, it's pretty easy to get your hands on through, so you might want to use it in your real applications.

Early adopters may make use of it for production use today as long as they understand and accept that spaGO is not fully tested and that APIs might change.

Known Limits

Sadly, at the moment, spaGO is not GPU friendly by design.

Contributing

We're glad you're thinking about contributing to spaGO! If you think something is missing or could be improved, please open issues and pull requests. If you'd like to help this project grow, we'd love to have you!

To start contributing, check the Contributing Guidelines.

Contact

We encourage you to write an issue. This would help the community grow.

If you really want to write to us privately, please email Matteo Grella with your questions or comments.

Links

Acknowledgments

spaGO is a personal project that is part of the open-source NLP Odyssey initiative initiated by members of the EXOP team. I would therefore like to thank EXOP GmbH here, which is providing full support for development by promoting the project and giving it increasing importance.

Sponsors

We appreciate contributions of all kinds. We especially want to thank spaGO fiscal sponsors who contribute to ongoing project maintenance.

Faire.ai logo

Our aim is simplifying people's life by making lending easy, fast and accessible, leveraging Open Banking and Artificial Intelligence. https://www.faire.ai/

Hype logo

We work on Artificial Intelligence based hardware and software systems, declining them in areas such as Energy Management, Personal Safety, E-Health and Sports equipment. https://hype-design.it/

BoxxApps logo

Professional services in the IT sector for Local Administrations, Enterprises and Local Authorities. https://www.boxxapps.com/

See our Open Collective page if you too are interested in becoming a sponsor.

Issues
  • Help with German zero shot

    Help with German zero shot

    I would like to run Sahajtomar/German_Zeroshot (https://huggingface.co/Sahajtomar/German_Zeroshot) model in spago.

    The import was successful: ./huggingface-importer --model=Sahajtomar/German_Zeroshot --repo=./models -> BERT has been converted successfully!

    Can I now run the model with Bart server(as I believe supports the zero shot, not the Bart server)?

    I receive:

    [email protected] spago % ./bart-server server --repo=./models --model=Sahajtomar/German_Zeroshot --tls-disable Start loading pre-trained model from "models/Sahajtomar/German_Zeroshot" [1/2] Loading configuration... ok panic: bart: unsupported architecture BertForSequenceClassification

    goroutine 1 [running]: github.com/nlpodyssey/spago/pkg/nlp/transformers/bart/loader.Load(0xc000038660, 0x21, 0x2, 0xc000038660, 0x21, 0x492f960) /Users/bassea/go/src/spago/pkg/nlp/transformers/bart/loader/loader.go:43 +0x819 github.com/nlpodyssey/spago/cmd/bart/app.newServerCommandActionFor.func1(0xc00022b740, 0x0, 0x0) /Users/bassea/go/src/spago/cmd/bart/app/server.go:106 +0x105 github.com/urfave/cli/v2.(*Command).Run(0xc000222ea0, 0xc00022b440, 0x0, 0x0) /Users/bassea/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:163 +0x4e0 github.com/urfave/cli/v2.(*App).RunContext(0xc0000351e0, 0x4ae0aa0, 0xc000036068, 0xc0000320a0, 0x5, 0x5, 0x0, 0x0) /Users/bassea/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:313 +0x814 github.com/urfave/cli/v2.(*App).Run(...) /Users/bassea/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:224 main.main() /Users/bassea/go/src/spago/cmd/bart/main.go:15 +0x72

    opened by abasse 16
  • Implement optional interactive model search for downloader

    Implement optional interactive model search for downloader

    This commit adds a optional "search huggingface" feature to the downloader.

    opened by paralin 12
  • float32 data type

    float32 data type

    This is just a question out of curiosity but: Do you have any plans to support the float32 data type (or any other types, like integers actually)?

    • It is very common to train a neural network with a float32 precision, as it reduces the computation cost without having any significant impact on the accuracy, and I was wondering what would be the speed gain for spago?
    • I was thinking about something like an Enum type given to the matrix creation function and maybe the possibility to convert an existing matrix from a type to another
    • Supporting types like uint8 could allow to implement some quantization schemes more easily, which seems like a good fit with Golang since it shines in distributed applications (e.g. sending quantized weights over the network is definitely more bandwidth friendly)

    For now I just find myself "fighting the matrix" by extracting the underlying data, convert them in the desired type, do some work with them and finally reloading them later in a matrix to run some computation.

    enhancement 
    opened by xiorcale 8
  • Running hugging_face_importer from docker container causes strange behavior

    Running hugging_face_importer from docker container causes strange behavior

    I was following instructions to test the question answering demo and noticed that the container never completed by outputting the spago model and left a zombie python process running on my machine (a mid-2013 MacBook Pro 2.3 ghz quad core i7, 16 gb ddr3). Here are steps to reproduce:

    # after cloning the repo in its latest form
    git rev-parse --short HEAD
    f91d1b8
    docker build -t spago:main . -f Dockerfile
    # that completes successfully
    mkdir ~/.spago
    # then i run the hugging face import step via the container
    docker run --rm -it -v ~/.spago:/tmp/spago spago:main ./hugging_face_importer --model=deepset/bert-base-cased-squad2 --repo=/tmp/spago
    
    Running command: './hugging_face_importer --model=deepset/bert-base-cased-squad2 --repo=/tmp/spago'
    Downloading dataset...
    Start downloading 🤗 `deepset/bert-base-cased-squad2`
    2020/06/27 18:55:30 Fetch the model configuration from `https://s3.amazonaws.com/models.huggingface.co/bert/deepset/bert-base-cased-squad2/config.json`
    Downloading... 508 B complete
    2020/06/27 18:55:30 Fetch the model vocabulary from `https://s3.amazonaws.com/models.huggingface.co/bert/deepset/bert-base-cased-squad2/vocab.txt`
    Downloading... 214 kB complete
    2020/06/27 18:55:30 Fetch the model weights from `https://s3.amazonaws.com/models.huggingface.co/bert/deepset/bert-base-cased-squad2/pytorch_model.bin` (it might take a while...)
    
    # this process runs for a _really_ long time - ive actually never seen it finish successfully (have let it run for over 60 minutes)
    ...
    

    In another shell session I was inspecting what Python processes are operating because I noted some cpu hogging after the pytorch model was fully downloaded ...

    $ ls -hal ~/.spago/deepset/bert-base-cased-squad2/
    total 417M
    drwxr-xr-x 6 anthcor staff  192 Jun 27 14:56 ./
    drwxr-xr-x 3 anthcor staff   96 Jun 27 14:55 ../
    -rw-r--r-- 1 anthcor staff  508 Jun 27 14:55 config.json
    drwx------ 6 anthcor staff  192 Jun 27 14:57 embeddings_storage/
    -rw-r--r-- 1 anthcor staff 414M Jun 27 14:56 pytorch_model.bin
    -rw-r--r-- 1 anthcor staff 209K Jun 27 14:55 vocab.txt
    $ ps aux | grep spago
    anthcor          29694   6.1  0.1  4444408  22784 s003  S+    2:55PM   0:03.44 docker run --rm -it -v /Users/anthcor/.spago:/tmp/spago spago:main ./hugging_face_importer --model=deepset/bert-base-cased-squad2 --repo=/tmp/spago
    anthcor          29705   0.0  0.0  4268300    700 s004  S+    2:56PM   0:00.00 grep spago
    anthcor          29693   0.0  0.0  4280612   6932 s003  S+    2:55PM   0:00.08 /usr/local/Cellar/[email protected]/3.8.3/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python /usr/local/bin/grc -es --colour=auto docker run --rm -it -v /Users/anthcor/.spago:/tmp/spago spago:main ./hugging_face_importer --model=deepset/bert-base-cased-squad2 --repo=/tmp/spago
    

    When running this workflow not with docker everything works as expected.

    In order to stop everything I just kill the docker container and the zombie local python process.

    ps aux | grep -i spago | awk '{print $2}' | xargs kill -9 $1
    

    Not really sure why this happens – looks like the container is using a local system binary and that causes hangups in the flow of things but I could totally be wrong as I haven't really spent too much time diving in. Hope this gives enough insight into my issue – lmk if you would like any more details. Cheers 🍻

    solved 
    opened by anthcor 6
  • BERT Server Prediction Doesn't Seem To Work

    BERT Server Prediction Doesn't Seem To Work

    I'm attempting to use the BERT server, and have successfully gotten the /answer API call to work. I can't seem to find much information on BERT prediction in general, but I'm guessing its used to predict what the next sentence will be?

    Based on this I tried sending a JSON request to the /predict route like so, but it gives an empty response:

    $> curl -d '{"text": "BERT is a technique for NLP developed by Google. BERT was created and published in 2018 by Jacob Devlin and his colleagues from Google."}' -H Content-Type=application/json http://localhost:1987/predict
    {"tokens":[]}
    

    Contrast this with discriminate which works:

    $> curl -d '{"text": "BERT is a technique for NLP developed by Google. BERT was created and published in 2018 by Jacob Devlin and his colleagues from Google."}' -H Content-Type=application/json http://localhost:1987/discriminate
    {"tokens":[{"text":"BERT","start":0,"end":4,"label":"FAKE"},{"text":"is","start":5,"end":7,"label":"FAKE"},{"text":"a","start":8,"end":9,"label":"FAKE"},{"text":"technique","start":10,"end":19,"label":"FAKE"},{"text":"for","start":20,"end":23,"label":"FAKE"},{"text":"NLP","start":24,"end":27,"label":"FAKE"},{"text":"developed","start":28,"end":37,"label":"FAKE"},{"text":"by","start":38,"end":40,"label":"FAKE"},{"text":"Google","start":41,"end":47,"label":"FAKE"},{"text":".","start":47,"end":48,"label":"FAKE"},{"text":"BERT","start":49,"end":53,"label":"FAKE"},{"text":"was","start":54,"end":57,"label":"FAKE"},{"text":"created","start":58,"end":65,"label":"FAKE"},{"text":"and","start":66,"end":69,"label":"FAKE"},{"text":"published","start":70,"end":79,"label":"FAKE"},{"text":"in","start":80,"end":82,"label":"FAKE"},{"text":"2018","start":83,"end":87,"label":"FAKE"},{"text":"by","start":88,"end":90,"label":"FAKE"},{"text":"Jacob","start":91,"end":96,"label":"FAKE"},{"text":"Devlin","start":97,"end":103,"label":"FAKE"},{"text":"and","start":104,"end":107,"label":"FAKE"},{"text":"his","start":108,"end":111,"label":"FAKE"},{"text":"colleagues","start":112,"end":122,"label":"FAKE"},{"text":"from","start":123,"end":127,"label":"FAKE"},{"text":"Google","start":128,"end":134,"label":"FAKE"},{"text":".","start":134,"end":135,"label":"FAKE"}]}
    

    Also, what does discriminate do exactly? I found an article explaining all the details of BERT but cant seem to find out what discriminate does

    question 
    opened by bonedaddy 5
  • bert train

    bert train

    I want to generate my own bert model,but I'm not sure how to train my corpus,Could you make out a tutorial,I am a rookie in the GO : (

    opened by HaoRan98 4
  • Cannot import project as library

    Cannot import project as library

    Hello,

    I am trying to import this project as a library by running the command go get -u github.com/nlpodyssey/spago (as instructed in the readme)

    I am getting the following error: cannot find package "github.com/dgraph-io/badger/v2"

    Anyone else got this error?

    solved 
    opened by csaade 4
  • Better BERT Server Capabilities

    Better BERT Server Capabilities

    (apologies if the many issues are annoying, really liking the library so far)

    Overview

    Right now when using the BERT server, a set of defaults are used without any control from the user:

    • HTTP server lacks customizability
    • HTTP server listens on 0.0.0.0
    • No ability to enable TLS

    Additionally it's not possible to build "external servers" as the functions used by the BERT http router (discriminateHandler, predictionHandler, qaHandler are private functions). If this was changed to instead export the handler functions (DiscriminateHandler, PredictionHandler, QaHandler) it would allow people to have more control over the BERT server, better middleware capabilities, TLS, etc...

    By using public handler functions, users would be able to define their own routers, say using chi, and overall have more control of the BERT server.

    I'd be more than happy to open a PR that implements these suggestions

    enhancement 
    opened by bonedaddy 4
  • This is the best thing since sliced bread

    This is the best thing since sliced bread

    Hello I do not have any issue so feel free to close this but I just wanted to say that spago is fantastic. Love what has been done here in native go!!! Thank you everybody involved. FORZA ITALIA

    opened by go-dockly 3
  • Not very robust update of Mean and StdDev in BatchNorm

    Not very robust update of Mean and StdDev in BatchNorm

    A second thought about the recent fix of the BatchNorm (#48) raises an interesting question.

    It is the first time that it happens that during a forward, the parameters are directly altered (in this case Mean and StdDev):

    https://github.com/nlpodyssey/spago/blob/cb6bad4ccf503bdafe0fde13510ba8868e830c3d/pkg/ml/nn/normalization/batchnorm/batchnorm.go#L80

    In general, I believe that the Optimizer should be the only responsible for updating the parameters. However, here we are not talking about gradients, so the situation is slightly different.

    The first thing that comes to mind is to update these values (Mean and StdDev) by saving them inside the Processor. A special method is then added to it which has the effect of updating the model parameters. It remains an ad hoc act, but it seems more robust and consistent with the rest.

    @jjviana What do you think? Let's discuss here :)

    opened by matteo-grella 3
  • bart-large-mnli multi_class does not agree with Python version

    bart-large-mnli multi_class does not agree with Python version

    If you convert facebook/bart-large-mnli and use it to evaluate the demo text at huggingface and compare against a local Python setup for verification, we find that:

    • the online demo card and the local Python agree on the label score
    • the label probabilities given back are vastly different
    • the Python version takes roughly 16 seconds on my local machine, but the Spago version takes 37 seconds - this is a MAC and there is no GPU available

    Python code is

        text = "A new model offers an explanation for how the Galilean satellites formed around the solar system’s " \
               "largest world. Konstantin Batygin did not set out to solve one of the solar system’s most puzzling " \
               "mysteries when he went for a run up a hill in Nice, France. Dr. Batygin, a Caltech researcher, " \
               "best known for his contributions to the search for the solar system’s missing “Planet Nine,” spotted a " \
               "beer bottle. At a steep, 20 degree grade, he wondered why it wasn’t rolling down the hill. He realized " \
               "there was a breeze at his back holding the bottle in place. Then he had a thought that would only pop " \
               "into the mind of a theoretical astrophysicist: “Oh! This is how Europa formed.” Europa is one of " \
               "Jupiter’s four large Galilean moons. And in a paper published Monday in the Astrophysical Journal, " \
               "Dr. Batygin and a co-author, Alessandro Morbidelli, a planetary scientist at the Côte d’Azur Observatory " \
               "in France, present a theory explaining how some moons form around gas giants like Jupiter and Saturn, " \
               "suggesting that millimeter-sized grains of hail produced during the solar system’s formation became " \
               "trapped around these massive worlds, taking shape one at a time into the potentially habitable moons we " \
               "know today. "
        cc = pipeline("zero-shot-classification", model="facebook/bart-large-mnli")
        labels = ['space & cosmos', 'scientific discovery', 'microbiology', 'robots', 'archeology']
        r = cc(text, labels, multi_class=True)
    

    Go code, with same text and classes, is:

    bartModel, err = zsc.LoadModel(bartDir)
    // ... check err
    result, err := bartModel.Classify(c.InputText, "", classes, true)
    

    Similarly using the model valhalla/distilbart-mnli-12-3 also gives wildly different results to the online huggingface demo, using the same text and label set as above.

    So, is there something else I need to do, or is the zsc code not working? My go code is essentially just like the zsc demo code.

    opened by jimidle 0
  • Benchmarks vs numpy, scipy, sklearn, Pytorch

    Benchmarks vs numpy, scipy, sklearn, Pytorch

    Did you guys compare this library for equivalent implementations in numpy, scipy, sklear, Pytorch, Tensorflow?

    If GPU is not supported, you can try reporting CPU versions. AFAIK both Pytorch and Tensorflow have cpu modes for their tensor operations.

    documentation 
    opened by nikolaydubina 4
  • Error handling

    Error handling

    Hey Guys, I saw that there are 227 uses of panic in 87 files and I do understand your motivation behind it. Go's implicit error returns stand in the way of function chaining youd like to achieve eg gx := r.x1.Value().(*mat.Dense).MulT(gy) Let's assume MulT can panic inside the backward scope but that panic should recover in eg the graph wrapper scope This should explain bit better what I mean What are your thoughts on this?

    enhancement 
    opened by go-dockly 1
  • Is it possible to pre-load passages from csv?

    Is it possible to pre-load passages from csv?

    Is it currently possible to preload let's say the go faq and run semantic search on the passages by only providing a question to spago serving squad2. I would like to create behavior like shown in this video semantic-search

    If not yet could you give me some pointers to dig into it?

    Thank you

    opened by go-dockly 7
  • Other integration / Telegram Bot

    Other integration / Telegram Bot

    Hi guys,

    Hope you are all well !

    I was looking for some implementation of AI chatbot golang and found your project spago.

    And, I am developing a multibot for telegram based on go plugins: https://github.com/paper2code/telegram-multibot

    So I was wondering how can I integrate spago as a QA bot plugin.

    Do you have any insights or advices for such integration ?

    Thanks in advance.

    Cheers, X

    opened by x0rzkov 6
  • Support mTLS authentication and authorizaton

    Support mTLS authentication and authorizaton

    Add support for optional certificate-based mTLS authentication and authorization.

    enhancement 
    opened by evanmcclure 0
  • Verify integrity of models fetched from the Interwebs

    Verify integrity of models fetched from the Interwebs

    Currently, models that are fetched from remote sources are uncompressed without having their integrity verified. Models fetched from remote sources should be verified first, using checksums that are published with the models.

    enhancement 
    opened by evanmcclure 0
  • Create a simple website

    Create a simple website

    @evanmcclure what do you think about a simple landing page?

    opened by matteo-grella 8
  • Will spago support other languages?

    Will spago support other languages?

    Cool project! I wonder if spaGO will support other languages? For example Russian?

    question 
    opened by alikhil 1
  • Combine Multiple Models With BERT Server?

    Combine Multiple Models With BERT Server?

    I'm experimenting with spago, and while attempting to start a BERT server, I noticed that it was impossible to load multiple models, and use multiple models with a single instance of a BERT server.

    Is this not possible because the code to do so hasn't been written, or its not possible due to the way BERT server works?

    question 
    opened by bonedaddy 2
Releases(v0.7.0)
  • v0.7.0(May 24, 2021)

    Added

    • New package ml/ag/encoding/dot, for simple serialization of a Graph to DOT (Graphviz) format.
    • New package ml/nn/sgu, implementing a Spatial Gating Unit (SGU) model.
    • New package ml/nn/conv1x1, implementing a simple 1-dimensional 1-sized-kernel convolution model.
    • New package ml/nn/gmlp, implementing a gMLP model.

    Changed

    • ml/nn/activation/Model.Forward now simply returns the input as it is if the activation function is the identity.
    Source code(tar.gz)
    Source code(zip)
  • v0.6.0(May 13, 2021)

    Added

    • ml/losses.WeightedCrossEntropy()
    • ml/losses.FocalLoss()
    • ml/losses.WeightedFocalLoss()
    • nlp/sequencelabeler.LoadModel() (it replaces Load() and LoadEmbeddings())
    • nlp/charlm.LoadModel()
    • nlp/transformers/bert.Model.PredictMLM()
    • nlp/transformers/bart/tasks package
    • nlp/transformers/bert.Model.Vectorize()
    • ml/ag.Graph.Nodes() and ml/ag.Nodes()
    • ml/nn.Model.Close()
    • ml/nn.ReifyForTraining() and ml/nn.ReifyForInference()
    • ml/ag.Graph.Backward() now panics if it is executed with nodes belonging to different graphs.
    • The new ml/graphviz package allows exporting a Graph to Graphviz DOT format. To make it possible, we introduced a new go-mod dependency gographviz.
    • A custom name can be optionally set to a Graph's Variables. This can be useful for debugging purposes and visual graph representation. You can now use Graph.NewVariableWithName() and Graph.NewScalarWithName() to create named Variables, and get the name of a Variable with Variable.Name().

    Changed

    • All UnaryElementwise functions provided by the package ag/fn have been promoted to separate dedicated structs. This improves debuggability and you can get appropriate function names when using reflection. Here is the full list of the modified functions: Tan, Tanh, Sigmoid, HardSigmoid, HardTanh, ReLU, Softsign, Cos, Sin, Exp, Log, Neg, Reciprocal, Abs, Mish, GELU, Sqrt, Swish. For the same reason, a dedicated Square function is introduced, replacing Prod with both operands set to the same value.
    • ml/ag types Operator, Variable, Wrapper are now public.
    • ml/nn.Reify() now expects a Graph and a Processing Mode arguments instead of a Context object (removed).
    • ml/nn.BaseModel has been modified, replacing the field Ctx Context with a direct reference to the model's Graph and the Processing Mode (fields G and ProcessingMode).
    • Refactoring server implementation of nlp/sequencelabeler, nlp/transformers/bert, and nlp/transformers/bart.
    • Upgrade various dependencies.
    • Regenerate protocol buffers files (with protoc-gen-go v1.26.0 and protoc v3.16.0).

    Removed

    • nlp/sequencelabeler.Load() and LoadEmbeddings() (now replaced by nlp/sequencelabeler.LoadModel())
    • ml/nn.Context (see related changes on Reify() and BaseModel)
    Source code(tar.gz)
    Source code(zip)
  • v0.5.2(Mar 16, 2021)

  • v0.5.1(Mar 7, 2021)

    Added

    Changed

    • Improve nlp.transformer.generation algorithms:
      • optimize Generator.getTopKScoredTokens().
      • optimize Generator.updateTokensScores().
    • Simplify mat32.Dense.Mul when doing Matrix-Vector multiplication.
    • Refactor math32 functions using chewxy/math32 functions.
    • Improve ag.Graph efficiency:
      • Use pre-computed cache doing ag.Graph.groupNodesByHeight().
      • Use sync.pool to reduce allocations of graph's operators.

    Fixed

    • Fix past key-values usage on self-attention and cross-attention
    Source code(tar.gz)
    Source code(zip)
  • v0.5.0(Feb 15, 2021)

    Added

    • Implement a beam-search algorithm for conditional generation:
      • nlp.transformer.generation package.
    • Add implementation of the Sentence-Piece tokenizer:
      • nlp.tokenizers.sentencepiece package.
    • BART improvements:
      • gRPC and HTTP API to perform Text Generation.
      • Add support for "Marian" architecture (used for translation tasks).
      • Add sinusoidal positional encoder (used by Marian).
      • Add "head" for conditional generation:
        • nlp.transformers.bart.head.conditionalgeneration package.
    • Add nn.Closer interface (e.g. embeddings.Model needs to close the underlying key-value store).
    • Add Swish act. function without trainable parameters.
    • Add SiLU act. function (it is just an alias for Swish).
    • New pe.SinusoidalPositionalEncoder (this implementation replaces unused pe.PositionalEncoder and pe.AxialPositionalEncoder)

    Changed

    • Update urfave/cli to v2.
    • Update dgraph-io/badger to v3.
    • Make the BART positional encoder an interface to support various encoding (i.e. trainable vs static).
    • Rename to fn.NewSwish into fn.NewSwishB as this was the Swish variant with trainable parameters (B).
    • Relax ag.GetOpName to match operator names in lower-case.
    • Allow arbitrary activation function on BART encoder/decoder layers.
    • Use precomputed "keys" and "values" in self-attention, multi-head attention and BART decoder.

    Removed

    • In relation to the aforementioned positional encoding changes:
      • pe.PositionalEncoder and related functions
      • pe.AxialPositionalEncoder and related functions

    Fixed

    • Fix causal-mask used by nn.ScaledDotProductAttention
    Source code(tar.gz)
    Source code(zip)
  • v0.4.1(Jan 22, 2021)

    Added

    • New function ReleaseMatrix to packages mat32 and mat64.
    • New methods to Matrix interface, from mat32 and mat64: Minimum, Maximum, MulT, Inverse, DoNonZero. However, the implementation on sparse matrices is not implemented yet (it always panics).

    Changed

    • Prefer handling Matrix interface values over specific Dense or Sparse matrices, also avoiding unnecessary type casts. Relevant changes to the public API are listed below.
      • mat(32|64).Stack function's arguments and returned value are now Matrix interfaces, instead of explicit Dense matrices.
      • Dense.Minimum and Dense.Maximum, from packages mat32 and mat64, return a Matrix interface, instead of a specific Dense type.
      • The return values of fofe.EncodeDense, fofe.Encode, and fofe.BiEncode are slices of Matrix values, instead of Dense or Sparse.
      • The z argument of the function fofe.Decode is of type Matrix, instead of Dense.
      • ml.optimizers.de (Differential Evolution optimizer) API was changed handling Matrix values, instead of specific Dense matrices. Changes include: Member.TargetVector, Member.DonorVector, ScoredVector.Vector, the vector argument of NewMember function, the solution argument of score and validate functions passed to NewOptimizer.
      • PositionalEncoder.Cache and AxialPositionalEncoder.Cache are slices of Matrix, instead of slices of Dense.
      • AxialPositionalEncoder.EncodingAt returns a Matrix value, instead of Dense.
      • nn.DumpParamsVector returns a Matrix value, instead of Dense.
      • The vector argument of the function nn.LoadParamsVector is a Matrix, instead of Dense.
      • The value argument of the method embeddings.Model.SetEmbedding is of type Matrix, instead of Dense.
      • The type of the struct field evolvingembeddings.WordVectorPair.Vector is Matrix, instead of Dense.
    Source code(tar.gz)
    Source code(zip)
  • v0.4.0(Jan 16, 2021)

    Added

    • Various new test cases (improving the coverage).
    • nlp.embeddings.syncmap package.
    • ml.nn.recurrent.srnn.BiModel which implements a bidirectional variant of the Shuffling Recurrent Neural Networks (SRNN).
    • Configurable timeout and request limit to all HTTP and gRPC servers (see also commands help).

    Changed

    • All CLI commands implementation has been refactored, so that the docker-entrypoint can reuse all other cli.App objects, instead of just running separate executables. By extension, now the Dockerfile builds a single executable file, and the final image is way smaller.
    • All dependencies have been upgraded to the latest version.
    • Simplify custom error definitions using fmt.Errorf instead of functions from github.com/pkg/errors.
    • Custom binary data serialization of matrices and models is now achieved with Go's encoding.gob. Many specific functions and methods are now replaced by fewer and simpler encoding/decoding methods compatible with gob. A list of important related changes follows.
      • utils.kvdb.KeyValueDB is no longer an interface, but a struct which directly implements the former "badger backend".
      • utils.SerializeToFile and utils.DeserializeFromFile now handle generic interface{} objects, instead of values implementing Serializer and Deserializer.
      • mat32 and mat64 custom serialization functions (e.g. MarshalBinarySlice, MarshalBinaryTo, ...) are replaced by implementations of BinaryMarshaler and BinaryUnmarshaler interfaces on Dense and Sparse matrix types.
      • PositionalEncoder.Cache and AxialPositionalEncoder.Cache fields (from ml.encoding.pe package) are now public.
      • All types implementing nn.Model interface are registered for gob serialization (in init functions).
      • embeddings.Model.UsedEmbeddings type is now nlp.embeddings.syncmap.Map.
      • As a consequence, you will have to re-serialize all your models.
    • Flair converter now sets the vocabulary directly in the model, instead of creating a separate file.
    • sequencelabeler.Model.LoadParams has been renamed to Load.

    Removed

    • In relation to the aforementioned gob serialization changes:
      • nn.ParamSerializer and related functions
      • nn.ParamsSerializer and related functions
      • utils.Serializer and utils.Deserializer interfaces
      • utils.ReadFull function
    • sequencelabeler.Model.LoadVocabulary

    Fixed

    • docker-entrypoint sub-command hugging-face-importer has been renamed to huggingface-importer, just like the main command itself.
    • docker-entrypoint sub-command can be correctly specified without leading ./ or / when run from a Docker container.
    • BREAKING: mat32.Matrix serialization has been fixed, now serializing single values to chunks of 4 bytes (instead of 8, like float64). Serialized 32-bit models will now be half the size! Unfortunately you will have to re-serialize your models (sorry!).
    Source code(tar.gz)
    Source code(zip)
  • v0.3.0(Jan 9, 2021)

    Added

    • Static analysis job (golint and gocyclo) to Go GitHub workflow.
    • You can set a limit for concurrent heavyweight Graph computations (e.g. forward and backward steps) - see ml.ag.ConcurrentComputations() (GraphOption) and ml.ag.Graph.ConcurrentComputations(). If no option is specified, by default the limit is set to runtime.NumCPU().
    • You can set a limit for concurrent heavyweight computations of ml.optimizers.gd.GradientDescent (e.g. params update step).
    • New package utils.processingqueue.
    • mat32 package, which operates on float32 data type.
    • It's possible to switch between float32 and float64 as default floating-point data type, using the script change-float-type.sh
    • Go GitHub workflow has been adapted to run tests using both float32 and float64 as main floating-point data type.
    • This CHANGELOG file.
    • Pull and convert Hugging Face models automatically if not found locally when starting BERT or BART server.
    • Move content from GitHub Wiki to README in related package folders.

    Changed

    • ml.ag.ConcurrentComputations (GraphOption) expects the maximum number of concurrent computations handled by heavyweight Graph operations (e.g. forward and backward steps).
    • ml.nn.linear.Model and ml.nn.convolution.Model read the concurrent computations limit set on the model's Graph, thus SetConcurrentComputations() methods have been removed.
    • mat has been renamed to mat64 and some functions have been renamed.
    • The whole project now works with float32 floating-point data type by default, by using the package mat32.
    • When imported, the new package mat32 is always aliased as mat. Then, explicit usages of float64 type have been replaced with mat.Float. Moreover, bitsize-specific functions have been made more generic (i.e. operating with mat.Float type) or split into separate implementation, in mat32 and mat64. In this way, switching the whole project between float32 and float64 is just a matter of changing all imports, from mat32 to mat64, or vice-versa (see also the new file change-float-type.sh).
    • Update internal links to pre-trained NER models to float32 versions.
    • nlp.sequencelabeler.Convert() now loads and converts original Flair models, instead of pre-processed dumps.
    • Change command line arguments to make them more consistent; please refer to the help messages of each command.
    • Update Dockerfile using a new base building image and adding bart server.

    Fixed

    • Added dedicated package names to different protocol buffers definition files to avoid name conflicts.
    Source code(tar.gz)
    Source code(zip)
  • v0.2.0(Dec 30, 2020)

    Notable changes:

    • Added support for BART model (tested on Natural Language Inference task);
    • Added BART API to perform Zero-Shot Text Classification;
    • Significant reduction of boilerplate code through the unification of nn.Model and nn.Processor interfaces:
      • there is now a single nn.Model interface that can be reified to become a neural processor. See nn.Reify();
      • there was no compelling reason to have a Forward method in the nn.Model interface so it has been removed, gracefully increasing flexibility in the implementation of a model.
    Source code(tar.gz)
    Source code(zip)
Owner
NLP Odyssey
NLP Odyssey
Selected Machine Learning algorithms for natural language processing and semantic analysis in Golang

Natural Language Processing Implementations of selected machine learning algorithms for natural language processing in golang. The primary focus for t

James Bowman 331 Jul 15, 2021
A Golang library for text processing, including tokenization, part-of-speech tagging, and named-entity extraction.

prose is a natural language processing library (English only, at the moment) in pure Go. It supports tokenization, segmentation, part-of-speech tagging, and named-entity extraction.

Joseph Kato 2.8k Jul 24, 2021
Natural language detection library for Go

Whatlanggo Natural language detection for Go. Features Supports 84 languages 100% written in Go No external dependencies Fast Recognizes not only a la

Abado Jack Mtulla 498 Jul 2, 2021
[UNMANTEINED] Extract values from strings and fill your structs with nlp.

nlp nlp is a general purpose any-lang Natural Language Processor that parses the data inside a text and returns a filled model Supported types int in

Juan Alvarez 369 Jun 5, 2021
Natural language detection package in pure Go

getlang getlang provides fast natural language detection in Go. Features Offline -- no internet connection required Supports 29 languages Provides ISO

Rylan 118 Jul 9, 2021
A natural language date/time parser with pluggable rules

when when is a natural language date/time parser with pluggable rules and merge strategies Examples tonight at 11:10 pm at Friday afternoon the deadli

Oleg Lebedev 1.1k Jul 17, 2021
Golang RESTful Client for HanLP.中文分词 词性标注 命名实体识别 依存句法分析 语义依存分析 新词发现 关键词短语提取 自动摘要 文本分类聚类 拼音简繁转换 自然语言处理

gohanlp 中文分词 词性标注 命名实体识别 依存句法分析 语义依存分析 新词发现 关键词短语提取 自动摘要 文本分类聚类 拼音简繁转换 自然语言处理 HanLP 的golang 接口 在线轻量级RESTful API 仅数KB,适合敏捷开发、移动APP等场景。服务器算力有限,匿名用户配额较少

xxj 15 Jul 14, 2021
Self-contained Japanese Morphological Analyzer written in pure Go

Kagome v2 Kagome is an open source Japanese morphological analyzer written in pure golang. The dictionary/statistical models such as MeCab-IPADIC, Uni

ikawaha 580 Jul 13, 2021
Translate your Go program into multiple languages.

go-i18n go-i18n is a Go package and a command that helps you translate Go programs into multiple languages. Supports pluralized strings for all 200+ l

Nick Snyder 1.7k Jul 26, 2021
A multilingual command line sentence tokenizer in Golang

Sentences - A command line sentence tokenizer This command line utility will convert a blob of text into a list of sentences. Demo Docs Install go get

Eric Bower 306 Jul 10, 2021
Cross platform locale detection for Golang

go-locale go-locale is a Golang lib for cross platform locale detection. OS Support Support all OS that Golang supported, except android: aix: IBM AIX

Xuanwo 78 Jun 8, 2021
A Go library for performing Unicode Text Segmentation as described in Unicode Standard Annex #29

segment A Go library for performing Unicode Text Segmentation as described in Unicode Standard Annex #29 Features Currently only segmentation at Word

bleve 63 Jun 9, 2021
i18n (Internationalization and localization) engine written in Go, used for translating locale strings.

go-localize Simple and easy to use i18n (Internationalization and localization) engine written in Go, used for translating locale strings. Use with go

Miles Croxford 25 Jun 24, 2021
Cgo binding for icu4c library

About Cgo binding for icu4c C library detection and conversion functions. Guaranteed compatibility with version 50.1. Installation Installation consis

Dmitry Bondarenko 19 Oct 29, 2020