Ngram index for golang


go-ngram Build Status

N-gram index for Go.

Key features

  • Unicode support.
  • Append only. Data can't be deleted from index.
  • GC friendly (all strings are pooled and compressed)
  • Application agnostic (there is no notion of document or something that user needs to implement)


index, err := ngram.NewNGramIndex(ngram.SetN(3))
tokenId, err := index.Add("hello") 
str, err := index.GetString(tokenId)  // str == "hello"
resultsList, err := index.Search("world")


  • Smoothing functions (Laplace etc)


docs examples

library users

  • Add locks for concurrent access

    Add locks for concurrent access

    When trying to search for a match in different goroutines, we get a panic because the underlying storage uses a map (which is not concurrent safe).

    This implements a simple lock mechanism to prevent panics.

    opened by phacops 2
Eugene Lazin
Eugene Lazin
Generate a global index for multiple markdown files recursively

markdown index Markdown-index is a library to help you generate a global index for multiple markdown files recursively in a directory, containing a su

Mateus Miranda 15 May 12, 2022
[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。

Goribot 一个分布式友好的轻量的 Golang 爬虫框架。 完整文档 | Document !! Warning !! Goribot 已经被迁移到 Gospider|。修复了一些调度问题并分离了网络请求部分到另一个仓库。此仓库会继续

null 208 Jul 4, 2022
golang 在线预览word,excel,pdf,MarkDown(Online Preview Word,Excel,PPT,PDF,Image by Golang)

Go View File 在线体验地址 (不会经常更新,保留最基本的预览功能。服务器配置较低,如果出现链接超时请等待几秒刷新重试,或者换Chrome) 目前已经完成 docker部署 (不用为运行环境烦恼) Wor

CZC 59 Jun 14, 2022
bluemonday: a fast golang HTML sanitizer (inspired by the OWASP Java HTML Sanitizer) to scrub user generated content of XSS

bluemonday bluemonday is a HTML sanitizer implemented in Go. It is fast and highly configurable. bluemonday takes untrusted user generated content as

Microcosm 2.4k Jul 4, 2022
Elegant Scraper and Crawler Framework for Golang

Colly Lightning Fast and Elegant Scraping Framework for Gophers Colly provides a clean interface to write any kind of crawler/scraper/spider. With Col

Colly 16.9k Jun 30, 2022
A golang package to work with Decentralized Identifiers (DIDs)

did did is a Go package that provides tools to work with Decentralized Identifiers (DIDs). Install go get Example packag

Ockam 61 Jul 2, 2022
wcwidth for golang

go-runewidth Provides functions to get fixed width of the character or string. Usage runewidth.StringWidth("つのだ☆HIRO") == 12 Author Yasuhiro Matsumoto

mattn 460 Jun 26, 2022
Parses the Graphviz DOT language in golang

Parses the Graphviz DOT language and creates an interface, in golang, with which to easily create new and manipulate existing graphs which can be writ

Walter Schulze 484 Jun 25, 2022
Go (Golang) GNU gettext utilities package

Gotext GNU gettext utilities for Go. Features Implements GNU gettext support in native Go. Complete support for PO files including: Support for multil

Leonel Quinteros 342 Jun 29, 2022
htmlquery is golang XPath package for HTML query.

htmlquery Overview htmlquery is an XPath query package for HTML, lets you extract data or evaluate from HTML documents by an XPath expression. htmlque

null 494 Jul 3, 2022
omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.

omniparser Omniparser is a native Golang ETL parser that ingests input data of various formats (CSV, txt, fixed length/width, XML, EDI/X12/EDIFACT, JS

JF Technology 467 Jul 3, 2022
Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler

Pagser Pagser inspired by page parser。 Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and str

foolin 62 Jun 13, 2022
iTunes and RSS 2.0 Podcast Generator in Golang

podcast Package podcast generates a fully compliant iTunes and RSS 2.0 podcast feed for GoLang using a simple API. Full documentation with detailed ex

Eric Duncan 109 May 31, 2022
TOML parser for Golang with reflection.

THIS PROJECT IS UNMAINTAINED The last commit to this repo before writing this message occurred over two years ago. While it was never my intention to

Andrew Gallant 3.9k Jun 26, 2022
Your CSV pocket-knife (golang)

csvutil - Your CSV pocket-knife (golang) #WARNING I would advise against using this package. It was a language learning exercise from a time before "e

Bryan Matsuo 45 Feb 7, 2022
Golang (Go) bindings for GNU's gettext (

gosexy/gettext Go bindings for GNU gettext, an internationalization and localization library for writing multilingual systems. Requeriments GNU gettex

Go toolbelt 61 Jun 6, 2022
agrep-like fuzzy matching, but made faster using Golang and precomputation.

goagrep There are situations where you want to take the user's input and match a primary key in a database. But, immediately a problem is introduced:

Zack 41 Nov 30, 2021
String-matching in Golang using the Knuth–Morris–Pratt algorithm (KMP)

gokmp String-matching in Golang using the Knuth–Morris–Pratt algorithm (KMP). Disclaimer This library was written as part of my Master's Thesis and sh

Patrick-Ranjit D. Madsen 38 Dec 16, 2021
Golang HTML to plaintext conversion library

html2text Converts HTML into text of the markdown-flavored variety Introduction Ensure your emails are readable by all! Turns HTML into raw text, usef

J. Elliot Taylor 408 Jun 18, 2022