An idiomatic Go query builder for ElasticSearch

Related tags

Utilities esquery
Overview

esquery

Build Status

A non-obtrusive, idiomatic and easy-to-use query and aggregation builder for the official Go client for ElasticSearch.

Table of Contents

Description

esquery alleviates the need to use extremely nested maps (map[string]interface{}) and serializing queries to JSON manually. It also helps eliminating common mistakes such as misspelling query types, as everything is statically typed.

Using esquery can make your code much easier to write, read and maintain, and significantly reduce the amount of code you write. Wanna know how much code you'll save? just check this project's tests.

Status

This is an early release, API may still change.

Installation

esquery is a Go module. To install, simply run this in your project's root directory:

go get github.com/aquasecurity/esquery

Usage

esquery provides a method chaining-style API for building and executing queries and aggregations. It does not wrap the official Go client nor does it require you to change your existing code in order to integrate the library. Queries can be directly built with esquery, and executed by passing an *elasticsearch.Client instance (with optional search parameters). Results are returned as-is from the official client (e.g. *esapi.Response objects).

Getting started is extremely simple:

package main

import (
	"context"
	"log"

	"github.com/aquasecurity/esquery"
	"github.com/elastic/go-elasticsearch/v7"
)

func main() {
    // connect to an ElasticSearch instance
    es, err := elasticsearch.NewDefaultClient()
    if err != nil {
        log.Fatalf("Failed creating client: %s", err)
    }

    // run a boolean search query
    res, err := esquery.Search().
        Query(
            esquery.
                Bool().
                Must(esquery.Term("title", "Go and Stuff")).
                Filter(esquery.Term("tag", "tech")),
        ).
        Aggs(
            esquery.Avg("average_score", "score"),
            esquery.Max("max_score", "score"),
        ).
        Size(20).
        Run(
            es,
            es.Search.WithContext(context.TODO()),
            es.Search.WithIndex("test"),
        )
        if err != nil {
            log.Fatalf("Failed searching for stuff: %s", err)
        }

    defer res.Body.Close()

    // ...
}

Notes

  • esquery currently supports version 7 of the ElasticSearch Go client.
  • The library cannot currently generate "short queries". For example, whereas ElasticSearch can accept this:
{ "query": { "term": { "user": "Kimchy" } } }

The library will always generate this:

{ "query": { "term": { "user": { "value": "Kimchy" } } } }

This is also true for queries such as "bool", where fields like "must" can either receive one query object, or an array of query objects. esquery will generate an array even if there's only one query object.

Features

Supported Queries

The following queries are currently supported:

ElasticSearch DSL esquery Function
"match" Match()
"match_bool_prefix" MatchBoolPrefix()
"match_phrase" MatchPhrase()
"match_phrase_prefix" MatchPhrasePrefix()
"match_all" MatchAll()
"match_none" MatchNone()
"multi_match" MultiMatch()
"exists" Exists()
"fuzzy" Fuzzy()
"ids" IDs()
"prefix" Prefix()
"range" Range()
"regexp" Regexp()
"term" Term()
"terms" Terms()
"terms_set" TermsSet()
"wildcard" Wildcard()
"bool" Bool()
"boosting" Boosting()
"constant_score" ConstantScore()
"dis_max" DisMax()

Supported Aggregations

The following aggregations are currently supported:

ElasticSearch DSL esquery Function
"avg" Avg()
"weighted_avg" WeightedAvg()
"cardinality" Cardinality()
"max" Max()
"min" Min()
"sum" Sum()
"value_count" ValueCount()
"percentiles" Percentiles()
"stats" Stats()
"string_stats" StringStats()
"top_hits" TopHits()
"terms" TermsAgg()

Supported Top Level Options

The following top level options are currently supported:

ElasticSearch DSL esquery.Search Function
"highlight" Highlight()
"explain" Explain()
"from" From()
"postFilter" PostFilter()
"query" Query()
"aggs" Aggs()
"size" Size()
"sort" Sort()
"source" SourceIncludes(), SourceExcludes()
"timeout" Timeout()

Custom Queries and Aggregations

To execute an arbitrary query or aggregation (including those not yet supported by the library), use the CustomQuery() or CustomAgg() functions, respectively. Both accept any map[string]interface{} value.

License

This library is distributed under the terms of the Apache License 2.0.

Comments
  • Features: SearchAfter、Term Aggregation Order、Aggregation  Include Filter Values

    Features: SearchAfter、Term Aggregation Order、Aggregation Include Filter Values

    Add some features base on https://github.com/aquasecurity/esquery/pull/12

    SearchAfter: https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#search-after Term Aggregation Order: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-order Aggregation Include Filter:https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_filtering_values_4

    opened by hardy4yooz 2
  • FEATURE: boost support for match statement.

    FEATURE: boost support for match statement.

    Motivation:

    I was wondering, why there is no ability to pass boost in match statement, however it is available for bool and term statements. So I decided to add this behavior.

    Done:

    In this pr I added support for boost inside match statement to omit overcomplicating queries, I also opened related issue and added some examples there.

    Related links

    See related issue by following link.

    opened by PesTospertnyj 1
  • Add support for the

    Add support for the "Delete by Query" API

    This commit adds support for ElasticSearch's Delete by Query API. Usage is very similar to that of Search and Count requests:

    esquery.Delete().
        Index("index_1, "index_2").
        Query(esquery.Bool()...).
        Run(
            es,
            esapi.WithAnalyzeWildcard(true),
        )
    
    opened by ido50 1
  • Support _source, sort and post_filter in search requests

    Support _source, sort and post_filter in search requests

    This PR adds the ability to use the "sort", "_source" and "post_filter" attributes in search requests, via the new methods Sort, SourceIncludes and SourceExcludes.

    For example:

            esquery.Search().
                Query(...).
                Aggs(...).
                Sort("field_1", esquery.OrderAsc).
                Sort("field_2", esquery.OrderDesc).
                PostFilter(esquery.Range(field).Gt(0)).
                SourceIncludes("field_1", "field_2").
                SourceExcludes("field_3").
                Run(...)
    
    opened by ido50 1
  • Sort support

    Sort support

    Hi,

    First thanks for creating this package. Its coming at a good time because I need a flexible query builder given criteria user's input and I found your package was just released.

    One feature I think I need is the ability to sort fields. Looking at the package docs I don't see how I can do that. Would this be a feature you'd consider adding?

    Here are the ES docs: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-body.html#request-body-search-sort

    Thanks!

    opened by dcardamo 1
  • Add support for Count requests

    Add support for Count requests

    This commit adds initial support for Count requests, which are simple search requests asking to get the number of matches for a query.

    The functionality is provided by the Count() function, which accepts a query object (implementing the Mappable interface), and can be executed just like a search request.

    res, err := esquery.
        Count(esquery.Match("user", "someone")).
        Run(es)
    
    opened by ido50 0
  • Add Search() function, two new aggregations

    Add Search() function, two new aggregations

    This commit implements a Search() function, which allow for running search requests with both a query and aggregations. This function is meant to more accurately implement the structure of search requests accepted by ElasticSearch's Search API.

    The Query() and Aggregate() functions are still included by the library, but now simply call Search() internally, making them simple shortcuts.

    Two new aggregations are also added: "terms" and "top_hits". These are implemented a bit differently than previously implemented ones. The structs and methods for ElasticSearch queries and aggregations will eventually be auto-generated from a specification file, and will look more like the new implementations of these new aggregations.

    opened by ido50 0
  • Add documentation, fix small bugs, improve CustomQuery

    Add documentation, fix small bugs, improve CustomQuery

    This commit performs the following modifications:

    • The library is properly documented in Godoc format.

    • The CustomQuery function is made to be a bit more versatile by allowing it to also be used standalone (i.e. instead of passing a CustomQuery as a parameter to the Query function, they now have their own Run method).

    • Queries and aggregations can now also be executed using the RunSearch method. This method is the same as the Run method, except that instead of an *elasticSearch.Client value, it accepts an esapi.Search value. This is provided for consuming code that needs to implement mock clients of ElasticSearch (e.g. for test purposes). The ElasticSearch client does not provide an interface type describing its API, so its Search function (which is actually a field of a function type) can be used instead.

    • Bugfix: the CustomAgg function was unusable as it did not accept a name parameter and thus did not implement the Aggregation interface.

    • Bugfix: the enumeration types are rewritten according to Go standards, and the RangeRelation type's default value is now empty.

    • The golint and godox linters are added.

    opened by ido50 0
  • Bugfix: Run() fails for queries, add MarshalJSON()

    Bugfix: Run() fails for queries, add MarshalJSON()

    The Run() method on the QueryRequest type would fail, since it would encode the inner body of the query to JSON and not the complete, outer body (i.e. including the "body: {}" portion).

    The commit also adds MarshalJSON() methods to both QueryRequest and AggregationRequest, allowing them to implement the json.Marshaller interface, and providing easier debugging of the library. A test skeleton for this is also added.

    opened by ido50 0
  • Support Elasticsearch v8

    Support Elasticsearch v8

    Hello. I propose to support elasticsearch v8. What do you think about renaming module and support tags such as https://github.com/elastic/go-elasticsearch?

    opened by 0legovich 1
  • How to use nested query filter?

    How to use nested query filter?

    Hi all,

    how to make nested query filtering parent based on nested child object, also filter the child to, something query like this?

    {
        "query": {
            "bool": {
                "must": [
                    {
                        "range": {
                            "date": {
                                "lt": "2018-07-05T10:00:00.000Z"
                            }
                        }               
                    },
                    {
                        "nested": {
                            "path": "child",
                            "query": {
                                "bool": {
                                    "must_not": {
                                        "term": {
                                            "child.foo": 0
                                        }       
                                    }
                                }
                            }
                        }
                    }
                ]
            }
        }
    }
    

    thank in advice

    opened by akhmadnurmuhammad 0
  • Consider support/compatibilty for Opensearch

    Consider support/compatibilty for Opensearch

    This library currently depends on go-elasticsearch to implement some of the Search functionality. Without getting into a discussion on the opensearch/elasticsearch fork, what this libary accomplishes with the the query builder component is amazing! Makes writing queries so much better.

    As a user of opensearch, I'd like to be able to continue to use esquery's Query builder.

    Currently while I can accomplish this (code below) I suspect later updates may break it. I'd like to open a discussion on potentially maintaining this feature. I noticed Query returns a full Search object when it's marshaled into JSON, currently this seems to be overridden by the opensearch client/backend. But it'd be great if the integration was a bit cleaner.

    client, err := opensearch.NewDefaultClient()
    query := esquery.Query(
    	esquery.Bool().
    	Filter(
    		esquery.Range("@timestamp").Gt("now"),
    	)
    )
    
    searchResponse, err := client.Search(
    	client.Search.WithContext(ctx),
    	client.Search.WithIndex("foo"),
    	client.Search.WithBody(opensearchutil.NewJSONReader(&query)),
    	client.Search.WithSort("@timestamp:asc"),
    )
    
    

    Thank you for such a great library.

    opened by dekimsey 0
  • Support query string query

    Support query string query

    Hello, Would it be possible to support a query string query with required must fields? References: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html

    opened by mooijtech 2
Releases(v0.2.0)
Owner
Aqua Security
Full lifecycle security for containers and cloud-native applications
Aqua Security
An idiomatic Go implementation of Leaky bucket.

lbucket lbucket is an idiomatic Go leaky bucket implementation. The library make use of plain old Go stdlib; in other words, there are no third-party

Alex Rios 11 Apr 17, 2022
This project contains an example that showcases different features from the official Go Client for Elasticsearch

Elasticsearch for Gophers This project contains an example that showcases different features from the official Go Client for Elasticsearch that you ca

Ricardo Ferreira 19 Oct 12, 2022
A Runtime Struct Builder for Go

A Runtime Struct Builder for Go

null 31 Jul 8, 2022
hdq - HTML DOM Query Language for Go+

hdq - HTML DOM Query Language for Go+ Summary about hdq hdq is a Go+ package for processing HTML documents. Tutorials Collect links of a html page How

Qiniu Cloud 28 Aug 10, 2022
Steampipe plugin to query your Scalingo apps, addons and more

Scalingo plugin for Steampipe Use SQL to query infrastructure including applications and addons from Scalingo. Get started → Documentation: Table defi

François de Metz 11 Nov 4, 2022
Quickly query a Terraform provider's data type.

Terraform Query Quickly query a Terraform provider's data type. Such as a GitHub repository: ➜ ~ tfq github_repository full_name hashicorp/terraform |

Matt Canty 1 Oct 12, 2021
Go-Postgresql-Query-Builder - A query builder for Postgresql in Go

Postgresql Query Builder for Go This query builder aims to make complex queries

Samuel Banks 7 Nov 17, 2022
Data-builder - Data builder with golang

databuilder import "github.com/go-coldbrew/data-builder" Index Variables func Is

Coldbrew 0 Feb 5, 2022
Simple filter query language parser so that you can build SQL, Elasticsearch, etc. queries safely from user input.

fexpr fexpr is a filter query language parser that generates extremely easy to work with AST structure so that you can create safely SQL, Elasticsearc

Gani Georgiev 36 Dec 2, 2022
Go database query builder library for PostgreSQL

buildsqlx Go Database query builder library Installation Selects, Ordering, Limit & Offset GroupBy / Having Where, AndWhere, OrWhere clauses WhereIn /

Arthur Kushman 85 Nov 29, 2022
SQL builder and query library for golang

__ _ ___ __ _ _ _ / _` |/ _ \ / _` | | | | | (_| | (_) | (_| | |_| | \__, |\___/ \__, |\__,_| |___/ |_| goqu is an expressive SQL bu

Doug Martin 1.7k Nov 30, 2022
SQL query builder for Go

GoSQL Query builder with some handy utility functions. Documentation For full documentation see the pkg.go.dev or GitBook. Examples // Open database a

Travis Harmon 26 Sep 26, 2022
Type safe SQL builder with code generation and automatic query result data mapping

Jet Jet is a complete solution for efficient and high performance database access, consisting of type-safe SQL builder with code generation and automa

null 912 Dec 2, 2022
Type safe SQL query builder and struct mapper for Go

sq (Structured Query) ?? ?? sq is a code-generated, type safe query builder and struct mapper for Go. ?? ?? Documentation • Reference • Examples This

null 180 Nov 26, 2022
Fast SQL query builder for Go

sqlf A fast SQL query builder for Go. sqlf statement builder provides a way to: Combine SQL statements from fragments of raw SQL and arguments that ma

Vlad Glushchuk 90 Dec 3, 2022
Simple query builder for MongoDB

?? greenleaf - simple, type safe and easy to use query builder for MongoDB Installation To install use: go get github.com/slavabobik/greenleaf Quick

Slava 75 Nov 27, 2022
A Go SQL query builder and struct mapper.

godb - a Go query builder and struct mapper godb is a simple Go query builder and struct mapper, not a full-featured ORM. godb does not manage relatio

Samuel GAY 717 Nov 13, 2022
dqlx is a fully featured DGraph Schema and Query Builder for Go.

dqlx is a fully featured DGraph Schema and Query Builder for Go. It aims to simplify the interaction with the awesome Dgraph database allowing you to fluently compose any queries and mutations of any complexity. It also comes with a rich Schema builder to easily develop and maintain your Dgraph schema.

Fabrizio 52 Nov 11, 2022
gosq is a parsing engine for a simplicity-focused, template-based SQL query builder for Go.

gosq is a parsing engine for a simplicity-focused, template-based SQL query builder for Go.

Sang-gon Lee 51 Oct 24, 2022
GSQL is a structured query language code builder for golang.

GSQL is a structured query language code builder for golang.

Leon Ding 104 Nov 22, 2022