Command-line JSON processor

Related tags

Command Line jq
Overview

jq

jq is a lightweight and flexible command-line JSON processor.

Coverage Status, Unix: Build Status, Windows: Windows build status

If you want to learn to use jq, read the documentation at https://stedolan.github.io/jq. This documentation is generated from the docs/ folder of this repository. You can also try it online at jqplay.org.

If you want to hack on jq, feel free, but be warned that its internals are not well-documented at the moment. Bring a hard hat and a shovel. Also, read the wiki: https://github.com/stedolan/jq/wiki, where you will find cookbooks, discussion of advanced topics, internals, release engineering, and more.

Source tarball and built executable releases can be found on the homepage and on the github release page, https://github.com/stedolan/jq/releases

If you're building directly from the latest git, you'll need flex, bison (3.0 or newer), libtool, make, automake, and autoconf installed. To get regexp support you'll also need to install Oniguruma or clone it as a git submodule as per the instructions below. (note that jq's tests require regexp support to pass). To build, run:

git submodule update --init # if building from git to get oniguruma
autoreconf -fi              # if building from git
./configure --with-oniguruma=builtin
make -j8
make check

To build without bison or flex, add --disable-maintainer-mode to the ./configure invocation:

./configure --with-oniguruma=builtin --disable-maintainer-mode

(Developers must not use --disable-maintainer-mode, not when making changes to the jq parser and/or lexer.)

To build a statically linked version of jq, run:

make LDFLAGS=-all-static

After make finishes, you'll be able to use ./jq. You can also install it using:

sudo make install

If you're not using the latest git version but instead building a released tarball (available on the website), then you won't need to run autoreconf (and shouldn't), and you won't need flex or bison.

To cross-compile for OS X and Windows, see docs/Rakefile's build task and scripts/crosscompile. You'll need a cross-compilation environment, such as Mingw for cross-compiling for Windows.

Cross-compilation requires a clean workspace, then:

# git clean ...
autoreconf -i
./configure
make distclean
scripts/crosscompile <name-of-build> <configure-options>

Use the --host= and --target= ./configure options to select a cross-compilation environment. See also "Cross compilation" on the wiki.

Send questions to https://stackoverflow.com/questions/tagged/jq or to the #jq channel (http://irc.lc/freenode/%23jq/) on Freenode (https://webchat.freenode.net/).

Comments
  • parsing error on centos platform

    parsing error on centos platform

    Describe the bug

    In a centos environment, the jq binary installed from dnf does not work in the same way as the one downloaded directly from github 1.6 releases page

    To Reproduce

    :information_source: the dockerfile used is at the end of this issue.

    [email protected]:~/$ podman run rootmout/debub-jq-parsing-error-in-centos:v1.0.0 cat entry_file.txt 
    I am not a jsonee line
    {"code":200,"message":"I am a json line","stack":"","level":"info"}
    [email protected]:~/$ podman run rootmout/debub-jq-parsing-error-in-centos:v1.0.0 jq --version
    jq-1.6
    [email protected]:~/$ podman run rootmout/debub-jq-parsing-error-in-centos:v1.0.0 jq -R 'fromjson' entry_file.txt
    jq: error (at entry_file.txt:1): Invalid numeric literal at line 1, column 2 (while parsing 'I am not a jsonee line')
    jq: error (at entry_file.txt:2): Invalid numeric literal at line 1, column 12 (while parsing '{"code":200,"message":"I am a json line","stack":"","level":"info"}')
    [email protected]:~/$ podman run rootmout/debub-jq-parsing-error-in-centos:v1.0.0 ./jq-linux64 --version
    jq-1.6
    [email protected]:~/$ podman run rootmout/debub-jq-parsing-error-in-centos:v1.0.0 ./jq-linux64 -R 'fromjson' entry_file.txt
    jq: error (at entry_file.txt:1): Invalid numeric literal at line 1, column 2 (while parsing 'I am not a jsonee line')
    {
      "code": 200,
      "message": "I am a json line",
      "stack": "",
      "level": "info"
    }
    

    rootmout/debub-jq-parsing-error-in-centos:v1.0.0 contain the binary jq installed by dnf and the ./jq-linux64 installed with wget from github, both are in version 1.6 and use the same file as entry but one installed by dnf fail with a parsing error while other is working fine.

    Expected behavior

    Both binary should work the same way.

    Environment (please complete the following information):

    • OS and Version: CentOS Stream release 8
    • jq version: 1.6

    Additional doc

    Dockerfile used to create debub-jq-parsing-error-in-centos:v1.0.0 image used in "To Reproduce" section:

    FROM quay.io/centos/centos:stream8
    
    RUN dnf -y update -d 10 && \
        dnf -y --nodocs install wget vim jq && \
        dnf clean all
    
    RUN wget https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64
    
    RUN chmod +x jq-linux64
    
    RUN echo -e \
      'I am not a json line\n{"code":200,"message":"I am a json line","stack":"","level":"info"}' \
      > entry_file.txt
    
    opened by rootmout 1
  • Feature request: boolean not operator

    Feature request: boolean not operator

    Sometimes, I just want to select elements in an array based on their boolean attributes, and it gets tiring to write == false all the time.

    It would be nice if I could just do something like:

    .[] | select(.attr1 and !.attr2)
    

    instead of:

    .[] | select(.attr1 and .attr2 == false)
    

    I understand I could do something like:

    .[] | select(.attr2 | not)
    

    but that seems unnecessarily wordy and doesn't look great when incorporated as part of complex boolean expressions:

    .attr1 and ((.attr2 and (.attr3 | not)) | not)
    

    This is difficult to understand rather than:

    .attr1 and !(.attr2 and !.attr3)
    

    I know I can use De Morgan's law to reduce parts of the expression, but I would like it if the tool could lower the cognitive burden.

    Is there a reason why the boolean not operator does not exist in the language? Or am I perhaps missing it?

    opened by alkim0 3
  • Strange behaviour when using two piped try catch error

    Strange behaviour when using two piped try catch error

    Describe the bug

    When having two piped try/catch the catch for the first try seems to be executed even thought its try expression did not throw an error.

    To Reproduce

    # current jq master (cff5336)
    $ jq --version
    jq-1.6-159-gcff5336
    
    $ jq -n 'try null catch error(["catch-a", .]) | try error("error-b") catch error(["catch-b", .])'
    jq: error (at <unknown>) (not a string): ["catch-a",["catch-b","error-b"]]
    

    Expected behavior

    gojq has the behavior i would expect

    $ gojq -n 'try null catch error(["catch-a", .]) | try error("error-b") catch error(["catch-b", .])'
    gojq: error: ["catch-b","error-b"]
    

    Additional context

    Possibly related to #1859

    opened by wader 0
  • Tidy up nomem_handler implementations to fix SEGVs and ensure more consistency between TLS and non-TLS implementations.

    Tidy up nomem_handler implementations to fix SEGVs and ensure more consistency between TLS and non-TLS implementations.

    This fixes #2483 and probably #2056 by making the nomem_handler instance truly thread specific.

    Specifically by:

    • Adding jv_default_nomem_handler which prints a consistent message jq: error: cannot allocate memory in the event of an allocation failure when the nomem handler has not been overriden. This is used as the default value for nomem_handler on jq_state instances given out by jq_init.
    • Splitting tsd_init into two parts: one which is called once per-process and one which is idempotent but called per-thread (possibly many times).
      • tsd_ensure_init_per_thread ensures the nomem_handler block is allocated and setup for the calling thread (allocating it if it isn't).
      • tsd_init_key sets up the TLS key and registers tsd_fini_per_thread to be called atexit to free the nomem_handler block for the main thread.

    Possible problems (that I can see right now):

    • This currently doesn't free the nomem_handler blocks used by any thread other than the one the atexit handlers run on. I'm not sure if this is really a problem but I might be able to fix it with better bookkeeping if necessary.
    • I'm not sure if you want to have the multi-threaded test integrated with make check or quite how it would be best to do this.
    • Trailing whitespace changes - happy to remove these if they're a problem.
    opened by edwardb96 0
  • RFC: Shared-nothing multithreaded usage of libjq.so

    RFC: Shared-nothing multithreaded usage of libjq.so

    Describe the bug Shared-nothing multi-threading crashes/segfaults due to a bug similar / to that described by #2056, the bug is only present when using a library compiled to use the pthreads implementation of thread-local storage (pthread_setspecific, pthread_getspecific and pthread_create_key) USE_TLS must be false. This isn't the default for the stated environment below when compiling manually but is in the distro packages.

    To Reproduce Compile and run the program from this gist https://gist.github.com/edwardb96/47aa1c0b0416846f6292a3818058624d which uses completely separate jq_state and jv instances on separate threads.

    I am not sure to what extent this library usage is supported so I'm not sure if I'm justified in referring to it as a bug. If multi-threaded usage is ill-advised/unsupported even in this specific 'shared-nothing' kind of scenario perhaps its worth making this more explicit in the documentation.

    Expected behavior The program should run to completion, exiting normally with code 0 and outputting the following to stdout

    Running multi-threading tests
    Passed multi-threading test
    

    Environment (please complete the following information):

    • OS and Version: Fedora Linux 36, using standard jq and jq-devel packages.
    • jq version 1.6

    Additional context The documentation / wiki seem to warn off multi-threaded usage due to the non-atomic reference counters in jv values. Despite this I get the impression from some of the pull request comments that this kind of multi-threading is quietly 'supported' - is this a correct assesment?

    For my use-case this kind of 'shared-nothing' multithreading seems to work fine when USE_TLS (or my fix) is enabled but I haven't yet performed extensive testing.

    Anyway, my analysis of the bug is as follows:

    1. The first thread to get there calls jq_compile_args which calls jv_nomem_handler with data=NULL, handler=NULL both of these pointers originate from the jq_state object obtained by jq_init which sets them to NULL by default.
    2. jv_nomem_handler calls tsd_init via pthread_once which sets up the thread specific key, allocates a nomem_handler object and stores a pointer to it in the thread specific storage. This thread happily continues it's business...
    3. The next thread to get there - also via jq_compile_args calls jv_nomem_handler with the same arguments, this time (this being the second call in the same process does not call tsd_init it then proceeds to call pthread_getspecific which gives back NULL since the key does not yet have a value on this thread.
    4. Since nomem_handler is NULL we then attempt to call the passed handler value passed as a parameter - this too is NULL and so we get UB/a crash.

    I am opening a PR with a proposed fix.

    opened by edwardb96 0
Releases(jq-1.6)
  • jq-1.6(Nov 2, 2018)

    New in this release since 1.5:

    • Destructuring Alternation
    • New Builtins:
      • builtins/0
      • stderr/0
      • halt/0, halt_error/1
      • isempty/1
      • walk/1
      • utf8bytelength/1
      • localtime/0, strflocaltime/1
      • SQL-style builtins
      • and more!
    • Add support for ASAN and UBSAN
    • Make it easier to use jq with shebangs (8f6f28c)
    • Add $ENV builtin variable to access environment
    • Add JQ_COLORS env var for configuring the output colors

    Bug fixes:

    • Calling jq without a program argument now always assumes . for the program, regardless of stdin/stdout. (5fe0536)
    • Make sorting stable regardless of qsort. (7835a72)
    • Adds a local oniguruma submodule and the ./configure --with-oniguruma=builtin option to make it easier to build with oniguruma support on systems where you can't install system-level libraries. (c6374b6 and 02bad4b)
    • And much more!
    Source code(tar.gz)
    Source code(zip)
    jq-1.6.tar.gz(1.66 MB)
    jq-1.6.zip(1.84 MB)
    jq-linux32(2.65 MB)
    jq-linux64(3.77 MB)
    jq-osx-amd64(843.78 KB)
    jq-win32.exe(2.58 MB)
    jq-win64.exe(3.36 MB)
  • jq-1.5(Aug 16, 2015)

    Thanks to the 20+ developers who have sent us PRs since 1.4, and the many contributors to issues and the wiki.

    The manual for jq 1.5 can be found at https://stedolan.github.io/jq/manual/v1.5/

    Salient new features since 1.4:

    • regexp support (using Oniguruma)!

    • a proper module system

      import "foo/bar" as bar; # import foo/bar.jq's defs into a bar::* namespace

      and

      include "foo/bar"; # import foo/bar.jq's defs into the top-level

    • destructuring syntax (. as [$first, $second, {$foo, $bar}] | ...)

    • math functions

    • an online streaming parser

    • minimal I/O builtions (inputs, debug)

      One can now write:

      jq -n 'reduce inputs as $i ( ... )'

      to reduce inputs in an online way without having to slurp them first! This works with streaming too.

    • try/catch, for catching and handling errors (this makes for a dynamic non-local exit system)

    • a lexical non-local exit system

      One can now say

      label $foo | ..... | break $foo

      where the break causes control to return to the label $foo, which then produces empty (backtracks). There's named and anonymous labels.

    • tail call optimization (TCO), which allows efficient recursion in jq

    • a variety of new control structure builtins (e.g., while(cond; exp), repeat(exp), until(cond; next)), many of which internally use TCO

    • an enhanced form of reduce: foreach exp as $name (init_exp; update_exp; extract_exp)

    • the ability to read module data files

      import "foo/bar" as $bar; # read foo/bar.json, bind to $bar::bar

    • --argjson var '<JSON text>'

      Using --arg var bit me too many times :)

    • --slurpfile var "filename"

      Replaces the --argfile form (which is now deprecated but remains for backward compatibility).

    • support for application/json-seq (RFC7464)

    • a large variety of new utility functions, many being community contributions (e.g., bsearch, for binary searching arrays)

    • datetime functions

    • a variety of performance enhancements

    • def($a): ...; is now allowed as an equivalent of def(a): a as $a | ...;

    • test and build improvements, including gcov support

    Lastly, don't forget the wiki! The wiki has a lot of new content since 1.4, much of it contributed by the community.

    Source code(tar.gz)
    Source code(zip)
    jq-1.5.tar.gz(721.98 KB)
    jq-1.5.zip(729.88 KB)
    jq-linux32(1.52 MB)
    jq-linux32-no-oniguruma(1.24 MB)
    jq-linux64(2.88 MB)
    jq-osx-amd64(634.76 KB)
    jq-win32.exe(1.16 MB)
    jq-win64.exe(2.21 MB)
  • jq-1.5rc2(Jul 27, 2015)

    Thanks to the 20+ developers who have sent us PRs since 1.4, and the many contributors to issues and the wiki. We're nearing a 1.5 release, finally.

    Salient new features since 1.4:

    • regexp support (using Oniguruma)!

    • a proper module system

      import "foo/bar" as bar; # import foo/bar.jq's defs into a bar::* namespace

      and

      include "foo/bar"; # import foo/bar.jq's defs into the top-level

    • destructuring syntax (. as [$first, $second, {$foo, $bar}] | ...)

    • math functions

    • an online streaming parser

    • minimal I/O builtions (inputs, debug)

      One can now write:

      jq -n 'reduce inputs as $i ( ... )'

      to reduce inputs in an online way without having to slurp them first! This works with streaming too.

    • try/catch, for catching and handling errors (this makes for a dynamic non-local exit system)

    • a lexical non-local exit system

      One can now say

      label $foo | ..... | break $foo

      where the break causes control to return to the label $foo, which then produces empty (backtracks). There's named and anonymous labels.

    • tail call optimization (TCO), which allows efficient recursion in jq

    • a variety of new control structure builtins (e.g., while(cond; exp), repeat(exp), until(cond; next)), many of which internally use TCO

    • an enhanced form of reduce: foreach exp as $name (init_exp; update_exp; extract_exp)

    • the ability to read module data files

      import "foo/bar" as $bar; # read foo/bar.json, bind to $bar::bar

    • --argjson var '<JSON text>'

      Using --arg var bit me too many times :)

    • --slurpfile var "filename"

      Replaces the --argfile form (which is now deprecated but remains for backward compatibility).

    • support for application/json-seq (RFC7464)

    • a large variety of new utility functions, many being community contributions (e.g., bsearch, for binary searching arrays)

    • datetime functions

    • a variety of performance enhancements

    • def($a): ...; is now allowed as an equivalent of def(a): a as $a | ...;

    • test and build improvements, including gcov support

    Lastly, don't forget the wiki! The wiki has a lot of new content since 1.4, much of it contributed by the community.

    Source code(tar.gz)
    Source code(zip)
    jq-1.5rc2.tar.gz(697.31 KB)
    jq-1.5rc2.zip(739.09 KB)
    jq-linux-x86(1.39 MB)
    jq-linux-x86_64(3.04 MB)
    jq-osx-x86_64(634.89 KB)
    jq-win32.exe(1.16 MB)
    jq-win64.exe(2.21 MB)
  • jq-1.5rc1(Jan 1, 2015)

    Salient new features since 1.4:

    • regexp support (using Oniguruma)

    • an online streaming parser

      Included is the ability to control reading of inputs from the jq program, using the new input and inputs builtins.

      Finally we can write:

      jq -n 'reduce inputs as $i ( ... )' # reduce online!

      to reduce inputs without slurping them first. This works with streaming too.

    • try/catch, for catching and handling errors (this makes for a dynamic non-local exit system)

    • a lexical non-local exit system

      Using try/catch to break out of control structures was not a good thing. A lexical mechanism is.

      You can now say

      label $foo | ..... | break $foo

      where the break causes control to return to the label $foo, which then produces empty (backtracks). There's named and anonymous labels.

    • tail call optimization (TCO), which allows efficient recursion in jq

    • a variety of new control structure builtins (e.g., while(cond; exp), repeat(exp), until(cond; next))

    • an enhanced form of reduce: foreach exp as $name (init_exp; update_exp; extract_exp)

    • a proper module system

      import "foo/bar" as bar; # import foo/bar.jq's defs into a bar::* namespace

    • the ability to read module data files

      import "foo/bar" as $bar; # read foo/bar.json, bind to $bar::bar

    • --argjson var '<JSON text>'

      Using --arg var bit me too many times :)

    • --in-place / -i for in-place editing of files

    • support for application/json-seq.

    • a variety of new utility functions, many being community contributions

    • a variety of performance enhancements (e.g., constant folding)

    • def($a): ...; is now allowed as an equivalent of def(a): a as $a | ...;

    Lastly, don't forget the wiki! It has a lot of new content since 1.4, much of it contributed by the community.

    Source code(tar.gz)
    Source code(zip)
    jq-1.5rc1.tar.gz(630.66 KB)
    jq-linux-x86_64-static(3.26 MB)
    jq-win32.exe(1.46 MB)
    jq-win64.exe(1.29 MB)
    jq.1(77.38 KB)
  • jq-1.2(Aug 8, 2015)

  • jq-1.1(Aug 8, 2015)

  • jq-1.0(Aug 8, 2015)

Json-match - Command line util for matching values in a JSON input

json-match Match JSON input by specifying key and value > json-match -json '{\"p

Trond Boksasp 0 Jan 12, 2022
An open-source GitLab command line tool bringing GitLab's cool features to your command line

GLab is an open source GitLab CLI tool bringing GitLab to your terminal next to where you are already working with git and your code without switching

Clement Sam 2.1k Oct 2, 2022
A command line tool that builds and (re)starts your web application everytime you save a Go or template fileA command line tool that builds and (re)starts your web application everytime you save a Go or template file

# Fresh Fresh is a command line tool that builds and (re)starts your web application everytime you save a Go or template file. If the web framework yo

null 0 Nov 22, 2021
A command line tool to prompt for a value to be included in another command line.

readval is a command line tool which is designed for one specific purpose—to prompt for a value to be included in another command line. readval prints

Venky 0 Dec 22, 2021
Query, update and convert data structures from the command line. Comparable to jq/yq but supports JSON, TOML, YAML, XML and CSV with zero runtime dependencies.

dasel Dasel (short for data-selector) allows you to query and modify data structures using selector strings. Comparable to jq / yq, but supports JSON,

Tom Wright 3.7k Sep 25, 2022
🔄 A command-line utility to export Protocol Buffers (proto) files to YAML, and JSON

proto2yaml ?? A command-line utility to export Protocol Buffers (proto) files to YAML, and JSON. Currently supported exports are for: Packages Service

Kristof Kowalski 8 Jul 25, 2022
A simple command line for convert CSV in JSON

C2J A simple command line for convert CSV in JSON list of objects based on header. Install With Go 1.17 or higher: go install github.com/edermanoel94/

Eder Costa 8 Jul 18, 2022
rj is a command line tool show the HTTP Response as JSON

rj rj is a command line tool show the HTTP Response as JSON Installation

Yusuke Wada 43 Aug 24, 2022
Go package to make lightweight ASCII line graph ╭┈╯ in command line apps with no other dependencies.

asciigraph Go package to make lightweight ASCII line graphs ╭┈╯. Installation go get github.com/guptarohit/asciigraph Usage Basic graph package main

Rohit Gupta 2k Sep 22, 2022
git-xargs is a command-line tool (CLI) for making updates across multiple Github repositories with a single command.

Table of contents Introduction Reference Contributing Introduction Overview git-xargs is a command-line tool (CLI) for making updates across multiple

Gruntwork 650 Sep 28, 2022
git-xargs is a command-line tool (CLI) for making updates across multiple GitHub repositories with a single command

git-xargs is a command-line tool (CLI) for making updates across multiple GitHub repositories with a single command. You give git-xargs:

Maxar Infrastructure 1 Feb 5, 2022
Package command provide simple API to create modern command-line interface

Package command Package command provide simple API to create modern command-line interface, mainly for lightweight usage, inspired by cobra Usage pack

chenen 0 Jan 16, 2022
A command line tool for simplified docker volume command built with go

dockervol A command line tool for simplified docker volume command built with go. Features: Remove anonymous volume (beta) Remove volume by matching n

Moh Achun Armando 0 Dec 18, 2021
Watcher - A simple command line app to watch files in a directory for changes and run a command when files change!

Watcher - Develop your programs easily Watcher watches all the files present in the directory it is run from of the directory that is specified while

Geet Sethi 1 Mar 27, 2022
Use the command to convert arbitrary formats to Go Struct (including json, toml, yaml, etc.)

go2struct-tool Use the command to convert arbitrary formats to Go Struct (including json, toml, yaml, etc.) Installation Run the following command und

Afeyer 1 Dec 16, 2021
LINE account link: Sample code for LINE account link

LINE account link: Sample code for LINE account link This is sample code to demostration LINE chatbot account link, refer to document https://develope

null 0 Dec 11, 2021
argv - Go library to split command line string as arguments array using the bash syntax.

Argv Argv is a library for Go to split command line string into arguments array. Documentation Documentation can be found at Godoc Example func TestAr

null 34 Aug 30, 2022
CLI - A package for building command line app with go

Command line interface Screenshot Key features Lightweight and easy to use. Defines flag by tag, e.g. flag name(short or/and long), description, defau

王仕晋 670 Sep 30, 2022
Simple and complete API for building command line applications in Go

Simple and complete API for building command line applications in Go Module cli provides a simple, fast and complete API for building command line app

teris.io 116 Sep 26, 2022