A CLI tool that generates OpenTelemetry Collector binaries based on a manifest.

Overview

OpenTelemetry Collector builder

This program generates a custom OpenTelemetry Collector binary based on a given configuration.

TL;DR

$ go get github.com/open-telemetry/opentelemetry-collector-builder
$ cat > ~/.otelcol-builder.yaml <<EOF
exporters:
  - gomod: "github.com/open-telemetry/opentelemetry-collector-contrib/exporter/alibabacloudlogserviceexporter v0.20.0"
EOF
$ opentelemetry-collector-builder --output-path=/tmp/dist
$ cat > /tmp/otelcol.yaml <<EOF
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317

processors:
  batch:

exporters:
  logging:

service:
  pipelines:
    traces:
      receivers:
      - otlp
      processors: 
      - batch
      exporters:
      - logging
EOF
$ /tmp/dist/otelcol-custom --config=/tmp/otelcol.yaml

Installation

Download the binary for your respective platform under the "Releases" page.

Running

A configuration file isn't strictly required, but the final artifact won't be different than a regular OpenTelemetry Collector. You probably want to specify at least one module (extension, exporter, receiver, processor) to add to your distribution. You can specify them via a configuration file. When no --config flag is provided with the location for the configuration file, ${HOME}/.otelcol-builder.yaml will be used, if available.

$ opentelemetry-collector-builder --config config.yaml

Use opentelemetry-collector-builder --help to learn about which flags are available.

Configuration

The configuration file is composed of two main parts: dist and module types. All dist options can be specified via command line flags:

$ opentelemetry-collector-builder --name="my-otelcol"

The module types are specified at the top-level, and might be: extensions, exporters, receivers and processors. They all accept a list of components, and each component is required to have at least the gomod entry. When not specified, the import value is inferred from the gomod. When not specified, the name is inferred from the import.

The import might specify a more specific path than what is specified in the gomod. For instance, your Go module might be gitlab.com/myorg/myrepo and the import might be gitlab.com/myorg/myrepo/myexporter.

The name will typically be omitted, except when multiple components have the same name. In such case, set a unique name for each module.

Optionally, a list of go mod replace entries can be provided, in case custom overrides are needed. This is typically necessary when a processor or some of its transitive dependencies have dependency problems.

dist:
    module: github.com/open-telemetry/opentelemetry-collector-builder # the module name for the new distribution, following Go mod conventions. Optional, but recommended.
    name: otelcol-custom # the binary name. Optional.
    description: "Custom OpenTelemetry Collector distribution" # a long name for the application. Optional.
    include_core: true # whether the core components should be included in the distribution. Optional.
    otelcol_version: "0.20.0" # the OpenTelemetry Collector version to use as base for the distribution. Optional.
    output_path: /tmp/otelcol-distributionNNN # the path to write the output (sources and binary). Optional.
    version: "1.0.0" # the version for your custom OpenTelemetry Collector. Optional.
    go: "/usr/bin/go" # which Go binary to use to compile the generated sources. Optional.
exporters:
  - gomod: "github.com/open-telemetry/opentelemetry-collector-contrib/exporter/alibabacloudlogserviceexporter v0.20.0" # the Go module for the component. Required.
    import: "github.com/open-telemetry/opentelemetry-collector-contrib/exporter/alibabacloudlogserviceexporter" # the import path for the component. Optional.
    name: "alibabacloudlogserviceexporter" # package name to use in the generated sources. Optional.
    path: "./alibabacloudlogserviceexporter" # in case a local version should be used for the module, the path relative to the current dir, or a full path can be specified. Optional.
replaces:
  # a list of "replaces" directives that will be part of the resulting go.mod
  - github.com/open-telemetry/opentelemetry-collector-contrib/internal/common => github.com/open-telemetry/opentelemetry-collector-contrib/internal/common v0.20.0
Comments
  • Unable to build binary due to missing go.sum

    Unable to build binary due to missing go.sum

    I am using 0.7.0 installed via go get and Go 1.16.

    My config file is:

    exporters:
      - gomod: "github.com/open-telemetry/opentelemetry-collector-contrib/exporter/newrelicexporter v0.21.0"
    

    Upon executing the builder using opentelemetry-collector-builder --output-path=/tmp/dist --config=.otelcol-builder.yaml, I get the following error:

    2021-03-05T15:50:35.630+1100    INFO    cmd/root.go:82  OpenTelemetry Collector distribution builder    {"version": "dev", "date": "unknown"}
    2021-03-05T15:50:35.630+1100    INFO    cmd/root.go:98  Using config file       {"path": ".otelcol-builder.yaml"}
    2021-03-05T15:50:35.633+1100    INFO    builder/main.go:80      Sources created {"path": "/tmp/dist"}
    2021-03-05T15:50:35.666+1100    INFO    builder/main.go:94      Using go from PATH      {"Go executable": "/usr/local/go/bin/go"}
    2021-03-05T15:50:35.666+1100    INFO    builder/main.go:97      Compiling
    Error: failed to compile the OpenTelemetry Collector distribution: exit status 1. Output: "go: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]: missing go.sum entry; to add it:\n\tgo mod download github.com/open-telemetry/opentelemetry-collector-contrib/exporter/newrelicexporter\n"
    Usage:
      opentelemetry-collector-builder [flags]
    
    Flags:
          --config string            config file (default is $HOME/.otelcol-builder.yaml)
          --description string       A descriptive name for the OpenTelemetry Collector distribution (default "Custom OpenTelemetry Collector distribution")
          --go string                The Go binary to use during the compilation phase (default "/usr/bin/go")
      -h, --help                     help for opentelemetry-collector-builder
          --include-core             Whether the core components should be included in the distribution (default true)
          --module string            The Go module for the new distribution (default "github.com/jpkroehling/opentelemetry-collector-builder")
          --name string              The executable name for the OpenTelemetry Collector distribution (default "otelcol-custom")
          --otelcol-version string   Which version of OpenTelemetry Collector to use as base (default "0.20.0")
          --output-path string       Where to write the resulting files (default "/tmp/otelcol-distribution776575606")
          --version string           The version for the OpenTelemetry Collector distribution (default "1.0.0")
    
    2021-03-05T15:50:35.702+1100    ERROR   cmd/root.go:77  failed to run   {"error": "failed to compile the OpenTelemetry Collector distribution: exit status 1. Output: \"go: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]: missing go.sum entry; to add it:\\n\\tgo mod download github.com/open-telemetry/opentelemetry-collector-contrib/exporter/newrelicexporter\\n\""}
    github.com/open-telemetry/opentelemetry-collector-builder/cmd.Execute
            /c/Users/User/go/pkg/mod/github.com/open-telemetry/[email protected]/cmd/root.go:77
    main.main
            /c/Users/User/go/pkg/mod/github.com/open-telemetry/[email protected]/main.go:22
    runtime.main
            /usr/local/go/src/runtime/proc.go:225
    

    From my understanding, the builder should take care of initializing the go.mod and go.sum files and user intervention is not required.

    enhancement good first issue 
    opened by F21 8
  • Backward compatibility issue

    Backward compatibility issue

    In https://github.com/open-telemetry/opentelemetry-collector-builder/commit/c9fe6966eba55a28368caccb589b9c913adba7e5 , the template file internal/scaffold/components.go has been changed to use the the new API, and hence the builder does not work for earlier versions

    opened by fangyi-zhou 6
  • Generate custom service code for Windows

    Generate custom service code for Windows

    This pull request pulls in the latest versions of the main*.go files so that running the generated collector as a Windows service works (fixes #73). The code has been pulled from upstream tag 0.35.0; there are further minor changes in 0.36.0 that are not yet compatible with the current code base.

    All of the current unit tests pass, and the change has been checked manually with a Windows service deployment.

    opened by loomis-relativity 5
  • Version & Date not updated by releaser

    Version & Date not updated by releaser

    https://github.com/open-telemetry/opentelemetry-collector-builder/blob/v0.28.0/cmd/root.go#L28

    It remains dev for released versions, making the version command not very informative

    bug 
    opened by fangyi-zhou 4
  • Verify whether the latest changes to the OpenTelemetry Collector require a change to the builder

    Verify whether the latest changes to the OpenTelemetry Collector require a change to the builder

    There were a few changes to the OpenTelemetry Collector that seems like would affect the builder, such as: https://github.com/open-telemetry/opentelemetry-collector/pull/3056

    Before releasing 0.26.0, we need to make sure that those changes are accounted for.

    enhancement good first issue help wanted 
    opened by jpkrohling 4
  • Failed Collector build does not exit with error code

    Failed Collector build does not exit with error code

    When running our pipeline for our opentelemetry collector build, a failure did not cause our pipeline to fail.

    I think this is due to no errors being passed backup to main. Is there a reason why errors cannot be passed back to the main package and the run of the build return non zero exit code if there was an issue.

    If not then I'll create a PR for proposal.

    bug good first issue 
    opened by eroteme 4
  • Account for go mod download in go1.17 not updating go.sum

    Account for go mod download in go1.17 not updating go.sum

    Because the behaviour of go mod download changes in go1.17 we need to account for that. For previous versions this will result in the same behaviour.

    go mod download When go mod download is invoked without arguments, it will no longer save sums for downloaded module content to go.sum. It may still make changes to go.mod and go.sum needed to load the build list. This is the same as the behavior in Go 1.15. To save sums for all modules, use go mod download all.

    ref: https://tip.golang.org/doc/go1.17

    opened by pmalek-sumo 3
  • Download go modules before building

    Download go modules before building

    Fixes #14

    Since go 1.16 build no longer updates go.sum and so we get errors such as:

    go: github.com/open-telemetry/opentelemetry-collector-contrib/processor/[email protected]: missing go.sum entry; to add it:\n\tgo mod download github.com/open-telemetry/opentelemetry-collector-contrib/processor/routingprocessor\n"
    

    This change runs go mod download to update go sum before the compile

    opened by eroteme 3
  • Build Collector binary on demond

    Build Collector binary on demond

    Is your feature request related to a problem? Please describe. The current Collector binary includes all of components, Collector-contrib includes more 3rd party receivers and exporters the binary size is over 100MB and would be larger in the future. For some resource sensitive environment like AWS Lambda, it calculates memory usage by Cache + RSS, the larger size of Collector the higher bill users may have to pay. So, it is better to provide a way to build Collector based on user's config. For example, if user want to use otlp receiver and xray exporter, he can build a small Collector with only these 2 components besides the core components.

    Describe the solution you'd like Since goLang is a compiled language, dynamically build binary is a challenge. A possible way but not safe is can we generate defaults.go by a template? The components are got from user's config.

    There is a 3rd party repo https://github.com/observatorium/opentelemetry-collector-builder, can we think about merge the logic into Collector?

    Describe alternatives you've considered Now I can manually delete the components what I don't want from defaults.go, but that is not a general solution for normal user.

    opened by wangzlei 3
  • Feature Request: Allow modules to be included based on golang build tags

    Feature Request: Allow modules to be included based on golang build tags

    For example, a journaldlogreceiver is being added to the contrib collector. This component will only have a meaningful implementation for Linux builds.

    Without this feature, such components must rely on dummy implementations. In the case of journaldreceiver, such an implementation will simply return an error stating that the receiver is not available on non-Linux distributions.

    opened by djaglowski 2
  • Admin rights for jpkrohling

    Admin rights for jpkrohling

    I would like to request admin rights for this repository. My current need is to enable a set of checks, making them required before merging, such as the successful execution of e2e tests, and I cannot do that as a maintainer only.

    cc @open-telemetry/collector-maintainers

    opened by jpkrohling 2
Releases(v0.37.0)
Owner
OpenTelemetry - CNCF
OpenTelemetry makes robust, portable telemetry a built-in feature of cloud-native software.
OpenTelemetry - CNCF
This is a Go Cli app that receives an string path to a log file, and based on it generates and prints in console an encoded polyline with the locations found in the log file.

GEOENCODE GO CLI APP DESCRIPTION This is a Go Cli app that receives an string path to a log file, and based on it generates and prints in console an e

Jose Luis Ojeda 1 Oct 1, 2021
linenoise-classic is a command-line tool that generates strings of random characters that can be used as reasonably secure passwords.

linenoise-classic is a command-line tool that generates strings of random characters that can be used as reasonably secure passwords.

Mark Cornick 0 Aug 21, 2022
Download and install binaries from GitHub Releases, interactively.

bget Download and install binary files from GitHub Releases. Preview Install For Mac/Linux users, you can use Homebrew to install it: brew install ego

EGOIST 216 Aug 24, 2022
Nebula Diagnosis CLI Tool is an information diagnosis cli tool for the nebula service and the node to which the service belongs.

Nebula Diagnosis CLI Tool is an information diagnosis cli tool for the nebula service and the node to which the service belongs.

Katz 1 Jan 12, 2022
:zap: boilerplate template manager that generates files or directories from template repositories

Boilr Are you doing the same steps over and over again every time you start a new programming project? Boilr is here to help you create projects from

Tamer Tas 1.5k Nov 23, 2022
Reads from existing Cloud Providers (reverse Terraform) and generates your infrastructure as code on Terraform configuration

TerraCognita Imports your current Cloud infrastructure to an Infrastructure As Code Terraform configuration (HCL) or/and to a Terraform State. At Cycl

Cycloid 1.4k Nov 28, 2022
A sample FaaS function that gets a stock quote and 30 day history by symbol and returns a HTML page with a generates SVG sparkline.

faas stonks This uses serverless technology to get a stock quote and 30 day sparkline from Yahoo Finance. Deployment Nimbella account Namespace with o

Jon Friesen 2 Sep 23, 2021
Small CLI based programs for solving structural engineering design problems based on the book 'Structural Concrete'

Small CLI based programs for solving structural engineering design problems based on the book 'Structural Concrete' written by M. Nadim Hassoun and Akhtem Al-Manaseer (edition-6)

Muhammad Usman Shamsi 1 Nov 26, 2021
Symfony-cli - The Symfony CLI tool For Golang

Symfony CLI Install To install Symfony CLI, please download the appropriate vers

Symfony CLI 387 Nov 22, 2022
bcrypt-cli is the CLI tool for hashing passwords with bcrypt.

bcrypt-cli bcrypt-cli is the CLI tool for hashing passwords with bcrypt. Install go install github.com/ryicoh/bcrypt-cli Usage It can be used like bas

Ryuichiroh Ikeuchi 1 Sep 11, 2022
GTDF-CLI - The official CLI tool to operate with Getting Things Done Framework

This is the official CLI tool to operate with Getting Things Done Framework. How

akrck02 1 Feb 14, 2022
A dead simple CLI tool that prints the next semantic version based on the last tag of your git repository

nextver A dead simple CLI tool that prints the next semantic version based on the last tag of your git repository. Install go install github.com/junk1

Tom 2 Sep 29, 2022
Branch is a small CLI tool to automatically create git branches based on tickets

________ ________ ________ ________ ________ ___ ___ |\ __ \ |\ __ \ |\ __ \ |\ ___ \ |\ ___

Maikel Veen 5 Nov 15, 2022
CLI tool for interacting with Nostr-based wikis

nwiki Command line client for the Nostr wiki. Run in container, without installation If you don't have go installed or don't like installing stuff jus

fiatjaf 7 Oct 30, 2022
This tool is a CLI-interactive tool for TA who use eeclass platform

NTHU eeclass TA helper. This tool is a CLI-interactive tool for TA who use eeclass platform. It helps TA to download all the submitted homework, and use CSV to record the score and comment, and upload CSV score directly to the eeclass platform with just 2 Enter key!

Bo-Wei Chen 1 Dec 11, 2021
Elegant CLI wrapper for kubeseal CLI

Overview This is a wrapper CLI ofkubeseal CLI, specifically the raw mode. If you just need to encrypt your secret on RAW mode, this CLI will be the ea

Elm 4 Jan 8, 2022
CLI to run a docker image with R. CLI built using cobra library in go.

BlueBeak Installation Guide Task 1: Building the CLI The directory structure looks like Fastest process: 1)cd into bbtools 2)cd into bbtools/bin 3)I h

Aniruddha Chattopadhyay 0 Dec 20, 2021
A wrapper of aliyun-cli subcommand alidns, run aliyun-cli in Declarative mode.

aliyun-dns A wrapper of aliyun-cli subcommand alidns, run aliyun-cli in Declarative mode. Installation Install aliyun-cli. Usage $ aliyun-dns -h A wra

许嘉华 0 Dec 21, 2021
Go-file-downloader-ftctl - A file downloader cli built using golang. Makes use of cobra for building the cli and go concurrent feature to download files.

ftctl This is a file downloader cli written in Golang which uses the concurrent feature of go to download files. The cli is built using cobra. How to

Dipto Chakrabarty 2 Jan 2, 2022