Frep - Generate file using template from environment, arguments, json/yaml/toml config files

Overview

Build Status License

frep

Generate file using template from environment, arguments, json/yaml/toml config files.

NAME:
   frep - Generate file using template

USAGE:
   frep [options] input-file[:output-file] ...

VERSION:
   1.3.x

AUTHORS:
   Guoqiang Chen <[email protected]>

OPTIONS:
   -e, --env name=value    set variable name=value, can be passed multiple times
       --json jsonstring   load variables from json object string
       --load file         load variables from json/yaml/toml file
       --no-sys-env        exclude system environments, default false
       --overwrite         overwrite if destination file exists
       --dryrun            just output result to console instead of file
       --strict            exit on any error during template processing
       --delims value      template tag delimiters (default: {{:}})
       --missing           handling of missing vars, one of: default/invalid, zero, error (default: default)
       --help              print this usage
       --version           print version information

EXAMPLES:
   frep nginx.conf.in -e webroot=/usr/share/nginx/html -e port=8080
   frep nginx.conf.in:/etc/nginx.conf -e webroot=/usr/share/nginx/html -e port=8080
   frep nginx.conf.in --json '{"webroot": "/usr/share/nginx/html", "port": 8080}'
   frep nginx.conf.in --load config.json --overwrite
   echo "{{ .Env.PATH }}"  | frep -

Downloads

v1.3.12 Release: https://github.com/subchen/frep/releases/tag/v1.3.12

  • Linux

    curl -fSL https://github.com/subchen/frep/releases/download/v1.3.12/frep-1.3.12-linux-amd64 -o /usr/local/bin/frep
    chmod +x /usr/local/bin/frep
    
    # centos / redhat
    yum install https://github.com/subchen/frep/releases/download/v1.3.12/frep-1.3.12-74.x86_64.rpm
    
    # ubuntu
    curl -fSL https://github.com/subchen/frep/releases/download/v1.3.12/frep_1.3.12-74_amd64.deb -o frep_1.3.12-74_amd64.deb
    dpkg -i frep_1.3.12-74_amd64.deb
    
  • macOS

    brew install subchen/tap/frep
    
  • Windows

    wget https://github.com/subchen/frep/releases/download/v1.3.12/frep-1.3.12-windows-amd64.exe
    

Docker

You can run frep using docker container

docker run -it --rm subchen/frep --help

Examples

Load template variables

  • Load from environment

    export webroot=/usr/share/nginx/html
    export port=8080
    frep nginx.conf.in
    
  • Load from arguments

    frep nginx.conf.in -e webroot=/usr/share/nginx/html -e port=8080
    
  • Load from JSON String

    frep nginx.conf.in --json '{"webroot": "/usr/share/nginx/html", "port": 8080}'
    
  • Load from JSON file

    cat > config.json << EOF
    {
      "webroot": "/usr/share/nginx/html",
      "port": 8080,
      "servers": [
        "127.0.0.1:8081",
        "127.0.0.1:8082"
      ]
    }
    EOF
    
    frep nginx.conf.in --load config.json
    
  • Load from YAML file

    cat > config.yaml << EOF
    webroot: /usr/share/nginx/html
    port: 8080
    servers:
      - 127.0.0.1:8081
      - 127.0.0.1:8082
    EOF
    
    frep nginx.conf.in --load config.yaml
    
  • Load from TOML file

    cat > config.toml << EOF
    webroot = /usr/share/nginx/html
    port = 8080
    servers = [
       "127.0.0.1:8081",
       "127.0.0.1:8082"
    ]
    EOF
    
    frep nginx.conf.in --load config.toml
    

Input/Output

  • Input from file

    // input file: nginx.conf
    frep nginx.conf.in
    
  • Input from console(stdin)

    // input from stdin pipe
    echo "{{ .Env.PATH }}" | frep -
    
  • Output to default file (Removed last file ext)

    // output file: nginx.conf
    frep nginx.conf.in --overwrite
    
  • Output to the specified file

    // output file: /etc/nginx.conf
    frep nginx.conf.in:/etc/nginx.conf --overwrite -e port=8080
    
  • Output to console(stdout)

    frep nginx.conf.in --dryrun
    frep nginx.conf.in:-
    
  • Output multiple files

    frep nginx.conf.in redis.conf.in ...
    

Template

Templates use Golang text/template.

You can access environment variables within a template

Env.PATH = {{ .Env.PATH }}

If your template file uses {{ and }} as part of it's syntax, you can change the template escape characters using the --delims.

frep --delims "<%:%>" ...

There are some built-in functions as well: Masterminds/sprig v2.22.0

More funcs added:

  • toJson
  • toYaml
  • toToml
  • toBool
  • fileSize
  • fileLastModified
  • fileGetBytes
  • fileGetString
  • fileExists
  • include
  • countRune
  • pipeline compatible regex functions from sprig
    • reReplaceAll
    • reReplaceAllLiteral
    • reSplit
  • awsSecret
  • awsParameterStore

Sample of nginx.conf.in

server {
    listen {{ .port }} default_server;

    root {{ .webroot | default "/usr/share/nginx/html" }};
    index index.html index.htm;

    location /api {
        {{ include "shared/log.nginx" | indent 8 | trim }}
        proxy_pass http://backend;
    }
}

upstream backend {
    ip_hash;
{{- range .servers }}
    server {{.}};
{{- end }}
}

Sample using secrets, first of all take into account that in order to use the secret functionality you need to have a proper AWS configuration in place and permissions enough to read secrets from AWS Secrets Manager. More details of how to configure AWSCLI can be found at https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html

Once you have all the requirements just create a template like this one:

# application.conf
mysql_host: {{ .mysql_host }}
mysql_user: {{ .mysql_user }}
mysql_pass: {{ awsSecret "application/mysql/password" }}

In above example mysql_host and mysql_user will be filled as usual by using frep config file or environment variables but mysql_pass will be fetch straight from AWS Secrets Manager by looking at secret name application/mysql/password

If you have multiple items in a single secret you can retrieve an specific key by specifying the key you want in template, for example:

# application.conf
mysql_host: {{ .mysql_host }}
mysql_user: {{ .mysql_user }}
mysql_pass: {{ awsSecret "application/mysql/password" }}

external_api_client: {{ awsSecret "application/external_api" "client_id" }}
external_api_secret: {{ awsSecret "application/external_api" "secret_key" }}

Sample using AWS Parameter Store, first of all take into account that in order to use the ssm functionality you need to have a proper AWS configuration in place and permissions enough to read parameters from AWS Parameter Store. More details of how to configure AWSCLI can be found at https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html

Once you have all the requirements just create a template like this one:

# application.conf
mysql_host: {{ .mysql_host }}
mysql_user: {{ .mysql_user }}
mysql_pass: {{ awsSecret "application/mysql/password" }}
mysql_dns: {{ awsParameterStore "application/mysql/dns" }}

In above example mysql_dns will be filled as usual by using frep config file or environment variables but mysql_pass will be fetch straight from AWS Parameter Store by looking at application/mysql/dns

SSM Limitation: You can get parameter from ParameterStore just in textplain.

Comments
  • func of count runes of string

    func of count runes of string

    I do know this is NOT a bug of frep itself.

    template :

    {{ len "中文" }}
    

    if i run frep like this:

    frep /path/to/template --dryrun
    

    i got :

    6
    

    So, I think chines users need a new func.

    How about:

    
    func CountRune(s string) int {
        return len([]rune(s))
    }
    

    and register is:

    funcMap := template.FuncMap() {
        "countRune": CountRune,
        // other funcs
    }
    
    opened by yingzhuo 5
  • Please preserve file permissions

    Please preserve file permissions

    File perms (specifically +x ) are lost when a template file is run though frep. Please preserve originating file permissions when writing out the rendered template.

    opened by jaisingh 4
  • Containerize application using Drone

    Containerize application using Drone

    Closes #3.

    Also changed the versioning setup so that it doesn't rely on a tag to populate other version info.

    You can own this by:

    • Modifying the repo: tonglil/frep to your own user on Docker hub
    • Sign up for Drone at https://beta.drone.io/
    • Activate this repo at https://beta.drone.io/subchen/frep
    • Add your Docker hub login info as docker_username and docker_password in https://beta.drone.io/subchen/frep/secrets
    opened by tonglil 3
  • Unable to install on macOS : Invalid formula

    Unable to install on macOS : Invalid formula

    Error on installing frep using homebrew.

    System setup: macOS Monterey Version 12.4

    % brew install subchen/tap/frep ==> Tapping subchen/tap Cloning into '/usr/local/Homebrew/Library/Taps/subchen/homebrew-tap'... remote: Enumerating objects: 180, done. remote: Counting objects: 100% (32/32), done. remote: Compressing objects: 100% (25/25), done. remote: Total 180 (delta 14), reused 19 (delta 7), pack-reused 148 Receiving objects: 100% (180/180), 32.90 KiB | 1.37 MiB/s, done. Resolving deltas: 100% (65/65), done. Error: Invalid formula: /usr/local/Homebrew/Library/Taps/subchen/homebrew-tap/Formula/gometalinter.rb gometalinter: wrong number of arguments (given 1, expected 0) Error: Cannot tap subchen/tap: invalid syntax in tap!

    opened by olgaakrekhovetska 2
  • Feature Request: Allow to define default value

    Feature Request: Allow to define default value

    Would it be possible to write a template file like this?

    something = {{ Env.FOO, "bar" }}
    

    which will populate the placeholder with bar if the env variable FOO does not exist

    opened by sgohl 2
  • Is it a bug or what ?

    Is it a bug or what ?

    Hi,

    I am using a template as below:

    作者:
        姓名: {{ .author.name }}
        邮件: {{ .author.email }}
    
    frep /path/to/template -e "author.name=应卓" -e "[email protected]" --dryrun
    

    output:

    作者:
        姓名: <no value>
        邮件: <no value>
    

    I was wondering maybe if the key contains . will cause golang's template rendering bug.

    frep version:

    Name:       frep
    Version:    1.3.7
    Patches:    68
    Git branch: tags/v1.3.7
    Git commit: 9eca7e8f14390b28f08255bb34d987dffd3f9b10
    Built:      Tue, 19 Mar 2019 07:11:21 +0000
    Go version: go1.11
    OS/Arch:    darwin/amd64
    
    opened by yingzhuo 2
  • How to access environment inside the range?

    How to access environment inside the range?

    Hi there,

    I want to access .ENV.Secret inside the range. How do I do that?

    {{- range (splitList "," .Env.XMPP_JVBS) }}
    Component "{{.}}"
        component_secret "{{.ENV.Secret}}"
    {{- end }}
    

    Thanks!

    opened by markmadlangbayan 2
  • Build on ppc64le fails to generate correct artifacts

    Build on ppc64le fails to generate correct artifacts

    The build is generating Intel binaries

    [[email protected] frep]$ make build rm -rf frep ./_releases ./_build go fmt ./... CGO_ENABLED=0 GOOS=linux GOARCH=amd64
    go build -a -installsuffix cgo -ldflags "-s -w -X 'main.BuildVersion=1.3.11' -X 'main.BuildGitBranch=heads/master' -X 'main.BuildGitRev=166' -X 'main.BuildGitCommit=e325a0c21a6db33f763aa8e1f7ea7bd65402661a' -X 'main.BuildDate=Tue, 08 Sep 2020 13:06:33 +0000'" -o _releases/frep-1.3.11-linux-amd64 [[email protected] frep]$ find . -name frep-1.3.11-linux-amd64 ./_releases/frep-1.3.11-linux-amd64

    [[email protected] frep]$ file ./_releases/frep-1.3.11-linux-amd64 ./_releases/frep-1.3.11-linux-amd64: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped

    opened by seth-priya 1
  • Change secretsmanager package in go.mod

    Change secretsmanager package in go.mod

    Hi @subchen

    I can notice you just merged my previous PR, thank you so much about that.

    I just like to add a simple change as I've changed the tag version of the secretsmanager package I am using (https://github.com/overdrive3000/secretsmanager).

    It would be great if you merge this.

    Thank you.

    opened by overdrive3000 1
  • [WIP] Add template function to fetch secrets from AWS Secrets Manager

    [WIP] Add template function to fetch secrets from AWS Secrets Manager

    This PR add a custom function that is added to the FuncMap map in order to be able to fetch secrets from AWS Secrets Manager.

    I found frep a great tool for generating templates it is simple yet flexible a powerful, but I've been strungled a little bit of not having the possiblity to fetch secrets without the need of using "external tricks".

    Hope this PR likes you and become part of the main frep project, thank you so much for this tool.

    opened by overdrive3000 1
  • Added option for specifying behaviour on missing vars

    Added option for specifying behaviour on missing vars

    Added command line option --missing for specifying behaviour on missing variables using template options from Go.

    Default option is "default", which does not change the current behaviour.

    More info on template options: https://golang.org/pkg/text/template/#Template.Option

    opened by leonardolang 1
  • Do not use environment variables by default - security issue

    Do not use environment variables by default - security issue

    Please exclude support for environment variables in the templates (the .Env.* insertions according to the documentation). This can expose a lot of unintentional information (just run set in your shell to see what is available).

    Preferred behaviour:

    1. Change default value of --no-sys-env to true (or rename option).
    2. Remove all support for system environments and only accept definitions from --env, --json or --load.

    Simple work around is to add --no-sys-env parameter.

    opened by FlipSky 1
  • `--missing error` does not work with 'include'

    `--missing error` does not work with 'include'

    Missing vars in included file does not rise error with --missing error.

    $ more a.in b.in | cat
    ::::::::::::::
    a.in
    ::::::::::::::
    {{- include "b.in" -}}
    ::::::::::::::
    b.in
    ::::::::::::::
    var1={{.var1}}
    $ frep  --no-sys-env --missing error  --overwrite b.in:b.out 
    fatal: render template error, caused:
    
       template: b.in:1:7: executing "b.in" at <.var1>: map has no entry for key "var1"
    
    $ frep  --no-sys-env --missing error --overwrite a.in:a.out 
    $ cat a.out 
    var1=<no value>
    $ 
    
    opened by n-arakawa 0
  • Please use uptodated 'Masterminds/sprig'

    Please use uptodated 'Masterminds/sprig'

    version info: frep-1.3.10-linux-amd64

    According to git.mod, frep now uses 'Masterminds/sprig' of V2.22.

    github.com/Masterminds/sprig v2.22.0+incompatible

    I hope to have a 'get' dict function which is available from V3 of 'Masterminds/sprig'.

    I always use frep with '--missing error'. With 'get', I can say somthing like

    {{ if eq (get . "optional_value") "ON" }}

    instead of

    {{ if hasKey . "optional_value" }} {{ if eq .optional_value "ON" }}

    opened by n-arakawa 3
Releases(v1.3.13)
Owner
Guoqiang Chen
Guoqiang Chen
`runenv` create gcloud run deploy `--set-env-vars=` option and export shell environment from yaml file.

runenv runenv create gcloud run deploy --set-env-vars= option and export shell environment from yaml file. Motivation I want to manage Cloud Run envir

sonatard 0 Feb 10, 2022
AWS environment config loader

awsenv AWS environment config loader. awsenv is a small binary that loads AWS environment variables for an AWS profile from ~/.aws/credentials - usefu

Sonia Hamilton 32 Nov 28, 2022
Dotnet-appsettings-env - Convert .NET appsettings.json file to Kubernetes, Docker and Docker-Compose environment variables

dotnet-appsettings-env Convert .NET appsettings.json file to Kubernetes, Docker

Daniel Dias de Assumpção 17 Nov 30, 2022
Dotnet-appsettings-env - Convert .NET appsettings.json file to Kubernetes, Docker and Docker-Compose environment variables

dotnet-appsettings-env Convert .NET appsettings.json file to Kubernetes, Docker

Daniel Dias de Assumpção 1 Feb 16, 2022
Envoy file based dynamic routing using kubernetes config map

Envoy File Based Dynamic Routing Config mapを使用してEnvoy File Based Dynamic Routingを実現します。 概要 アーキテクチャとしては、 +----------+ +--------------+ +-----------

null 1 Mar 1, 2022
A handy utility to generate configmap and values.yaml of your application for helmifying them

Helmfig Are you tired of writing values.yaml for configmap of your project when you are helmifying them? Helmfig is a handy tool that can generate the

Snapp Cab Incubators 24 Nov 20, 2022
go-opa-validate is an open-source lib that evaluates OPA (open policy agent) policy against JSON or YAML data.

go-opa-validate go-opa-validate is an open-source lib that evaluates OPA (open policy agent) policy against JSON or YAML data. Installation Usage Cont

chenk 6 Nov 17, 2022
How you can use Go to replace YAML files with Kubernetes.

YamYams A small project that is free to use. ?? I wanted to offer a starting point for anyone interested in replacing YAML with Go. You can read more

Kris Nóva 1.2k Nov 27, 2022
kubectl plugin for signing Kubernetes manifest YAML files with sigstore

k8s-manifest-sigstore kubectl plugin for signing Kubernetes manifest YAML files with sigstore ⚠️ Still under developement, not ready for production us

sigstore 70 Nov 28, 2022
Provider-template - Template for writing providers for crossplane

provider-template provider-template is a minimal Crossplane Provider that is mea

maros.zeleny 0 Feb 3, 2022
Prestic - Lets you define and run restic commands from a YAML file

Pete's Restic Lets you define and run restic commands from a YAML file. Features

Pete Taylor 0 Jan 10, 2022
Library/tool to change a yaml given a rules file

golang-yaml-rules/yaml-transform Library/tool to change a yaml given a rules file Using jsonpath ( https://github.com/vmware-labs/yaml-jsonpath ), thi

null 0 Feb 11, 2022
A tool to build, deploy, and release any environment using System Containers.

Bravetools Bravetools is an end-to-end System Container management utility. Bravetools makes it easy to configure, build, and deploy reproducible envi

null 126 Nov 29, 2022
Docker file for go template

Overview If you are wondering how to make correct Dockerfile for web applications that wrote in GO Lang you are in the right place In this repository

Farhad 4 Dec 2, 2021
A small utility to generate a kubectl configuration file for all clusters you have access to in GKE.

gke-config-helper A small utility to generate a kubectl configuration file for all clusters you have access to in GKE. Usage $ gke-config-helper The b

Calle Pettersson 4 Feb 9, 2022
A simple multi-layered config loader for Go. Made for smaller projects. No external dependencies.

gocfg ⚠️ Work in progress! A simple multi-layered config loader for Go. Made for smaller projects. No external dependencies. Example From main.go: //

Kai Dinghofer 2 Dec 26, 2021
Example goreleaser + github actions config with keyless signing and SBOM generation

supply-chain-example GoReleaser + Go Mod proxying + Cosign keyless signing + Syft SBOM generation example. How it works GoReleaser manages the entire

GoReleaser 47 Nov 15, 2022
Etcd config dispenser

etcd-config-dispenser Some things are best explained with an example: I use lets

Jille Timmermans 1 Jan 20, 2022
µTask is an automation engine that models and executes business processes declared in yaml. ✏️📋

µTask, the Lightweight Automation Engine µTask is an automation engine built for the cloud. It is: simple to operate: only a postgres DB is required s

OVHcloud 793 Nov 28, 2022