falco is VCL parser and linter optimized for Fastly.

Related tags

Video falco
Overview

Go Reference Build

falco

falco is VCL parser and linter optimized for Fastly.

falco-demo

Disclaimer

This is a VCL parser, but dedicated to Fastly's VCL (version 2.x), so we don't care about the latest Varnish (7.x or later) syntax. The Varnish may have additional syntax, builtin function, predefined variables, but this tool may not parse correctly.

Additionally, Fastly provides its special builtin function, predefined variables. It's not compatible with Varnish. But this tool is optimized for them, we could parse and lint their declarations.

Motivation

Fastly is a really fantastic CDN, but sometimes we have problems with deployment operations. On deploy custom VCL to the Fastly, VCLs are validated when activating a new service version. Typically our deployment flow using custom VCLs is following:

  1. Clone active service and create new version
  2. Delete existing custom VCLs
  3. Upload new VCL files to the Fastly
  4. Activate new device version <= Validate VCLs on the Fastly cloud

Above flows take a time, and then if we have some mistakes on VCL e.g. missing semicolon X(, the deployment will fail. Additionally, unnecessary service versions will be created by our trivial issue.

To solve them, we made a Fastly dedicated VCL parser and linter tool to notice syntax errors and unexpected mistakes before starting the deployment flow.

Installation

Download binary from releases page according to your platform and place it under the $PATH.

Usage

Command help displays following:

falco -h
=======================================
  falco: Fastly VCL parser / linter
=======================================
Usage:
    falco [main vcl file]

Flags:
    -I, --include_path : Add include path
    -t, --transformer  : Specify transformer
    -h, --help         : Show this help
    -V, --version      : Display build version
    -v,                : Verbose warning lint result
    -vv,               : Verbose all lint result

Example:
    falco -I . -vv /path/to/vcl/main.vcl

Note:

Your VCL will have dependent modules loaded via include [module]. falco accept include path from -I, --include_path flag and search and load destination module from include path.

User defined subroutine

On linting, falco could not recognize when the user-defined subroutine is called, so you should apply the subroutine scope by adding annotation or its subroutine name. falco understands call scope by following rules:

Subroutine name

If the subroutine name has a suffix of _[scope], falco lint within that scope.

sub custom_recv { // name has `_recv` suffix, lint with RECV scope
  ...
}

sub custom_fetch { // name has `_fetch` suffix, lint with FETCH scope
  ...
}

Following table describes subroutine name and recognizing scope:

suffix scope example
_recv RECV sub custom_recv {}
_miss MISS sub custom_miss {}
_hash HASH sub custom_hash {}
_pass PASS sub custom_pass {}
_fetch FETCH sub custom_fetch {}
_error ERROR sub custom_error {}
_deliver DELIVER sub custom_deliver {}
_log LOG sub custom_log {}

Annotation

For some reasons, the subroutine name could not be changed. Then, if you apply a hint of scope on annotation, falco also understands scope:

// @recv
sub custom_process { // subroutine has `recv` annotation, lint with RECV scope
  ...
}

// @fetch
sub custom_request { // subroutine has `fetch` annotation, lint with FETCH scope
  ...
}

Following table describes annotation name and recognizing scope:

annotation scope example
@recv RECV // @recv
sub custom {}
@miss MISS // @miss
sub custom {}
@hash HASH // @hash
sub custom {}
@pass PASS // @pass
sub custom {}
@fetch FETCH // @fetch
sub custom {}
@error ERROR // @error
sub custom {}
@deliver DELIVER // @deliver
sub custom {}
@log LOG // @log
sub custom {}

Fastly related features

Currently, we don't support snippets which are managed in Fastly:

  • Edge Dictionary
  • VCL Snippets
  • Log definitions
  • Etc

Above snippets will be injected to your VCL top or extracting FASTLY XXX macro, but this tool aims to run locally, not communicating with Fastly service. However, we're planning to solve them using the Fastly API.

Lint error

falco has built in lint rules. see rules in detail. falco may report lots of errors and warnings because falco lints with strict type checks, disallows implicit type conversions even VCL is fuzzy typed language.

Overriding Severity

To avoid them, you can override severity levels by putting a configuration file named .falcorc on working directory. the configuration file contents format is following:

## /path/to/working/directory/.falcorc
regex/matched-value-override: IGNORE
...

Format is simply a yaml key-value object. The key is rule name, see rules.md and value should be one of IGNORE, INFO, WARNING and ERROR, case insensitive.

In the above case, the rule of regex/matched-value-override reports INFO as default, but overrides IGNORE which does not report it.

Error Levels

falco reports three of severity on linting:

ERROR

VCL may cause errors on Fastly, or may cause unexpected behavior for actual works.

WARNING

VCL could work, but may have potential bug and cause unexpected behavior for actual works.

falco does not output warnings as default. To see them, run with -v option.

INFORMATION

VCL is fine, but we suggest you improve your VCL considering Fastly recommendation.

falco does not output information as default. To see them, run with -vv option.

Transforming

falco is planning to transpile Fastly VCL to the other programming language e.g Go (HTTP service), node.js ([email protected]) to use temporal CDN instead of Fastly.

Contribution

  • Fork this repository
  • Customize / Fix problem
  • Send PR :-)
  • Or feel free to create issues for us. We'll look into it

License

MIT License

Contributors

Issues
  • feat(lint): Adds support for subs with return type

    feat(lint): Adds support for subs with return type

    Extends falco to support subroutines with return values

    Fixes https://github.com/ysugimoto/falco/issues/8

    Signed-off-by: Sotiris Nanopoulos [email protected]

    opened by davinci26 9
  • Fixes #42 - Linting `client.geo.city.utf8`

    Fixes #42 - Linting `client.geo.city.utf8`

    As discussed in #42, setting the split depth to 3 causes variables such as client.geo.city.utf8 to not be recognized.

    Signed-off-by: Sotiris Nanopoulos [email protected]

    opened by davinci26 7
  • Support PCRE Regular Expressions

    Support PCRE Regular Expressions

    Why? Falco (the go regexp pkg to be more accurate!) right now doesn't know how to parse PCRE regular expressions that Varnish uses. An example of that would be this file: https://github.com/varnishcache/varnish-devicedetect/blob/master/devicedetect.vcl

    opened by shadialtarsha 6
  • [Feature] Add support for Fastly Rate Limit Variables

    [Feature] Add support for Fastly Rate Limit Variables

    Hey,

    We are interested in contributing support for https://developer.fastly.com/reference/vcl/declarations/ratecounter and all the other rate-limit variables that fastly offers.

    opened by davinci26 5
  • Release candidate of v0.11.0

    Release candidate of v0.11.0

    We'd like to do release version of v0.11.0, which contains the following PRs:

    • https://github.com/ysugimoto/falco/pull/57
    • https://github.com/ysugimoto/falco/pull/64
    • https://github.com/ysugimoto/falco/pull/67

    @davinci26 @shadialtarsha

    opened by ysugimoto 4
  • [BUG] Unexpected token

    [BUG] Unexpected token "callback" found

    Describe the problem Run into some unexpected token issue when linting the VCL code.

    VCL code that cause the problem

    # uses fastly's map function to filter cookies
    callback filter_cookies {
      ...
    }
    

    Screenshots

    $ falco -I service/vcl/ -vv service/main.vcl
    💥  Unexpected token "callback" found
    in /Users/ruichen/mup/production-fastly-vcl/service/main.vcl at line 1692, position 1
     1687|
     1688|  return(lookup);
     1689|}
     1690|
     1691|# uses fastly's map function to filter cookies sent by the backend for logged in users
     1692|callback filter_cookies
          ^^^^^^^^
    

    Expected behavior Passing the linting

    Desktop (please complete the following information):

    • OS: macOS
    • Shell: bash

    Additional context Add any other context about the problem here.

    opened by chenrui333 4
  • revert regex related linting

    revert regex related linting

    My apologies that Varnish regular expression is not compatible for Golang's regexp package.

    It causes:

    if (req.url ~ "^/([^\?]*)?(\?.*)?$") {
    ...
    }
    

    Then raises [ERROR] regex string is invalid, error parsing regexp: invalid or unsupported Perl syntax: (?. I assume that VCL regular expression is using Perl syntax but Golang does not have it... so I revert these changes 😢

    opened by ysugimoto 4
  • Improve include statement parsing

    Improve include statement parsing

    • Resolve include staments with ".vcl" suffix. This is correct per Fastly linting so we make falco a bit more flexible.
    • Include statements are not required to have a ; at the end of the line by fastly. It seems like it is optional.

    Signed-off-by: Sotiris Nanopoulos [email protected]

    opened by davinci26 3
  • Add support for Fastly Rate Limiting

    Add support for Fastly Rate Limiting

    What is this PR?

    This PR introduces the ability to parse and lint the Fastly Rate Limiting feature. (also did some cleanup)

    How does this PR do that?

    Fastly Rate Limiting introduces 2 new declarations:

    The PR adds these 2 declarations to the parsers and adds support to lint them as well.

    Note: The two declarations don't allow any statements inside them.

    Also, Faslty has 4 builtin functions for the rate-limiting functionality as well:

    What can I improve/add?

    • Better error message when the user adds statements inside the penaltybox and the ratecounter blocks
    • More tests
    opened by shadialtarsha 3
  • Adds support for `min_tls_version`/`max_tls_version` in backend object

    Adds support for `min_tls_version`/`max_tls_version` in backend object

    min_tls_version/max_tls_version is missing but it is supported in https://developer.fastly.com/reference/api/services/backend/

    Signed-off-by: Sotiris Nanopoulos [email protected]

    opened by davinci26 3
  • [BUG] Parser/Linter error: Falco doesnt recognize return types for subs

    [BUG] Parser/Linter error: Falco doesnt recognize return types for subs

    Describe the problem A clear and concise description of what the bug is.

    https://developer.fastly.com/reference/vcl/subroutines/ can have return types however Falco doesnt allow for that and it causes a parse error.

    VCL code that cause the problem If applicable, put your VCL code to reproduce on our environment

    sub returns_true BOOL {
      return (true);
    }
    
    opened by davinci26 3
  • Add support for PCRE

    Add support for PCRE

    Why? Falco (the go regexp pkg to be more accurate!) right now doesn't know how to parse PCRE regular expressions that Varnish uses. An example of that would be this file: https://github.com/varnishcache/varnish-devicedetect/blob/master/devicedetect.vcl

    Edit1: I found a source code that is CGO-free which uses ccgo compiler to translate the PCRE2 source code to Go. It is actually working 😄 It has support already for multiple OS and ARCH.

    Would love to hear your opinion about this

    Edit2: CI is green. I added a build step for pullrequest.yaml and it passed, then I removed it.

    opened by shadialtarsha 0
  • [BUG] cross-compile failing

    [BUG] cross-compile failing

    Describe the problem

    When I make a new release, accidentally CI process failed. I'm guessing our code includes platform dependency since https://github.com/ysugimoto/falco/pull/66 has been merged.

    On CI, use ubuntu-latest, linux-amd64:

    • linux: success
    • darwin-amd64: fail
    • darwin-arm64: fail

    It should be fixed to release successfully.

    VCL code that cause the problem N/A Screenshots Screen Shot 2022-07-21 at 18 11 28

    Expected behavior All platform binaries should be compiled and release successfully. A clear and concise description of what you expected to happen.

    Desktop (please complete the following information):

    • OS: Ubuntu (GitHub Actions)
    • Shell bash

    Additional context Add any other context about the problem here.

    opened by ysugimoto 3
  • Add support for subroutines operating in multiple scopes

    Add support for subroutines operating in multiple scopes

    The current way we set annotations only allows for subroutines to be used in a single vcl state function.

    Fastly is allows that and it also leads to copy-pasting functions with different annotations, which is not a good pattern.

    I only implemented them as annotations in the form @<state_name> instead of supporting them in the function name because I think it will be weird to have sub foo_recv_miss_deliver

    Fixes #84

    Signed-off-by: Sotiris Nanopoulos [email protected]

    enhancement 
    opened by davinci26 3
  • Allow functions to be used in multiple subroutines

    Allow functions to be used in multiple subroutines

    In falco you need to add @recv as a comment on top of the funciton but some subroutines can be used in multiple functions if they only use req object for example.

    opened by davinci26 0
  • [BUG] Parser/Linter error: Falco doesnt support calling sub with `()`

    [BUG] Parser/Linter error: Falco doesnt support calling sub with `()`

    Falco doesn't accept the following syntax:

    sub foo {
      log "1234";
    }
    
    sub vcl_recv {
    
      #Fastly recv
      call foo();
      set req.backend = httpbin_org;
    
      return (pass);
    }
    

    With error:

    💥  Missing semicolon
    in /Users/sotiris.nanopoulos/dev/falco/examples/default01.vcl at line 35, position 8
     30|}
     31|
     32|sub vcl_recv {
     33|
     34|  #Fastly recv
     35|  call foo()
    

    But fiddle is ok with it:

    https://fiddle.fastlydemo.net/fiddle/2a1d3753

    opened by davinci26 1
  • String list arguments

    String list arguments

    This PR follows #73 so should be merged after it.

    Built-in function h2.disable_header_compression accepts variadic arguments but currently we don't support STRING_LIST argument type. This PR supports linting of variadic arguments.

    enhancement 
    opened by ysugimoto 0
Releases(v0.13.0)
Owner
Yoshiaki Sugimoto
Yoshiaki Sugimoto
Package implement reading and writing popular playlist formats: PLS, ASX, M3U, XSPF and others.

go-playlist ⚠️ WARNING The API is not stable yet and can change. Package playlist implement reading and writing popular playlist formats: PLS, ASX, M3

Roman Kulish 0 Oct 14, 2021
ffcommander - An easy frontend to FFmpeg and Imagemagick to automatically process video and manipulate subtitles.

% FFCOMMANDER(1) ffcommander 2.39 % Mikael Hartzell (C) 2018 % 2021 Name ffcommander - An easy frontend to FFmpeg and Imagemagick to automatically pro

Mikael Hartzell 2 May 9, 2022
Parse and demux MPEG Transport Streams (.ts) natively in GO

This is a Golang library to natively parse and demux MPEG Transport Streams (ts) in GO. WARNING: this library is not yet production ready. Use at your

Quentin Renard 426 Jul 10, 2022
Parse and generate m3u8 playlists for Apple HTTP Live Streaming (HLS) in Golang (ported from gem https://github.com/sethdeckard/m3u8)

go-m3u8 Golang package for m3u8 (ported m3u8 gem https://github.com/sethdeckard/m3u8) go-m3u8 provides easy generation and parsing of m3u8 playlists d

Tan Quang Ngo 97 Jul 24, 2022
golang library to read and write various subtitle formats

libgosubs Golang library to read and write subtitles in the following formats Advanced SubStation Alpha v4 SRT TTML v1.0 - This is based on the spec p

Wargarblgarbl 19 Jul 8, 2022
Go bindings for libVLC and high-level media player interface

Go bindings for libVLC 2.X/3.X/4.X and high-level media player interface. The package can be useful for adding multimedia capabilities to applications

Adrian-George Bostan 310 Jul 28, 2022
Plays videos using Prometheus and Grafana, e.g. Bad Apple.

prometheus_video_renderer Plays videos using Prometheus and Grafana, e.g. Bad Apple. Modes Currently 3 different modes are supported. Bitmap The bitma

Jacob Colvin 88 Jun 29, 2022
RTSP 1.0 client and server library for the Go programming language

RTSP 1.0 client and server library for the Go programming language

Alessandro Ros 278 Jul 27, 2022
Desktop application to download videos and playlists from youtube by simply copying its url.

tubemp3 Desktop application to download videos and playlists from youtube by simply copying its url. You just need to run tubemp3 and copy (CTRL + C)

Fernando Guevara Sanchez 12 Feb 16, 2022
A simple library to extract video and audio frames from media containers (based on libav).

Reisen A simple library to extract video and audio frames from media containers (based on libav, i.e. ffmpeg). Dependencies The library requires libav

NightGhost 53 Jul 20, 2022
Project to get Youtube video descriptions and search those videos as required

FamPayProject Project to get Youtube video descriptions and search those videos as required Prerequisities Postgres DB for persisting data Youtube Dat

null 0 Nov 5, 2021
golang library for mux and demux file containers such as mpeg-ts,mpeg-ps,flv

gomedia mpeg-ts,mpeg-ps,flv muxer/demuxer mpeg-ts muxer mpeg-ts demuxer mpeg-ps muxer mpeg-ps demuxer flv muxer flv demuxer mpeg-ps will be done in th

yaping 50 Aug 5, 2022
Synthetic media is a realistic transformation of audio and video using artificial intelligence.

Synthetic media is a realistic transformation of audio and video using artificial intelligence.

null 1 Nov 20, 2021
golang function that download a video from youtube, and convert it to a mp3 file using ffmpeg

echedwnmp3 echedwnmp3 is a function that download a video from youtube, and convert it to a mp3 file using ffmpeg example package main import(echedwn

pai 4 Dec 7, 2021
lmmp3 is a little golang library that download a video from youtube, and convert it to a mp3 file using ffmpeg

lmmp3 lmmp3 is a function that download a video from youtube, and convert it to a mp3 file using ffmpeg You need to have installed ffmpeg in your syst

pai 9 Aug 1, 2022
👾 Annie is a fast, simple and clean video downloader built with Go.

?? Annie is a fast, simple and clean video downloader built with Go. Installation Prerequisites Install via go install Homebrew (macOS only) Arch Linu

Rohan 1 Jun 1, 2022
[WIP] a very simple, tiny and intuitive ffmpeg wrapper with a cli interface for inspecting & transforming media files supported by the original ffmpeg software

About a very simple, tiny and intuitive ffmpeg wrapper with a cli interface for inspecting & transforming media files supported by the original ffmpeg

Mohammed Al Ashaal 9 Jun 28, 2022
CLI program for SEO 301 & 302 url rewrites in fastly with more magento features soon to come

Magento-Fastly Table of Contents Magento-Fastly Development & Testing Install fastly cli Features Installation Usage Development & Testing To test thi

null 1 Oct 29, 2021
xlsx2mysql: An tool of helping your fastly generate SQL from Excel.

xlsx2mysql An tool of helping your fastly generate SQL from Excel 中文文档 Origin In order to convert Excel to MySQL and I made a tool to implement.But Wh

WeLong 1 Nov 13, 2021
Demonstrating how you can take an action to your intrusions detected by Falco using OpenFaaS functions

Kubernetes Response Engine powered by OpenFaaS Although Falco can be used to detect any intrusion attempts and sends alerts to channels according to t

Batuhan Apaydın 24 Nov 15, 2021
This is a POC for a Falco Plugin allowing to gather events from a locale docker daemon.

Docker Events Plugin This is a POC for a Falco Plugin allowing to gather events from a locale docker daemon. ⚠️ This is a POC, don't use in Production

Thomas Labarussias 3 Apr 15, 2022
Twitter-plugin - Falco Plugin for Twitter Stream

Twitter Plugin This repository contains the twittter plugin for Falco, which fol

Thomas Labarussias 4 Mar 17, 2022
🧩 Redify is the optimized key-value proxy for quick access and cache of any other database throught Redis and/or HTTP protocol.

Redify (Any database as redis) License Apache 2.0 Redify is the optimized key-value proxy for quick access and cache of any other database throught Re

Dmitry Ponomarev 2 Mar 21, 2022
Optimized bit-level Reader and Writer for Go.

bitio Package bitio provides an optimized bit-level Reader and Writer for Go. You can use Reader.ReadBits() to read arbitrary number of bits from an i

András Belicza 192 Jul 26, 2022
'go test' runner with output optimized for humans, JUnit XML for CI integration, and a summary of the test results.

gotestsum gotestsum runs tests using go test --json, prints formatted test output, and a summary of the test run. It is designed to work well for both

null 1k Jul 30, 2022
Go library that provides fuzzy string matching optimized for filenames and code symbols in the style of Sublime Text, VSCode, IntelliJ IDEA et al.

Go library that provides fuzzy string matching optimized for filenames and code symbols in the style of Sublime Text, VSCode, IntelliJ IDEA et al. This library is external dependency-free. It only depends on the Go standard library.

Sahil Muthoo 1.1k Jul 16, 2022
A Rancher and Kubernetes optimized immutable Linux distribution based on openSUSE

RancherOS v2 WORK IN PROGRESS RancherOS v2 is an immutable Linux distribution built to run Rancher and it's corresponding Kubernetes distributions RKE

Rancher 81 Aug 3, 2022
Microshift is a research project that is exploring how OpenShift1 Kubernetes can be optimized for small form factor and edge computing.

Microshift is a research project that is exploring how OpenShift1 Kubernetes can be optimized for small form factor and edge computing.

Oleg Silkin 0 Nov 1, 2021
Tape backup software optimized for large WORM data and long-term recoverability

Mixtape Backup software for tape users with lots of WORM data. Draft design License This codebase is not open-source software (or free, or "libre") at

Dave Anderson 15 Apr 23, 2022