An image resizing server written in Go

Overview

picfit

Build Status

https://d262ilb51hltx0.cloudfront.net/max/800/1*oR04S6Ie7s1JctwjsDsN0w.png

picfit is a reusable Go server to manipulate images (resize, thumbnail, etc.).

It will act as a proxy on your storage engine and will be served ideally behind an HTTP cache system like varnish.

It supports multiple storage backends and multiple key/value stores.

Installation

Build it

  1. Make sure you have a Go language compiler and git installed.
  2. Make sure you have the following go system dependencies in your $PATH: bzr, svn, hg, git
  3. Ensure your GOPATH is properly set.
  4. Download it:
git clone https://github.com/thoas/picfit.git
  1. Run make build

You have now a binary version of picfit in the bin directory which fits perfectly with your architecture.

picfit has also a Docker integration to built a unix binary without having to install it

make docker-build

Debian and Ubuntu

We will provide Debian package when we will be completely stable ;)

Configuration

Configuration should be stored in a readable file and in JSON format.

The location of the configuration is specified by --config or the PICFIT_CONFIG_PATH environment variable.

config.json

{
  "kvstore": {
    "type": "[KVSTORE]"
  },
  "storage": {
    "src": {
      "type": "[STORAGE]"
    }
  }
}

[KVSTORE] can be:

  • redis - generated keys stored in Redis, see below how you can customize connection parameters
  • cache - generated keys stored in an in-memory cache
  • redis-cluster - generated keys stored in Redis cluster

[STORAGE] can be:

  • fs - generated images stored in your File system
  • http+fs - generated images stored in your File system and loaded using HTTP protocol
  • s3 - generated images stored in Amazon S3
  • dos3 - generated images stored in DigitalOcean S3
  • gcs - generated images stored in Google Cloud Storage
  • http+s3 - generated images stored in Amazon S3 and loaded using HTTP protocol
  • http+dos3 - generated images stored in DigitalOcean S3 and loaded using HTTP protocol

Basic

  • no key/value store
  • no image storage
  • images are given in absolute url

config.json

{
  "port": 3001
}

Images are generated on the fly at each request.

Store images on file system and keys in an in-memory cache

  • key/value in-memory store
  • file system storage

An image is generated from your source storage (src) and uploaded asynchronously to this storage.

A unique key is generated and stored in a in-memory key/value store to process a request only once.

config.json

{
  "port": 3001,
  "storage": {
    "src": {
      "type": "fs",
      "location": "/path/to/directory/"
    }
  },
  "kvstore": {
    "type": "cache"
  },
}

Store images on DigitalOcean S3

It's mostly the same as Amazon S3 storage, the difference are accepted regions So, regions can be:

  • nyc1
  • nyc2
  • nyc3
  • ams2
  • ams3
  • sfo1
  • sfo2
  • sgp1
  • lon1
  • fra1
  • tor1
  • blr1

Store images on Amazon S3, keys in Redis and shard filename

  • key/value store provided by Redis
  • Amazon S3 storage
  • shard filename

config.json

{
  "kvstore": {
    "type": "redis",
    "redis": {
      "host": "127.0.0.1",
      "port": 6379,
      "password": "",
      "db": 0
    }
  },
  "port": 3001,
  "storage": {
    "src": {
      "type": "s3",
      "access_key_id": "[ACCESS_KEY_ID]",
      "secret_access_key": "[SECRET_ACCESS_KEY]",
      "bucket_name": "[BUCKET_NAME]",
      "acl": "[ACL]",
      "region": "[REGION_NAME]",
      "location": "path/to/directory"
    }
  },
  "shard": {
    "width": 1,
    "depth": 2,
    "restonly": true
  }
}

Keys will be stored on Redis, (you better need to setup persistence).

Image files will be loaded and stored on Amazon S3 at the location path/to/directory in the bucket [BUCKET_NAME].

[ACL] can be:

  • private
  • public-read
  • public-read-write
  • authenticated-read
  • bucket-owner-read
  • bucket-owner-full-control

[REGION_NAME] can be:

  • us-gov-west-1
  • us-east-1
  • us-west-1
  • us-west-2
  • eu-west-1
  • eu-central-1
  • ap-southeast-1
  • ap-southeast-2
  • ap-northeast-1
  • sa-east-1
  • cn-north-1

Filename will be sharded:

  • depth - 2 directories
  • width - 1 letter for each directory
  • restonly - true, filename won't contain characters in sharding path

Example:

06102586671300cd02ae90f1faa16897.png will become 0/6/102586671300cd02ae90f1faa16897.jpg

with restonly=false it would become 0/6/06102586671300cd02ae90f1faa16897.jpg

It would be useful if you are using the file system storage backend.

Load images from file system and store them in Amazon S3, keys on Redis cluster

  • key/value store provided by Redis cluster
  • File system to load images
  • Amazon S3 storage to process images

config.json

{
  "kvstore": {
    "type": "redis-cluster",
    "redis": {
      "addrs": [
        "127.0.0.1:6379"
      ],
      "password": "",
    }
  },
  "port": 3001,
  "storage": {
    "src": {
      "type": "fs",
      "location": "path/to/directory"
    },
    "dst": {
      "type": "s3",
      "access_key_id": "[ACCESS_KEY_ID]",
      "secret_access_key": "[SECRET_ACCESS_KEY]",
      "bucket_name": "[BUCKET_NAME]",
      "acl": "[ACL]",
      "region": "[REGION_NAME]",
      "location": "path/to/directory"
    }
  }
}

You will be able to load and store your images from different storages backend.

In this example, images will be loaded from the file system storage and generated to the Amazon S3 storage.

Load images from storage backend base url, store them in Amazon S3, keys prefixed on Redis

  • key/value store provided by Redis
  • File system to load images using HTTP method
  • Amazon S3 storage to process images

config.json

{
  "kvstore": {
    "type": "redis",
    "redis": {
      "host": "127.0.0.1",
      "port": 6379,
      "password": "",
      "db": 0
    },
    "prefix": "dummy:"
  },
  "port": 3001,
  "storage": {
    "src": {
      "type": "http+fs",
      "base_url": "http://media.example.com",
      "location": "path/to/directory"
    },
    "dst": {
      "type": "s3",
      "access_key_id": "[ACCESS_KEY_ID]",
      "secret_access_key": "[SECRET_ACCESS_KEY]",
      "bucket_name": "[BUCKET_NAME]",
      "acl": "[ACL]",
      "region": "[REGION_NAME]",
      "location": "path/to/directory"
    }
  }
}

In this example, images will be loaded from the file system storage using HTTP with base_url option and generated to the Amazon S3 storage.

Keys will be stored on Redis using the prefix dummy:.

Running

To run the application, issue the following command:

$ picfit -c config.json

By default, this will run the application on port 3001 and can be accessed by visiting:

http://localhost:3001

The port number can be configured with port option in your config file.

To see a list of all available options, run:

$ picfit --help

Usage

General parameters

Parameters to call the picfit service are:

<img src="http://localhost:3001/{method}?url={url}&path={path}&w={width}&h={height}&upscale={upscale}&sig={sig}&op={operation}&fmt={format}&q={quality}&deg={degree}&pos={position}"
  • path - The filepath to load the image using your source storage
  • operation - The operation to perform, see Operations
  • sig - The signature key which is the representation of your query string and your secret key, see Security
  • method - The method to perform, see Methods
  • url - The url of the image to generate (not required if path provided)
  • width - The desired width of the image, if 0 is provided the service will calculate the ratio with height
  • height - The desired height of the image, if 0 is provided the service will calculate the ratio with width
  • upscale - If your image is smaller than your desired dimensions, the service will upscale it by default to fit your dimensions, you can disable this behavior by providing 0
  • format - The output format to save the image, by default the format will be the source format (a GIF image source will be saved as GIF), see Formats
  • quality - The quality to save the image, by default the quality will be the highest possible, it will be only applied on JPEG format
  • degree - The degree (90, 180, 270) to rotate the image
  • position - The position to flip the image

To use this service, include the service url as replacement for your images, for example:

<img src="https://www.google.fr/images/srpr/logo11w.png" />

will become:

<img src="http://localhost:3001/display?url=https%3A%2F%2Fwww.google.fr%2Fimages%2Fsrpr%2Flogo11w.png&w=100&h=100&op=resize&upscale=0"

This will retrieve the image used in the url parameter and resize it to 100x100.

Using source storage

If an image is stored in your source storage at the location path/to/file.png, then you can call the service to load this file:

<img src="http://localhost:3001/display?w=100&h=100&path=path/to/file.png&op=resize"

or

<img src="http://localhost:3001/display/resize/100x100/path/to/file.png"

Formats

picfit currently supports the following image formats:

  • image/jpeg with the keyword jpg or jpeg
  • image/png with the keyword png
  • image/gif with the keyword gif
  • image/bmp with the keyword bmp

Operations

Resize

This operation will able you to resize the image to the specified width and height.

If width or height value is 0, the image aspect ratio is preserved.

  • w - The desired image's width
  • h - The desired image's height

You have to pass the resize value to the op parameter to use this operation.

Thumbnail

Thumbnail scales the image up or down using the specified resample filter, crops it to the specified width and height and returns the transformed image.

  • w - The desired width of the image
  • h - The desired height of the image

You have to pass the thumbnail value to the op parameter to use this operation.

Flip

Flip flips the image vertically (from top to bottom) or horizontally (from left to right) and returns the transformed image.

  • pos - The desired position to flip the image, h will flip the image horizontally, v will flip the image vertically

You have to pass the flip value to the op parameter to use this operation.

Rotate

Rotate rotates the image to the desired degree and returns the transformed image.

  • deg - The desired degree to rotate the image

You have to pass the rotate value to the op parameter to use this operation.

Flat

Flat draws a given image on the image resulted by the previous operation. Flat can be used only with the [multiple operation system].

  • path - the foreground image path
  • color - the foreground color in Hex (without #), default is transparent
  • pos - the destination rectangle

In order to understand the Flat operation, please read the following docs.

Methods

Display

Display the image, useful when you are using an img tag.

The generated image will be stored asynchronously on your destination storage backend.

A couple of headers (Content-Type, If-Modified-Since) will be set to allow you to use an http cache system.

Redirect

Redirect to an image.

Your file will be generated synchronously then the redirection will be performed.

The first query will be slower but next ones will be faster because the name of the generated file will be stored in your key/value store.

Get

Retrieve information about an image.

Your file will be generated synchronously then you will get the following information:

  • filename - Filename of your generated file
  • path - Path of your generated file
  • url - Absolute url of your generated file (only if base_url is available on your destination storage)

The first query will be slower but next ones will be faster because the name of the generated file will be stored in your key/value store.

Expect the following result:

{
    "filename":"a661f8d197a42d21d0190d33e629e4.png",
    "path":"cache/6/7/a661f8d197a42d21d0190d33e629e4.png",
    "url":"https://ds9xhxfkunhky.cloudfront.net/cache/6/7/a661f8d197a42d21d0190d33e629e4.png"
}

Upload

Upload is disabled by default for security reason. Before enabling it, you must understand you have to secure yourself this endpoint like only allowing the /upload route in your nginx or apache webserver for the local network.

Exposing the /upload endpoint without a security mechanism is not SAFE.

You can enable it by adding the option and a source storage to your configuration file.

config.json

{
  "storage": {
    "src": {
      "type": "[STORAGE]"
    }
  },
  "options": {
    "enable_upload": true
  }
}

To work properly, the input field must be named "data"

Test it with the excellent httpie:

http -f POST localhost:3000/upload [email protected]

You will retrieve the uploaded image information in JSON format.

Multiple operations

Multiple operations can be done on the same image following a given order.

First operation must be described as above then other operation are described in parameters op. The order of op parameters is the order used.

Each options of the operation must be described with subparameters separated by : with the operation name as argument to op.

Example of a resize followed by a rotation:

<img src="http://localhost:3001/display?w=100&h=100&path=path/to/file.png&op=resize&op=op:rotate+deg:180"

Security

Request signing

In order to secure requests and avoid unknown third parties to use the service, the application can require a request to provide a signature. To enable this feature, set the secret_key option in your config file.

The signature is an hexadecimal digest generated from the client key and the query string using the HMAC-SHA1 message authentication code (MAC) algorithm.

The below python code provides an implementation example:

import hashlib
import hmac
import six
import urllib

def sign(key, *args, **kwargs):
    m = hmac.new(key, None, hashlib.sha1)

    for arg in args:
        if isinstance(arg, dict):
            m.update(urllib.urlencode(arg))
        elif isinstance(arg, six.string_types):
            m.update(arg)

    return m.hexdigest()

The implementation has to sort and encode query string to generate a proper signature.

The signature is passed to the application by appending the sig parameter to the query string; e.g. w=100&h=100&sig=c9516346abf62876b6345817dba2f9a0c797ef26.

Note, the application does not include the leading question mark when verifying the supplied signature. To verify your signature implementation, see the signature command described in the Tools section.

Limiting allowed sizes

Depending on your use case it may be more appropriate to simply restrict the image sizes picfit is allowed to generate. See the Allowed sizes section for more information on this configuration.

Tools

To verify that your client application is generating correct signatures, use the command:

$ picfit signature --key=abcdef "w=100&h=100&op=resize"
Query String: w=100&h=100&op=resize
Signature: 6f7a667559990dee9c30fb459b88c23776fad25e
Signed Query String: w=100&h=100&op=resize&sig=6f7a667559990dee9c30fb459b88c23776fad25e

Error reporting

picfit logs events by default in stderr and stdout. You can implement sentry to log errors using raven.

To enable this feature, set sentry option in your config file.

config.json

{
  "sentry": {
    "dsn": "[YOUR_SENTRY_DSN]",
    "tags": {
      "foo": "bar"
    }
  }
}

Debug

Debug is disabled by default.

To enable this feature set debug option to true in your config file:

config.json

{
  "debug": true
}

CORS

picfit supports CORS headers customization in your config file.

To enable this feature, set allowed_origins, allowed_headers and allowed_methods, for example:

config.json

{
  "allowed_headers": ["Content-Type", "Authorization", "Accept", "Accept-Encoding", "Accept-Language"],
  "allowed_origins": ["*.ulule.com"],
  "allowed_methods": ["GET", "HEAD"]
}

Image engine

Quality

The quality rendering of the image engine can be controlled globally without adding it at each request:

config.json

{
  "engine": {
    "quality": 70
  }
}

With this option, each image will be saved in 70 quality.

By default the quality is the highest possible: 95

Format

The format can be forced globally without adding it at each request:

config.json

{
  "engine": {
    "format": "png"
  }
}

With this option, each image will be forced to be saved in .png.

By default the format will be chosen in this order:

  • The fmt parameter if exists in query string
  • The original image format
  • The default format provided in the application

Options

Deletion

Deletion is disabled by default for security reason, you can enable it in your config:

config.json

{
  "options": {
    "enable_delete": true
  }
}

You will be able to delete root image and its children, for example if you upload an image with the file path /foo/bar.png, you can delete the main image on stockage by sending the following HTTP request:

DELETE https://localhost:3001/foo/bar.png

or delete a child:

DELETE https://localhost:3001/display/thumbnail/100x100/foo/bar.png

If you want to delete the main image and cascade its children, you can enable it in your config:

config.json

{
  "options": {
    "enable_delete": true,
    "enable_cascade_delete": true
  }
}

when a new image will be processed, it will be linked to the main image and stored in the kvstore.

Upload

Upload is disabled by default for security reason, you can enable it in your config:

config.json

{
  "options": {
    "enable_upload": true
  }
}

Stats

Stats are disabled by default, you can enable them in your config.

config.json

{
  "options": {
    "enable_stats": true
  }
}

It will store various information about your web application (response time, status code count, etc.).

To access these information, you can visit: http://localhost:3001/sys/stats

Health

Health is disabled by default, you can enable it in your config.

config.json

{
  "options": {
    "enable_stats": true
  }
}

It will show various internal information about the Go runtime (memory, allocations, etc.).

To access these information, you can visit: http://localhost:3001/sys/health

Profiler

Profiler is disabled by default, you can enable it in your config.

config.json

{
  "options": {
    "enable_pprof": true
  }
}

It will start pprof then use the pprof tool to look at the heap profile:

go tool pprof http://localhost:3001/debug/pprof/heap

Or to look at a 30-second CPU profile:

go tool pprof http://localhost:3001/debug/pprof/profile

Or to look at the goroutine blocking profile, after calling runtime.SetBlockProfileRate in your program:

go tool pprof http://localhost:3001/debug/pprof/block

Or to collect a 5-second execution trace:

wget http://localhost:3001/debug/pprof/trace?seconds=5

Logging

By default the logger level is debug, you can change it in your config:

config.json

{
  "logger": {
    "level": "info"
  }
}

Levels available are:

  • debug
  • info
  • error
  • warning
  • fatal

Allowed sizes

To restrict the sizes picfit is allowed to generate you may specify the allowed_sizes option as an array of sizes. Note that if you omit a width or height from a size it will allow requests that exclude height or width to preserve aspect ratio.

config.json

{
  "options": {
    "allowed_sizes": [
      {"width": 1920, "height": 1080},
      {"width": 720, "height": 480},
      {"width": 480}
    ]
  }
}

IP Address restriction

You can restrict access to upload, stats, health, delete and pprof endpoints by enabling restriction in your config:

config.json

{
  "options": {
    "allowed_ip_addresses": [
      "127.0.0.1"
    ]
  }
}

Deployment

It's recommended that the application run behind a CDN for larger applications or behind varnish for smaller ones.

Provisioning is handled by Ansible, you will find files in the repository.

You must have Ansible installed on your laptop, basically if you have python already installed you can do

$ pip install ansible

Roadmap

see issues

Don't hesitate to send patch or improvements.

Clients

Client libraries will help you generate picfit urls with your secret key.

In production

  • Ulule: an european crowdfunding platform

Inspirations

Thanks to these beautiful projects.

Comments
  • Too many open files

    Too many open files

    Hello,

    After a few day of use, I encounter "too many open files" issues.

    Here is a log of these errors :

    time="2015-12-01T10:04:16+01:00" level="info" msg="PANIC: open /home/www/picfit/storage/c53a33374205508dda23b27f6f0a6b5e.jpg: too many open files\ngoroutine 28338 [running]:\ngithub.com/thoas/picfit/middleware.func·001()\n\t/home/www/gowork/src/github.com/thoas/picfit/middleware/middleware.go:36 +0x12b\nruntime.panic(0x864120, 0xc208ffb170)\n\t/usr/lib/go/src/pkg/runtime/panic.c:248 +0x18d\ngithub.com/thoas/picfit/application.func·001(0x7f047d09c338, 0xc209134380, 0xc20d20d860, 0xc20803e120)\n\t/home/www/gowork/src/github.com/thoas/picfit/application/handlers.go:22 +0xaf\ngithub.com/thoas/picfit/application.func·017(0x7f047d09c338, 0xc209134380, 0xc20a4c1ba0)\n\t/home/www/gowork/src/github.com/thoas/picfit/application/application.go:122 +0x2d1\nnet/http.HandlerFunc.ServeHTTP(0xc2080bf6c0, 0x7f047d09c338, 0xc209134380, 0xc20a4c1ba0)\n\t/usr/lib/go/src/pkg/net/http/server.go:1235 +0x40\ngithub.com/gorilla/mux.(_Router).ServeHTTP(0xc208019130, 0x7f047d09c338, 0xc209134380, 0xc20a4c1ba0)\n\t/home/www/gowork/src/github.com/gorilla/mux/mux.go:98 +0x292\ngithub.com/codegangsta/negroni.func·001(0x7f047d09c338, 0xc209134380, 0xc20a4c1ba0, 0xc209197ac0)\n\t/home/www/gowork/src/github.com/codegangsta/negroni/negroni.go:41 +0x56\ngithub.com/codegangsta/negroni.HandlerFunc.ServeHTTP(0xc2080f3ba0, 0x7f047d09c338, 0xc209134380, 0xc20a4c1ba0, 0xc209197ac0)\n\t/home/www/gowork/src/github.com/codegangsta/negroni/negroni.go:24 +0x4a\ngithub.com/codegangsta/negroni.middleware.ServeHTTP(0x7f047d09aed0, 0xc2080f3ba0, 0xc2080e5460, 0x7f047d09c338, 0xc209134380, 0xc20a4c1ba0)\n\t/home/www/gowork/src/github.com/codegangsta/negroni/negroni.go:33 +0x8c\ngithub.com/codegangsta/negroni.middleware.ServeHTTP·fm(0x7f047d09c338, 0xc209134380, 0xc20a4c1ba0)\n\t/home/www/gowork/src/github.com/codegangsta/negroni/negroni.go:33 +0x56\ngithub.com/meatballhat/negroni-logrus.(_Middleware).ServeHTTP(0xc2080e5320, 0x7f047d09c338, 0xc209134380, 0xc20a4c1ba0, 0xc2091978e0)\n\t/home/www/gowork/src/github.com/meatballhat/negroni-logrus/middleware.go:42 +0x349\ngithub.com/codegangsta/negroni.middleware.ServeHTTP(0x7f047d09af20, 0xc2080e5320, 0xc2080e5440, 0x7f047d09c338, 0xc209134380, 0xc20a4c1ba0)\n\t/home/www/gowork/src/github.com/codegangsta/negroni/negroni.go:33 +0x8c\ngithub.com/codegangsta/negroni.middleware.ServeHTTP·fm(0x7f047d09c338, 0xc209134380, 0xc20a4c1ba0)\n\t/home/www/gowork/src/github.com/codegangsta/negroni/negroni.go:33 +0x56\ngithub.com/rs/cors.(_Cors).ServeHTTP(0xc20803f950, 0x7f047d09c338, 0xc209134380, 0xc20a4c1ba0, 0xc2091978c0)\n\t/home/www/gowork/src/github.com/rs/cors/cors.go:188 +0x155\ngithub.com/codegangsta/negroni.middleware.ServeHTTP(0x7f047d09aef8, 0xc20803f950, 0xc2080e5420, 0x7f047d09c338, 0xc209134380, 0xc20a4c1ba0)\n\t/home/www/gowork/src/github.com/codegangsta/negroni/negroni.go:33 +0x8c\ngithub.com/codegangsta/negroni.middleware.ServeHTTP·fm(0x7f047d09c338, 0xc209134380, 0xc20a4c1ba0)\n\t/home/www/gowork/src/github.com/codegangsta/negroni/negroni.go:33 +0x56\ngithub.com/thoas/picfit/middleware.(_Logger).ServeHTTP(0xc20802a398, 0x7f047d09c338, 0xc209134380, 0xc20a4c1ba0, 0xc209197620)\n\t/home/www/gowork/src/github.com/thoas/picfit/middleware/middleware.go:63 +0x18c\ngithub.com/codegangsta/negroni.middleware.ServeHTTP(0x7f047d09aea8, 0xc20802a398, 0xc2080e5400, 0x7f047d09c338, 0xc209134380, 0xc20a4c1ba0)\n\t/home/www/gowork/src/github.com/codegangsta/negroni/negroni.go:33 +0x8c\ngithub.com/codegangsta/negroni.middleware.ServeHTTP·fm(0x7f047d09c338, 0xc209134380, 0xc20a4c1ba0)\n\t/home/www/gowork/src/github.com/codegangsta/negroni/negroni.go:33 +0x56\ngithub.com/thoas/picfit/middleware.(_Recovery).ServeHTTP(0xc2080e51e0, 0x7f047d09c338, 0xc209134380, 0xc20a4c1ba0, 0xc209197600)\n\t/home/www/gowork/src/github.com/thoas/picfit/middleware/middleware.go:56 +0x9b\ngithub.com/codegangsta/negroni.middleware.ServeHTTP(0x7f047d09ae80, 0xc2080e51e0, 0xc2080e53e0, 0x7f047d09c338, 0xc209134380, 0xc20a4c1ba0)\n\t/home/www/gowork/src/github.com/codegangsta/negroni/negroni.go:33 +0x8c\ngithub.com/codegangsta/negroni.(_Negroni).ServeHTTP(0xc2080adf80, 0x7f047d09c2a0, 0xc2084a5b80, 0xc20a4c1ba0)\n\t/home/www/gowork/src/github.com/codegangsta/negroni/negroni.go:73 +0x10a\nnet/http.serverHandler.ServeHTTP(0xc2080fc420, 0x7f047d09c2a0, 0xc2084a5b80, 0xc20a4c1ba0)\n\t/usr/lib/go/src/pkg/net/http/server.go:1673 +0x19f\nnet/http.(_conn).serve(0xc20d42cd80)\n\t/usr/lib/go/src/pkg/net/http/server.go:1174 +0xa7e\ncreated by net/http.(_Server).Serve\n\t/usr/lib/go/src/pkg/net/http/server.go:1721 +0x313\n" 2015/12/01 10:05:25 http: Accept error: accept tcp [::]:3001: too many open files; retrying in 5ms

    bug need investigation 
    opened by alexd6631 13
  • Downstream changes in mholt/binding has broken build

    Downstream changes in mholt/binding has broken build

    First time gets or installs updating with go get -u github.com/thoas/picfit reveals a downstream change to the mholt/binding package (https://github.com/mholt/binding/commit/58246f9c398c372b9ea02fa16cc5ee6d054285f7) that breaks builds:

    # github.com/thoas/picfit/application
    ../../github.com/thoas/picfit/application/handlers.go:43: cannot use multipartForm (type *MultipartForm) as type binding.FieldMapper in argument to binding.Bind:
        *MultipartForm does not implement binding.FieldMapper (wrong type for FieldMap method)
            have FieldMap() binding.FieldMap
            want FieldMap(*"net/http".Request) binding.FieldMap
    
    opened by steevee 12
  • Add image deletion.

    Add image deletion.

    So, how it works.

    First, there is a new config parameter under "options". Which is named "enable_delete". Same way as it is done with uploads. One should keep in mind and probably I should add that to readme. That only the database built with this parameter can delete images successfully. I did it intentionally, so that you don't pay for deletion feature if you don't need it. In other words, children set will not be created if this option is not set to true.

    Deletion is a GET request. I know you wanted a DELETE request. I guess we can change that. But since I don't know how to handle different types of requests in your muxer I've left it this way. Also The image URLs for displaying are: /display/. And it doesn't make sense to me sending DELETE requests on /display/ URLs. Anyways. The handler is bound to the following path right now: /delete/{path:[\\w\\-/.]+} (delete plus same path argument as in display and others).

    Deletion feature maintains a set of hash keys under special key: <src-filepath>:children as you've suggested. Each element in the set points to the same KVStore.

    Deletion feature does the following:

    1. Delete source file in the source storage.
    2. Get all children set elements for the file we're deleting.
    3. Remove that children set from KVStore.
    4. For each children in the set:
    5. Get that key from KVStore.
    6. Remove the file in the destination storage.
    7. Remove the key from KVStore.

    It's hard to tell if concurrent deletion requests and display requests are allowed. But it seems so. The main sync point is the source file. When the source file is gone, new display requests cannot create new images and hence won't write anything to KVStore or destination storage.

    I'm pretty sure that simulatenous deletion and upload request will break things.

    There.. any comments?

    P.S. Example log:

    2015/09/03 14:35:52 Serving [::]:3001 with pid 17012
    INFO[0013] Started GET /display/resize/100x140/test1.jpg
    INFO[0013] started handling request                      method=GET remote=[::1]:53370 request=/display/resize/100x140/test1.jpg
    INFO[0013] Key fd875c2d5e6cbca8a79467d76d1089af not found in kvstore
    INFO[0013] completed handling request                    measure#web.latency=55361768 method=GET remote=[::1]:53370 request=/display/resize/100x140/test1.jpg status=200 text_status=OK took=55.361768ms
    INFO[0013] Completed 200 OK in 55.527683ms
    INFO[0013] Save thumbnail fd875c2d5e6cbca8a79467d76d1089af.jpg to storage
    INFO[0013] Save key fd875c2d5e6cbca8a79467d76d1089af => fd875c2d5e6cbca8a79467d76d1089af.jpg to kvstore
    INFO[0013] Put key into set test1.jpg:children => fd875c2d5e6cbca8a79467d76d1089af in kvstore
    INFO[0016] Started GET /display/resize/100x130/test1.jpg
    INFO[0016] started handling request                      method=GET remote=[::1]:53370 request=/display/resize/100x130/test1.jpg
    INFO[0016] Key f065a9618010d1c632cade29036c4d0c not found in kvstore
    INFO[0016] completed handling request                    measure#web.latency=43879245 method=GET remote=[::1]:53370 request=/display/resize/100x130/test1.jpg status=200 text_status=OK took=43.879245ms
    INFO[0016] Completed 200 OK in 43.987467ms
    INFO[0016] Save thumbnail f065a9618010d1c632cade29036c4d0c.jpg to storage
    INFO[0016] Save key f065a9618010d1c632cade29036c4d0c => f065a9618010d1c632cade29036c4d0c.jpg to kvstore
    INFO[0016] Put key into set test1.jpg:children => f065a9618010d1c632cade29036c4d0c in kvstore
    INFO[0019] Started GET /display/resize/100x120/test1.jpg
    INFO[0019] started handling request                      method=GET remote=[::1]:53370 request=/display/resize/100x120/test1.jpg
    INFO[0019] Key ee95aee309647ddd6711d26faf61fcc6 not found in kvstore
    INFO[0019] completed handling request                    measure#web.latency=44762160 method=GET remote=[::1]:53370 request=/display/resize/100x120/test1.jpg status=200 text_status=OK took=44.76216ms
    INFO[0019] Completed 200 OK in 44.871535ms
    INFO[0019] Save thumbnail ee95aee309647ddd6711d26faf61fcc6.jpg to storage
    INFO[0019] Save key ee95aee309647ddd6711d26faf61fcc6 => ee95aee309647ddd6711d26faf61fcc6.jpg to kvstore
    INFO[0019] Put key into set test1.jpg:children => ee95aee309647ddd6711d26faf61fcc6 in kvstore
    INFO[0024] Started GET /display/resize/100x110/test1.jpg
    INFO[0024] started handling request                      method=GET remote=[::1]:53370 request=/display/resize/100x110/test1.jpg
    INFO[0024] Key 2bdf7a15b14705854ead475c0b0e4f88 not found in kvstore
    INFO[0024] completed handling request                    measure#web.latency=49457115 method=GET remote=[::1]:53370 request=/display/resize/100x110/test1.jpg status=200 text_status=OK took=49.457115ms
    INFO[0024] Completed 200 OK in 49.568047ms
    INFO[0024] Save thumbnail 2bdf7a15b14705854ead475c0b0e4f88.jpg to storage
    INFO[0024] Save key 2bdf7a15b14705854ead475c0b0e4f88 => 2bdf7a15b14705854ead475c0b0e4f88.jpg to kvstore
    INFO[0024] Put key into set test1.jpg:children => 2bdf7a15b14705854ead475c0b0e4f88 in kvstore
    INFO[0033] Started GET /delete/test1.jpg
    INFO[0033] started handling request                      method=GET remote=[::1]:53370 request=/delete/test1.jpg
    INFO[0033] Deleting source storage file: test1.jpg
    INFO[0033] Deleting children set: test1.jpg:children
    INFO[0033] Deleting child f065a9618010d1c632cade29036c4d0c.jpg and its KV store entry f065a9618010d1c632cade29036c4d0c
    INFO[0033] Deleting child ee95aee309647ddd6711d26faf61fcc6.jpg and its KV store entry ee95aee309647ddd6711d26faf61fcc6
    INFO[0033] Deleting child 2bdf7a15b14705854ead475c0b0e4f88.jpg and its KV store entry 2bdf7a15b14705854ead475c0b0e4f88
    INFO[0033] Deleting child fd875c2d5e6cbca8a79467d76d1089af.jpg and its KV store entry fd875c2d5e6cbca8a79467d76d1089af
    INFO[0033] completed handling request                    measure#web.latency=457966 method=GET remote=[::1]:53370 request=/delete/test1.jpg status=200 text_status=OK took=457.966µs
    INFO[0033] Completed 200 OK in 552.788µs
    
    opened by nsf 11
  • Add 'allowed_sizes' configuration option

    Add 'allowed_sizes' configuration option

    This fixes #20.

    Provides a new configuration option for restricting the number of sizes:

    A few things about this PR I wasn't completely sure on:

    1. [No longer a question] I opted to go with an actual type for the AvailableSize configurations. I also considered using a []map[string]int, but wasn't sure what you would prefer.

    2. [No longer a question] I wasn't sure if it was appropriate for the engine to have knowledge of the errs package. I would have liked to keep the checking logic back in the application package, but the height and width parameters aren't coerced with defaults until the Transform call, so I think it was a little more appropriate to do the check there.

    3. I was up in the air between a couple formats for the config:

      "options": {
        "available_sizes": [
          "1920x1080",
          "720x480",
          "480x"
        ]
      }
      

      And what it currently is:

      "options": {
        "available_sizes": [
          {"w": 1920, "h": 1080},
          {"w": 720, "h": 480},
          {"w": 480}
        ]
      }
      

    Thanks for the consideration :-)

    opened by evanpurkhiser 7
  • Mimetype and quality

    Mimetype and quality

    Fixes #37

    We're no longer using the file extension to determine the mime and Content-Type (to cater for cases where extensions aren't available), and the ImageFile.Header["Content-Type"] is updated during transformation with the determined format's Content-Type.

    Also, I noticed the "quality" option and param weren't working which is now fixed.

    opened by steevee 7
  • Cannot load image with query string (Error: Request should contains parameters or query string)

    Cannot load image with query string (Error: Request should contains parameters or query string)

    Image loaded normally with url:

    http://<host>/display/thumbnail/300x225/location/image1.jpg
    

    but error with url:

    http://<host>/display?op=thumbnail&w=300&h=225&path=image1.jpg
    

    Error is:

    Request should contains parameters or query string
    

    This is my config:

    {
      "port": 80,
      "kvstore": {
        "type": "redis",
        "host": "<host>",
        "port": "<port>",
        "password": "<password>",
        "db": 0
      },
      "storage": {
        "src": {
          "type": "fs",
          "location": "/data/static/"
        },
        "dst": {
          "type": "http+fs",
          "base_url": "http://<url>/media",
          "location": "/data/static/generated"
        }
      }
    }
    

    p/s: In the past, the query string url worked fine.

    bug 
    opened by ndtuan412 6
  • ENV variables in config

    ENV variables in config

    Greetings,

    I was wondering if you'd welcome the addition of support for ENV varibales in config? We found it helpful from a docker image perspective to extract some config variables to ENV variables we set on the fly. This allowed us to use the same image regardless of configuration. Is this a feature you think we should submit a PR for?

    The other idea we had was to allow te passing of config inline, which might be even better than the ENV variables support.

    Let us know!

    Thanks,

    O

    opened by omarvelous 5
  • undefined syscall.Stat_t

    undefined syscall.Stat_t

    I would like to try to compile for windows if possible.

    I did a git clone on repo then

    go build 
    

    I got the following error

    undefined syscall.Stat_t

    Is there a simple solution to this issue?

    opened by tobya 4
  • Using picfit as a

    Using picfit as a "function"

    I want to use picfit as a function or as a simple webservice that will receive a image(url), crop it and output my thumbnail in the display/ endpoint. (then my app will download and do other things with the image)

    Just by using an empty config.json {}

    And something like: http://localhost:3001/display?url=http://domain/image.jpg&op=thumbnail&w=250&h=200

    I'm kinda getting the desired behavior, I just don't know if it the application is designed to achieve something like this.

    feature 
    opened by douglasmiranda 3
  • Cache misses

    Cache misses

    Hello

    Thank you for such a great little utility.

    I'm having some issue with caching the results of a image thumbnail and then the image not being recognised as being on disk when requested maybe a day later.

    This is my picfit config file

     {
      "port": 28080,
      "storage": {
      "src": {
      "type": "fs",
      "location": "d:\\picfit\\"
      }
     },
      "kvstore" : {
       "type" : "cache"
      },
      "options": {
        "enable_stats": true,
       "enable_delete": true
       }
     }
    

    (I tried including the sharding options but had same issue)

    I start it like this

    I get errors like this when I visit a page with multiple thumbnail requests produced by this request

    img src="https://imgur.azure.ccc.ie/display?url=https://bbb.ccc.ie/studentimages/623706.jpg&w=105&h=105&op=thumbnail" border=0>

    There is no possibility these have not been created before as they are on one of our most used pages.

    ←[36mINFO←[0m[2948] Key b74f659b92b4c5c4483cc2b3a9eca39b not found in kvstore ←[36mINFO←[0m[2954] Save thumbnail b74f659b92b4c5c4483cc2b3a9eca39b.jpg to storage ←[36mINFO←[0m[2954] Save key b74f659b92b4c5c4483cc2b3a9eca39b => b74f659b92b4c5c4483cc2b3a9eca39b.jpg to kvstore ←[36mINFO←[0m[2954] Put key into set d41d8cd98f00b204e9800998ecf8427e:children () => b74f659b92b4c5c4483cc2b3a9eca39b in kvstore

    I am using Windows Server 2012 with caddy and php 5.6. Is there anything obvious that would be causign these misses?

    need investigation 
    opened by tobya 3
  • User Agent for image factor (closes #64).

    User Agent for image factor (closes #64).

    DefaultUserAgent leaving room for overrides in future. Options section of config seemed most logical place for DefaultUserAgent. HTTPStorage.UserAgent property added to save passing user-agent around for every function. HTTPStorage can be used when creating http Srcand Dst storages, but as not involved in image factory DefaultUserAgent is not set.

    opened by steevee 3
  • Option to avoid using absolute url

    Option to avoid using absolute url

    Hi,

    What do you think if picfit has an option to disabled absolute url, only path is allowed? So the service only works with storage src. It will help to avoid the client/app uses external images that are not in the storage.

    Thanks.

    opened by huynhphan89 0
  • backblaze s3 v4 signing option

    backblaze s3 v4 signing option

    trying to integrate picfit and backblaze storage, and getting:

    The V2 signature authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256
    

    seems like goamz still not support v4 https://github.com/goamz/goamz/pull/118 but I can see workaround discussion https://github.com/cloudflare/complainer/issues/13

    does anybody made picfit working with backblaze?

    PS: this is my test code

    package main
    
    import (
      "github.com/mitchellh/goamz/aws"
      "github.com/mitchellh/goamz/s3"
      "log"
      "fmt"
    )
    
    func main() {
      auth, err := aws.EnvAuth()
      fmt.Printf("%#v\n", auth)
      if err != nil {
        log.Fatal(err)
      }
      client := s3.New(auth, aws.USEast)
      client.S3Endpoint = "https://s3.us-west-002.backblazeb2.com"
      fmt.Printf("%#v\n", client)
      resp, err := client.ListBuckets()
    
      if err != nil {
        log.Fatal(err)
      }
    
      log.Print(fmt.Sprintf("%T %+v", resp.Buckets[0], resp.Buckets[0]))
    }
    
    opened by theromis 1
  • [Feature Request] Maintain aspect ratio but add padding

    [Feature Request] Maintain aspect ratio but add padding

    I would like to request a feature where you can resize an image to a certain height and width while maintaining aspect ratio with the remaining space being filled with a foreground color

    Screen Shot 2020-05-31 at 7 51 46 PM

    Something like this where it adds the padding to the left and right

    opened by jafri 0
  • Wrong signature generation

    Wrong signature generation

    The command line signature test produce difference signature and url. The url always return invalid signature.

    ./picfit signature --key=abcdef "w=100&h=100&op=resize" Query String: w=100&h=100&op=resize Signature: 954eec9dfa9390f8f4264e15bb95f5f54e6fe953 Signed Query String: w=100&h=100&op=resize&sig=6f7a667559990dee9c30fb459b88c23776fad25e

    opened by beannt 1
  • GIF transform memory usage

    GIF transform memory usage

    Today stepped on issue of transforming 37mb animated gif, used all memory on my 5$ DigitalOcean VM and killed server completely. Maybe we can add ability to limit memory processing buffer or it even exists?

    feature 
    opened by theromis 1
Releases(0.9.0)
Owner
Florent Messa
CTO @ulule
Florent Messa
Pure golang image resizing

This package is no longer being updated! Please look for alternatives if that bothers you. Resize Image resizing for the Go programming language with

Jan Schlicht 2.9k Nov 24, 2022
Image resizing in pure Go and SIMD

rez Package rez provides image resizing in pure Go and SIMD. Download: go get github.com/bamiaux/rez Full documentation at http://godoc.org/github.com

Benoît Amiaux 209 Oct 18, 2022
Image resizing for the Go programming language with common interpolation methods

This package is no longer being updated! Please look for alternatives if that bothers you. Resize Image resizing for the Go programming language with

null 1 Dec 14, 2021
Fast and secure standalone server for resizing and converting remote images

imgproxy imgproxy is a fast and secure standalone server for resizing and converting remote images. The main principles of imgproxy are simplicity, sp

imgproxy 6.7k Nov 29, 2022
Image - This repository holds supplementary Go image librariesThis repository holds supplementary Go image libraries

Go Images This repository holds supplementary Go image libraries. Download/Insta

null 0 Jan 5, 2022
Cloud function + website for resizing, cropping and bordering images for pragalicious.com

Cloud function + website for resizing, cropping and bordering images for pragalicious.com

Ard Scheirlynck 0 Jan 23, 2022
darkroom - An image proxy with changeable storage backends and image processing engines with focus on speed and resiliency.

Darkroom - Yet Another Image Proxy Introduction Darkroom combines the storage backend and the image processor and acts as an Image Proxy on your image

Gojek 202 Nov 27, 2022
Easily customizable Social image (or Open graph image) generator

fancycard Easily customizable Social image (or Open graph image) generator Built with Go, Gin, GoQuery and Chromedp Build & Run Simply, Clone this rep

Youngbin Han 4 Jan 14, 2022
An API which allows you to upload an image and responds with the same image, stripped of EXIF data

strip-metadata This is an API which allows you to upload an image and responds with the same image, stripped of EXIF data. How to run You need to have

Cristina Simionescu 0 Nov 25, 2021
Imgpreview - Tiny image previews for HTML while the original image is loading

imgpreview This is a Go program that generates tiny blurry previews for images t

Dmitry Chestnykh 8 May 22, 2022
Storage and image processing server written in Go

Mort An S3-compatible image processing server written in Go. Still in active development. Features HTTP server Resize, Rotate, SmartCrop Convert (JPEG

Marcin Kaciuba 472 Nov 30, 2022
An image server toolkit in Go (Golang)

Image Server An image server toolkit in Go (Golang) Features HTTP server Resize (GIFT, nfnt resize, Graphicsmagick) Rotate Crop Convert (JPEG, GIF (an

Pierre Durand 2k Nov 28, 2022
Pixelizer is an efficient image pixelizer written in go

Pixelizer is an image pixelizer written in go. It allows very simple and intuitive CLI pixelization. Installation To install Pixelizer, you

Rewan_ 103 Nov 10, 2022
ColorX is a library to determine the most prominent color in an image written in golang

ColorX is a library to determine the most prominent color in an image. ColorX doesn't use any sort of complex algorithms to calculate the prominent color, it simply loops over the image pixels and returns the color that occurs the most.

Hesham Abourgheba 1 Nov 11, 2021
Pbm - Package ppm implements a Portable Bit Map (PBM) image decoder and encoder written in Go

Package pbm import "github.com/slashformotion/pbm" Package pbm implements a Portable Bit Map (PBM) image decoder and encoder. The supported image col

slashformotion 0 Jan 5, 2022
Image processing algorithms in pure Go

bild A collection of parallel image processing algorithms in pure Go. The aim of this project is simplicity in use and development over absolute high

Anthony N. Simon 3.7k Dec 3, 2022
Go package for fast high-level image processing powered by libvips C library

bimg Small Go package for fast high-level image processing using libvips via C bindings, providing a simple programmatic API. bimg was designed to be

Tom 2.1k Dec 3, 2022
Image processing library and rendering toolkit for Go.

blend Image processing library and rendering toolkit for Go. (WIP) Installation: This library is compatible with Go1. go get github.com/phrozen/blend

Guillermo Estrada 61 Nov 11, 2022
Decode embedded EXIF meta data from image files.

goexif Provides decoding of basic exif and tiff encoded data. Still in alpha - no guarantees. Suggestions and pull requests are welcome. Functionality

Robert Carlsen 540 Nov 19, 2022