Live on-demand transcoding in go using ffmpeg. Also with NVIDIA GPU hardware acceleration.

Overview

Go live HTTP on-demand transcoding

Transcoding is expensive and resource consuming operation on CPU and GPU. For big companies with thousands of customers it is essential, to have a dedicated 24/7 transcoding servers. But we, single sporadic users of transcoding, need to have different approach. Transcoding should be done only when its output is really needed. This tool is trying to solve this problem by offering transcoding on demand.

This tool is intended to be used with live streams only. Seeking is not supported, yet.

Config

Specify streams as object in yaml file.

Streams

Create streams.yaml file, with your streams:

streams:
  : 

Example:

streams:
  cam: rtmp://localhost/live/cam
  ch1_hd: http://192.168.1.34:9981/stream/channelid/85
  ch2_hd: http://192.168.1.34:9981/stream/channelid/43

HTTP streaming is accessible via:

  • http://localhost:8080//

HLS is accessible via:

  • http://localhost:8080///index.m3u8
  • http://localhost:8080///play.html

CPU Profiles

Profiles (HTTP and HLS) with CPU transcoding can be found in profiles:

  • h264_360p
  • h264_540p
  • h264_720p
  • h264_1080p

Profile names must match flowing regex: ^[0-9A-Za-z_-]+$

GPU Profiles

Profiles (HTTP and HLS) with GPU transcoding can be found in profiles_nvidia:

  • h264_360p
  • h264_540p
  • h264_720p
  • h264_1080p

Profile names must match flowing regex: ^[0-9A-Za-z_-]+$

Docker

Build

docker build -t go-transcode:latest .

Run

docker run --rm -d \
  --name="go-transcode" \
  -p "8080:8080" \
  -v "${PWD}/streams.yaml:/app/streams.yaml" go-transcode:latest

Nvidia GPU support (docker)

You will need to have nvidia-docker installed.

Build

First, you need to build previous container. Then, build Nvidia container.

docker build --build-arg "TRANSCODE_IMAGE=go-transcode:latest" -t go-transcode-nvidia:latest -f Dockerfile.nvidia .

Run

docker run --rm -d \
  --gpus=all \
  --name="go-transcode-nvidia" \
  -p "8080:8080" \
  -v "${PWD}/streams.yaml:/app/streams.yaml" go-transcode-nvidia:latest

Supported inputs

Input codec will be automatically determined from given stream. Please check your graphic card's supported codec and maximum concurrent sessions here.

Codec CUVID Codec Name
h264 h264_cuvid H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
hevc hevc_cuvid H.265 / HEVC
mjpeg mjpeg_cuvid Motion JPEG
mpeg1video mpeg1_cuvid MPEG-1 video
mpeg2video mpeg2_cuvid MPEG-2 video
mpeg4 mpeg4_cuvid MPEG-4 part 2
vc1 vc1_cuvid SMPTE VC-1
vp8 vp8_cuvid On2 VP8
vp9 vp9_cuvid Google VP9
Comments
  • transcode static video files

    transcode static video files

    inspiration hls-vod-too

    • ffprobe to find duration and segment count
    • transcode/cache by segment number in file

    edit: for static streams maybe config should support directories? library: /media/video in config can serve all files/directories with /profile/library/folder/video.mkv/index.m3u8 (or other url format from #13)

    opened by klahaha 13
  • 2nd+ streams lead to errors that can cause some players to not work.

    2nd+ streams lead to errors that can cause some players to not work.

    The first person to get the stream works, video plays back fine. any one after will get errors that cause the video to freeze or the player to refuse to play

    See the attached video

    https://user-images.githubusercontent.com/39884745/141701010-e01b7afc-674c-4485-b930-6abff0671df8.mp4

    This happens on lan and over hls proxy. I have tried changing the type of audio as well happens to both aac and mp3 Using the latest git docker build as well.

    bug 
    opened by noideaman 12
  • Some refactoring

    Some refactoring

    • [x] streams are loaded from viper config, not streams.yaml
    • [x] cmd is built by go build in repo root (main package)
    • [x] support /etc/transcode
      • [x] remove hardcoded paths
      • [x] config file is loaded from /etc/transcode or ./
      • [x] profiles/data is served from /etc/transcode or ./
      • [x] everything is relative to basedir, which is passed via CLI, or is /etc/transcode (if exists), or is ./
    • [x] config file is automatically reloaded in serve mode
    • [x] remove some unused stuff
      • [x] removed Config type
      • [x] merge some small files together for clarity
      • [x] ~~remove Server.Proxy unused~~ (moved to #4)
      • [x] ~~remove TLS support (Server.Key and Server.Cert) encourage reverse proxy~~ add warning that TLS mode should not be used for security
    • [x] ~~move CMD parsing in internal/config to cmd package~~ future
    • [x] fix Docker build
    • [x] Write some tests
      • [x] settings cascade (test_args.sh)
      • [x] settings auto-reload
    • [x] Use settings for ffmpeg profiles folder
      • [x] Move all to profiles/default and profiles/nvidia
      • [x] Use a profiles directory setting to find the profiles
    • [x] make room in repo root
      • [x] embed data/play.html in binary from internal/api/hls/
      • [x] move data/*-test.sh to tests/
      • [x] remove data dir from repo
      • [x] remove bin dir from repo
      • [x] ~~move everything docker to packaging/docker dir?~~ future
    • [x] ~~transcodeStart takes URL not stream name~~ future
    • [x] Update readme
      • [x] build and run with docker
      • [x] build and run without docker
      • [x] info about project structure
      • [x] ~~info about code flow for request process~~ future

    feedback welcome (i'm new to golang)

    also there's a lot of stuff in root dir it's littlee confusing, do you like to move source to src/ (not sure golang likes that), build/test scripts (dev/ data/ and tests) to bin/, and profiles mixed together, like this:

    • bin/
    • src/
    • profiles/
    • what about docker files? is there a standard directory to place it?
    • LICENSE README.md go.mod go.sum ...
    opened by klahaha 12
  • VAAPI support

    VAAPI support

    Updated hls_h264 and http to support vaapi detection Updated readme on how to add dri devices to containers added helper vaapi script to detect supported encoders/decoders updated Dockerfile to use alpine edge to get access to drivers and utilities added alpine edge testing repo to get access to drivers and utilities

    opened by noideaman 3
  • Auto-detect hardware acceleration

    Auto-detect hardware acceleration

    We can detect hardware acceleration in program. Using ffmpeg cli

    ffmpeg -init_hw_device list 2> /dev/null
    Supported hardware device types:
    vdpau
    vaapi
    qsv
    drm
    opencl
    

    then no need for profiles config just print what profiles are used on startup

    what outputs the command with nvidia proprietaire driver?

    opened by klahaha 2
  • Error on probe

    Error on probe

    Error: 4:47PM ERR unable to load metadata error="unable probe video for keyframes: strconv.ParseFloat: parsing "": invalid syntax"

    When I do the ffprobe in terminal I get this result: ffprobe -v error -skip_frame nokey -show_entries frame=pkt_pts_time,side_data_list -show_entries format=duration -show_entries stream=duration,width,height -select_streams v -of json MYVIDEO.mkv

    {
        "frames": [
            {
                "side_data_list": [
                    {
    
                    },
                    {
    
                    }
                ]
            },
            {
                "pkt_pts_time": "0.000000"
            },
            {
                "pkt_pts_time": "2.002000"
            },
            {
                "pkt_pts_time": "3.712000"
            },
            {
                "pkt_pts_time": "8.592000"
            },
    .............................
    

    Could be the side_data_list?

    opened by rudnypc 1
  • Add pprof debug endpoint

    Add pprof debug endpoint

    This endpoint should help with debugging in the future.

    More info here: https://pkg.go.dev/net/http/pprof


    Types of profiles available:

    • allocs
    • block
    • cmdline
    • goroutine
    • heap
    • mutex
    • profile
    • threadcreate
    • trace

    Profile Descriptions:

    • allocs:
      A sampling of all past memory allocations
    • block:
      Stack traces that led to blocking on synchronization primitives
    • cmdline:
      The command line invocation of the current program
    • goroutine:
      Stack traces of all current goroutines
    • heap:
      A sampling of memory allocations of live objects. You can specify the gc GET parameter to run GC before taking the heap sample.
    • mutex:
      Stack traces of holders of contended mutexes
    • profile:
      CPU profile. You can specify the duration in the seconds GET parameter. After you get the profile file, use the go tool pprof command to investigate the profile.
    • threadcreate:
      Stack traces that led to the creation of new OS threads
    • trace:
      A trace of execution of the current program. You can specify the duration in the seconds GET parameter. After you get the trace file, use the go tool trace command to investigate the trace.
    opened by m1k1o 0
  • HLS exit on error

    HLS exit on error

    Right now, when command finishes earlier (because of error) or does not start properly, it won't communincate error and we need to wait for timeout to finish. I would expect the function onStop to have error parameter, and firing when command stops.

    enhancement 
    opened by m1k1o 0
  • static file access

    static file access

    Trying to test static files. the config file is pointed to /media has full access to the folder mounted

    mpv http://IPADDR:8888/vod/TV/SouthPark/Specials/South_Park_-_S00E44_-_South_Park-_Post_COVID-_The_Return_of_COVID_WEBDL-1080p.mkv/index.m3u8

    get an error 500 and the output with debug on on go-transcode

    12:46AM INF fetching metadata module=hlsvod submodule=manager
    2021/12/28 00:46:13 Unrecognized option 'show_format'.
    Error splitting the argument list: Option not found
    
    12:46AM WRN unable to preload metadata error="unable probe media for metadata: exit status 1" module=hlsvod
    12:46AM DBG request complete (500) module=http req={"agent":"libmpv","id":"go-transcode/NeOacMNoCW-000004","method":"GET","proto":"HTTP/1.1","remote":"192.168.6.133:1082","scheme":"http","uri":"http://IPADDR:8888/vod/TV/SouthPark/Specials/South_Park_-_S00E44_-_South_Park-_Post_COVID-_The_Return_of_COVID_WEBDL-1080p.mkv/index.m3u8"} res={"bytes":31,"elapsed":11.91186,"status":500,"time":"Tue, 28 Dec 2021 00:46:13 UTC"}
    12:46AM DBG request complete (405) module=http req={"agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36","id":"go-transcode/NeOacMNoCW-000005","method":"HEAD","proto":"HTTP/1.1","remote":"192.168.6.133:1083","scheme":"http","uri":"http://IPADDR:8888/vod/TV/SouthPark/Specials/South_Park_-_S00E44_-_South_Park-_Post_COVID-_The_Return_of_COVID_WEBDL-1080p.mkv/index.m3u8"} res={"bytes":0,"elapsed":0.013375,"status":405,"time":"Tue, 28 Dec 2021 00:46:14 UTC"}
    12:46AM INF fetching metadata module=hlsvod submodule=manager
    2021/12/28 00:46:14 Unrecognized option 'show_format'.
    Error splitting the argument list: Option not found
    
    12:46AM WRN unable to preload metadata error="unable probe media for metadata: exit status 1" module=hlsvod
    12:46AM DBG request complete (500) module=http req={"agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36","id":"go-transcode/NeOacMNoCW-000006","method":"GET","proto":"HTTP/1.1","remote":"192.168.6.133:1084","scheme":"http","uri":"http://IPADDR:8888/vod/TV/SouthPark/Specials/South_Park_-_S00E44_-_South_Park-_Post_COVID-_The_Return_of_COVID_WEBDL-1080p.mkv/index.m3u8"} res={"bytes":31,"elapsed":11.783193,"status":500,"time":"Tue, 28 Dec 2021 00:46:14 UTC"}
    
    opened by noideaman 8
  • ffprobe cache

    ffprobe cache

    When the cache is true, can we save the ffprobe stream result? Every request to the same file with different resolution (maybe the same too), the ffprobe is running again.

    bug 
    opened by rudnypc 9
  • Closing ffmpeg when isn't used

    Closing ffmpeg when isn't used

    The ffmpeg will be killed after 50 segments are created. But When I start a video with a index file, the vlc start all playlists and some ffmpeg instances are started, this slows down the server, maybe there should be a better control to disable ffmpeg when not used, and when used, increase the limit from 50 to another after new requests, so there is no cut in audio between old and new segments.

    enhancement 
    opened by rudnypc 4
  • Cache data write race

    Cache data write race

    Sometimes this fails. Index out of range.

    https://github.com/m1k1o/go-transcode/blob/4ec3da08795fba33312a88c7e5698c1518e73921/internal/utils/cache.go#L78-L80

    bug 
    opened by m1k1o 0
Releases(v0.3.5)
Owner
Miroslav Šedivý
Miroslav Šedivý
[WIP] a very simple, tiny and intuitive ffmpeg wrapper with a cli interface for inspecting & transforming media files supported by the original ffmpeg software

About a very simple, tiny and intuitive ffmpeg wrapper with a cli interface for inspecting & transforming media files supported by the original ffmpeg

Mohammed Al Ashaal 9 Jun 28, 2022
A small program in Go that efficiently compresses videos using ffmpeg.

discordcompressor A small program in Go that efficiently compresses videos using ffmpeg. Dependencies FFmpeg including FFprobe Usage discordcompressor

null 25 Aug 28, 2022
golang function that download a video from youtube, and convert it to a mp3 file using ffmpeg

echedwnmp3 echedwnmp3 is a function that download a video from youtube, and convert it to a mp3 file using ffmpeg example package main import(echedwn

pai 4 Dec 7, 2021
lmmp3 is a little golang library that download a video from youtube, and convert it to a mp3 file using ffmpeg

lmmp3 lmmp3 is a function that download a video from youtube, and convert it to a mp3 file using ffmpeg You need to have installed ffmpeg in your syst

pai 8 Aug 12, 2022
Golang bindings for FFmpeg

goav Golang binding for FFmpeg A comprehensive binding to the ffmpeg video/audio manipulation library. Usage import "github.com/giorgisio/goav/avforma

H. Giorgis 1.9k Sep 29, 2022
A Go implementation of fluent-ffmpeg

A Go implementation of fluent-ffmpeg

Modfy 312 Sep 16, 2022
Stream video from ffmpeg to webrtc

ffmpeg-to-webrtc demonstrates how to send video from ffmpeg to your browser using pion.

Artur Shellunts 867 Sep 8, 2022
Videncode - Media Encoder (with ffmpeg)

Videncode - Media Encoder (with ffmpeg) Powered by yellyoshua (With2 easy steps) - Build JSON with folder of videos > Process the videos to the new fo

Yoshua Lopez 2 Oct 25, 2021
ffmpeg core for golang

go-ffmpeg-core ffmpeg core for golang 基于ffmpeg命令封装简单功能,因此运行环境需事先安装有ffmpeg命令。 ffmpeg官方下载地址: http://ffmpeg.org/download.html 1. 剥离视频文件的音频/视频 package mai

Resynz 0 Nov 26, 2021
ffcommander - An easy frontend to FFmpeg and Imagemagick to automatically process video and manipulate subtitles.

% FFCOMMANDER(1) ffcommander 2.39 % Mikael Hartzell (C) 2018 % 2021 Name ffcommander - An easy frontend to FFmpeg and Imagemagick to automatically pro

Mikael Hartzell 2 May 9, 2022
A go program that relies on back-end ffmpeg to process video-related content

Video Compress A go program that relies on back-end ffmpeg to process video-related content Installation v-go You can download the corresponding v-go

JokerHeyra 0 Dec 22, 2021
Parse and generate m3u8 playlists for Apple HTTP Live Streaming (HLS) in Golang (ported from gem https://github.com/sethdeckard/m3u8)

go-m3u8 Golang package for m3u8 (ported m3u8 gem https://github.com/sethdeckard/m3u8) go-m3u8 provides easy generation and parsing of m3u8 playlists d

Tan Quang Ngo 98 Aug 28, 2022
🔥 Golang live stream lib/client/server. support RTMP/RTSP/HLS/HTTP[S]-FLV/HTTP-TS, H264/H265/AAC, relay, cluster, record, HTTP API/Notify, GOP cache. 官方文档见 https://pengrl.com/lal

lal是一个开源GoLang直播流媒体网络传输项目,包含三个主要组成部分: lalserver:流媒体转发服务器。类似于nginx-rtmp-module等应用,但支持更多的协议,提供更丰富的功能。lalserver简介 demo:一些小应用,比如推、拉流客户端,压测工具,流分析工具,调度示例程序等

yoko 1.7k Sep 22, 2022
Take control over your live stream video by running it yourself. Streaming + chat out of the box.

Take control over your content and stream it yourself. Explore the docs » View Demo · Use Our Server for Testing · FAQ · Report Bug Table of Contents

Owncast 6.4k Sep 30, 2022
live video streaming server in golang

中文 Simple and efficient live broadcast server: Very simple to install and use; Pure Golang, high performance, and cross-platform; Supports commonly us

浩麟 8.3k Sep 27, 2022
live streaming server in golang

中文 Simple and efficient live broadcast server: Very simple to install and use; Pure Golang, high performance, and cross-platform; Supports commonly us

BigSillyBear 26 Aug 18, 2022
A live streaming tool more suitable for non-graphical servers

KPlayer KPlayer可以帮助你快速的在服务器上进行视频资源的循环直播推流。

null 130 Oct 1, 2022
Plays videos using Prometheus and Grafana, e.g. Bad Apple.

prometheus_video_renderer Plays videos using Prometheus and Grafana, e.g. Bad Apple. Modes Currently 3 different modes are supported. Bitmap The bitma

Jacob Colvin 89 Sep 5, 2022
Synthetic media is a realistic transformation of audio and video using artificial intelligence.

Synthetic media is a realistic transformation of audio and video using artificial intelligence.

null 1 Nov 20, 2021