Win32 IO-related utilities for Go

Overview

go-winio Build Status

This repository contains utilities for efficiently performing Win32 IO operations in Go. Currently, this is focused on accessing named pipes and other file handles, and for using named pipes as a net transport.

This code relies on IO completion ports to avoid blocking IO on system threads, allowing Go to reuse the thread to schedule another goroutine. This limits support to Windows Vista and newer operating systems. This is similar to the implementation of network sockets in Go's net package.

Please see the LICENSE file for licensing information.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

We also require that contributors sign their commits using git commit -s or git commit --signoff to certify they either authored the work themselves or otherwise have permission to use it in this project. Please see https://developercertificate.org/ for more info, as well as to make sure that you can attest to the rules listed. Our CI uses the DCO Github app to ensure that all commits in a given PR are signed-off.

Code of Conduct

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Special Thanks

Thanks to natefinch for the inspiration for this library. See https://github.com/natefinch/npipe for another named pipe implementation.

Comments
  • fixes #67 DialPipe problem with multiple calls / waiting for busy pipe

    fixes #67 DialPipe problem with multiple calls / waiting for busy pipe

    (I believe this change also should be used instead of PR #75 -- that PR is potentially extraordinarily buggy.)

    This changes a few things to try to ensure that we never wind up with a result of ERROR_FILE_NOT_FOUND, due to a race between closing the last pipe instance and opening the next.

    First we keep an "open" client instance (unused) while the listener is open, so that we are guaranteed to always have an active pipe instance. This means attempts to open while no other instances exist result in ERROR_PIPE_BUSY instead of ERROR_FILE_NOT_FOUND.

    Second we have changed the loop for dialing to eliminate a race condition that is more or less inherent in WaitNamedPipe when synchronizing with CreateFile. The real timeout needs to be some larger value than the WaitNamedPipe timeout, and furthermore WaitNamedPipe is not very nice with the Go runtime, since it is a blocking system call. Instead we just put the goroutine to sleep for 10 milliseconds, and keep retrying the CreateFile until the maximum timeout is reached. If no timeout is specified we assume a reasonable and large default of 5 seconds, which is similar to a TCP connection timeout.

    This isn't perfect, as a client attempting to connect to an extremely busy pipe server can be starved out by other clients coming in while it is in that brief sleep, but this potential race was already present with WaitNamedPipe. The numerous retries (by default 500 retries!) mean its pretty unlikely to occur, and if a single client hits the race once, it has an excellent chance of getting in the next cycle.

    (A real "fix" that is completely race free and fair would require changes in the underlying Named Pipe implementation, or some other kind of external coordination.)

    opened by gdamore 30
  • Retry named pipe connection on race condition

    Retry named pipe connection on race condition

    If the named pipe server does not have an outstanding call to ConnectNamedPipe() at the time the client calls CreateFile() or WaitNamedPipe(), the client can receive ERROR_FILE_NOT_FOUND and ERROR_BAD_PATHNAME errors. When that's the case, there is no choice but to retry after a short delay in the hope that next time, the server will be able to accept the connection.

    This error occurs infrequently in low traffic situations, but it manifests quite reliably when clients open concurrent connections and/or open many short-lived connections in a short period of time.

    Fixes #67.

    opened by AndreLouisCaron 21
  • Apply read and write deadlines to pending IO

    Apply read and write deadlines to pending IO

    In order to upgrade to go1.8 all IO operations including previous pending IO must have the deadline set when SetReadDeadline or SetWriteDeadline is called.

    This moves the timeout cancelation responsibility out of the wait routine and into timers in the deadline code.

    @jstarks PTAL

    Signed-off-by: Darren Stahl [email protected]

    opened by darstahl 11
  • file: Fix race in closing

    file: Fix race in closing

    Hello!

    It looks like there's a race condition created in (*win32File).closeHandle() where multiple concurrent goroutines are accessing the (*win32File).closing field. This shows up in the go race detector. This change adds a lock around the closing field to remove the race.

    Thanks! Sam

    opened by samuelkarp 9
  • provide access to underlying file handle

    provide access to underlying file handle

    It is convenient to be able to pull off the underlying file handle to pass to other Win32 functions that aren't supported by this library.

    This change adds an Fd() uintptr method to the file implementation (matches the os.File method of the same name).

    opened by azdagron 8
  • DialPipe problem with multiple calls / waiting for busy pipe

    DialPipe problem with multiple calls / waiting for busy pipe

    Currently there is some work in progress (https://github.com/portainer/portainer/pull/1186) to add mapped named pipe support into Portainer so it will be easier to run it in a Windows container ( docker run -u ContainerAdministrator -it -v //./pipe/docker_engine://./pipe/docker_engine -p 9000:9000 stefanscherer/portainer:insider -H npipe:////./pipe/docker_engine )

    We're using winio.DialPipe() to connect to the Docker engine from inside a microsoft/nanoserver-insider container.

    At the moment we see problems in Portainer as it sometimes makes multiple requests requested by the web frontend. Here's a WireShark screenshot with the problem.

    bildschirmfoto 2017-09-17 um 07 42 23

    The two /api/endpoints/1/docker/xx GET requests use separate connections and run in parallel. One of the calls aborts with "open //./pipe/docker_engine: The system cannot find the file specified."

    I have seen the check for cERROR_PIPE_BUSY and first thought that createFile() returns cERROR_FILE_NOT_FOUND and aborts immediatelly.

    We are using the default timeout right now

        return winio.DialPipe(namedPipePath, nil)
    

    but I also tried giving a 5 second timeout

        timeout := 5000 * time.Millisecond
        return winio.DialPipe(namedPipePath, &timeout)
    

    Digging deeper it seems like the waitNamedPipe() just does not wait long enough or cannot start wait for the pipe.

    Adding some Println in DialPipe I now see this running Portainer in the second DialPipe call:

    DialPipe createFile err All pipe instances are busy. DialPipe waitNamedPipe ms 4998 DialPipe waitNamedPipe returns err The system cannot find the file specified. DialPipe returns err The system cannot find the file specified.

    The problem is that waitNamedPipe returns immediatelly and does not wait the given 5000 milliseconds.

    How can we make DialPipe more reliable?

    opened by StefanScherer 8
  • HVsock Registry Application Registration

    HVsock Registry Application Registration

    Using Hyper-V sockets requires registering the application in the Windows Registry.

    This PR adds (de)registration functionality for an application name/GUID.

    This PR is split from #235

    Signed-off-by: Hamza El-Saawy [email protected]

    opened by helsaawy 7
  • Fix regression in DetachVhd

    Fix regression in DetachVhd

    When opening a disk for detach mode no parameters are required. This regression was introduced when the OpenVirtualDisk method was added. Moving the code back to the previous openVirtualDisk internal method so we can pass nil parameters.

    Signed-off-by: Justin Terry (VM) [email protected]

    opened by jterry75 7
  • Prevent data race when closing active file

    Prevent data race when closing active file

    The race usually happens when closeHandle() and prepareIo() are called concurrently; the former tries to set closing to true the latter tries to read its value.

    In order to avoid this issue, we added a lock around the variable.

    Signed-off-by: Kenfe-Mickael Laventure [email protected]

    opened by mlaventure 7
  • Added runtime.KeepAlive for the new stricter 1.8 GC

    Added runtime.KeepAlive for the new stricter 1.8 GC

    fixes #41

    As far as I can tell, this should contain all the runtime.KeepAlive calls needed to conform to the new argument liveness guarantees in Go1.8.x

    It turns out that the main issue was due to passing a Go pointer via native calls to the ioCompletionProcessor, causing the new garbage collection rules to cleanup the struct containing the channel even though it did not have a finalizer.

    /cc @jhowardmsft @jstarks PTAL

    Thanks @mappu for the initial investigation and repro test.

    This also updates zsyscall_windows.go with the new 1.8 go generated code, though it is not related to the KeepAlive, it should be done with the update to 1.8.

    Signed-off-by: Darren Stahl [email protected]

    opened by darstahl 7
  • Implement `winio.GetFileStandardInfo`

    Implement `winio.GetFileStandardInfo`

    This get-only API will be useful for detecting when a file has multiple links to it.

    Specifically, I plan to use this to provide an optimisation for https://github.com/microsoft/hcsshim/pull/901 (or afterwards, it's not a blocker...) by only attempting to recognise hard-links for files which have multiple links.

    See https://github.com/TBBle/hcsshim/compare/base-layer-manipulation...TBBle:hack-base-layer-manipulation-with-branched-go-winio for the example use-case.

    opened by TBBle 6
  • CloseWrite() is non-portable and may leave client connection hanging

    CloseWrite() is non-portable and may leave client connection hanging

    We currently use winio's named pipe implementation to create http duplex sockets.

    We use CloseWrite() to close the server side of the socket.

    This works ok if the client is also using winio. But if the client uses a different named pipe implemenation, CloseWrite() does not behave correctly. (in particular, NodeJS/libuv's implementation)

    My (very naive) understanding is the winio uses a 0-length message to signal the end of the socket. https://github.com/microsoft/go-winio/blob/d68e55cd0b80e6e3cac42e945f1cff6ddb257090/pipe.go#L161 which other libraries do not support.

    Can you all advise on if there's something we should be doing differently here?

    Thank you!


    Detailed repro steps:

    Here's how this breaks things in application-level code: https://github.com/apocas/dockerode/issues/534

    Here's a more detailed repro case:

    server side:

    // main.go
    
    package main
    
    import (
    	"log"
    	"fmt"
    	"github.com/Microsoft/go-winio"
    )
    
    type CloseWriter interface {
    	CloseWrite() error
    }
    
    func main() {
    	pipe, err := winio.ListenPipe("//./pipe/test-socket", &winio.PipeConfig{
    		MessageMode:        true,
    		InputBufferSize:    65536,
    		OutputBufferSize:   65536,
    	})
    	if err != nil {
    		log.Fatal(err)
    	}
    	fmt.Println("Accepting connections on //./pipe/test-socket")
    	for {
    
    		conn, err := pipe.Accept()
    		if err != nil {
    			log.Fatal(err)
    	  }
    		fmt.Println("opened socket")
      	go func() {
    
    			conn.Write([]byte("HTTP/1.1 101 Web Socket Protocol Handshake\r\n" +
    				"Upgrade: WebSocket\r\n" +
    				"Connection: Upgrade\r\n" +
    				"\r\n"));
    			conn.Write([]byte("closing socket\r\n"))
    			conn.(CloseWriter).CloseWrite()
    			fmt.Println("closed socket")
    		}()
    	}
    }
    

    client side:

    // index.js
    const http = require('http');
    const options = {
      headers: {
        'Connection': 'Upgrade',
        'Upgrade': 'tcp'
      },
      socketPath: '//./pipe/test-socket'
    };
    
    const req = http.request(options);
    req.on('upgrade', (res, socket, upgradeHead) => {
        console.log('connected to socket');
        socket.allowHalfOpen = false;
        socket.on('data', (data) => {
            console.log('socket received data:', data.toString());
        });
        socket.on('end', () => {
            console.log('socket closed');
        });
    });
    req.end();
    

    Then run (in separate terminals):

    go run ./main.go
    node index.js
    

    expected behavior: WinIO should close the socket and nodejs should detect the 'end' event.

    actual behavior: nodejs hangs indefinitely

    Other notes: The workaround is to use Close() on the Go side instead of CloseWrite(). This closes the whole duplex (rather than closing just one side)

    opened by nicks 1
  • It doesn't support etw NTStatus field.

    It doesn't support etw NTStatus field.

    I make etw events by "github.com/Microsoft/go-winio/pkg/etw". but it doesn't support NTStatus type field. example in pkg\etw\fieldopt.go, Uint32Field. but outTypeDefault change to outTypeNTStatus. Thanks!

    opened by wesley2005 0
  • DialPipe || DialPipeContext permission denied reading/writting to pipe

    DialPipe || DialPipeContext permission denied reading/writting to pipe

    Windows 10 \.\pipe\openssh-ssh-agent

    I'm getting permission denied when trying to list keys or write keys to ssh-agent. I was able to do this in the past. All I see in event viewer is LRPC permission denied but very little details.

    Is there something I need to do to set permissions in order to access a named pipe?

    I usually stay in the Linux world where his is easier.

    opened by r4yv3n 0
  • bluetooth socket question

    bluetooth socket question

    Can I ask, the oficial library do not contain bluetooth socket api, do you provide it? Or is it on plan? The problem is: I crreate socket and I do not know bind becouse bluetooth address do not exist.........

    package main
    
    import "golang.org/x/sys/windows"
    
    func main() {
    	sock, err := windows.Socket(windows.AF_BTH, windows.SOCK_STREAM, windows.BTHPROTO_RFCOMM)
    	if err != nil {
    		panic(err)
    	}
    	err = windows.Bind(sock,?)
    	if err != nil {
    		panic(err)
    	}
    	return
    }
    
    
    opened by MatejMagat305 0
Releases(v0.5.2)
  • v0.5.2(Feb 17, 2022)

    What's Changed

    • Allow guid package to be used on non-Windows GOOS targets by @dcormier in https://github.com/microsoft/go-winio/pull/169
    • Fix TestLookupEmptyNameFails by @bitgestalt in https://github.com/microsoft/go-winio/pull/225
    • Replace github.com/pkg/errors with stdlib errors by @bitgestalt in https://github.com/microsoft/go-winio/pull/227
    • Fix 'OpenVirtualDiskParameters' BOOL fields by @dcantah in https://github.com/microsoft/go-winio/pull/226
    • Add functions for SDDL, EA & Reparse encoding by @ambarve in https://github.com/microsoft/go-winio/pull/220
    • HvsockConn shutdown bugfix, added .IsClosed() functions by @helsaawy in https://github.com/microsoft/go-winio/pull/231

    Full Changelog: https://github.com/microsoft/go-winio/compare/v0.5.1...v0.5.2

    Source code(tar.gz)
    Source code(zip)
  • v0.4.20(Oct 15, 2021)

    What's Changed

    • [release/0.4] Backport fixes from v0.5.1 to 0.4 by @kevpar in https://github.com/microsoft/go-winio/pull/222

    Full Changelog: https://github.com/microsoft/go-winio/compare/v0.4.19...v0.4.20

    Source code(tar.gz)
    Source code(zip)
  • v0.5.1(Oct 15, 2021)

    What's Changed

    • Fix syscall error handling in vhd and pkg/security by @kevpar in https://github.com/microsoft/go-winio/pull/210
    • Fix corruption in x86 callback by @kylewo in https://github.com/microsoft/go-winio/pull/214
    • Update README with DCO information + re-order into sections by @dcantah in https://github.com/microsoft/go-winio/pull/219
    • backuptar: Fix sparse file handling by @kevpar in https://github.com/microsoft/go-winio/pull/221
    • Fixing corruption in callbacks introduced by x86 changes by @kylewo in https://github.com/microsoft/go-winio/pull/217

    Full Changelog: https://github.com/microsoft/go-winio/compare/v0.5.0...v0.5.1

    Source code(tar.gz)
    Source code(zip)
  • v0.5.0(Apr 23, 2021)

    • Added GetFileStandardInfo which returns information from the GetFileInformationByHandleEx syscall with FileStandardInfo specified.
      • This is a potentially breaking change, see https://github.com/moby/moby/pull/42307
    Source code(tar.gz)
    Source code(zip)
  • v0.4.19(Apr 22, 2021)

    • Temporarily reverted implementation of GetFileStandardInfo which returns information from the GetFileInformationByHandleEx syscall with FileStandardInfo specified to address https://github.com/moby/moby/pull/42307
    Source code(tar.gz)
    Source code(zip)
  • v0.4.18(Apr 20, 2021)

  • v0.4.17(Apr 13, 2021)

    • Added build constraints to Windows specific files.
    • Fixed error handling for GetFileSystemType.
    • pkg/etw now supports setting a provider group ID.
    • Switched from os/exec to golang.org/x/sys/execabs for launching processes. This removes a generally unintended affect where the current directory would be searched for the binary to be launched.
    • Added GetFileStandardInfo which returns information from the GetFileInformationByHandleEx syscall with FileStandardInfo specified.
    Source code(tar.gz)
    Source code(zip)
  • v0.4.16(Dec 9, 2020)

    • Added new bindings, functions, and exported all of the flags in the vhd package. This is to facilitate finer grained control in managing vhds.
    Source code(tar.gz)
    Source code(zip)
  • v0.4.15(Nov 4, 2020)

    • Added new pkg/process with some process enumeration and querying functionality.
    • Added winio.DialPipeAccess which allows specifying the access that the pipe is opened with.
    • Removed winio's archive/tar package and replaced it with direct usage of archive/tar from Go's standard library.
    Source code(tar.gz)
    Source code(zip)
  • v0.4.14(Aug 6, 2019)

  • v0.4.13(Jul 19, 2019)

    This release contains various fixes and additions to the go-winio package.

    It adds:

    • go module support.
    • ListenHvsock support for opening and managing Hyper-V Sockets from a Windows host.
    • DialPipeContext that allows the caller to pass in the appropriate cancellation logic via the ctx parameter.
    • The pkg/etw package that enables the use of go binaries write to Windows etw logging. For a sample see the pkg/etw/sample package.
    • The pkg/etwlogrus package which defines a logrus hook that can be used to export logrus messages to etw.
    • The pkg/fs package which unifies the GetFileSystemType function across Docker and containerd to avoid duplication.
    • The pkg/guid package which add easy support for the golang.org/x/sys/windows.GUID to make NewV4 and NewV5 variant GUID structures that support TextMarshaler for easy encoding and decoding from JSON .
    • The pkg/security package that enables support for GrantVmGroupAccess on a file or directory path. This grants the appropriate VM Group SID for the path or all files in the directory path so that these files may be used at container activation or modification.
    • The OpenVirtualDisk function which enables go binaries to get a syscall.Handle to a vhd that can be used for other various vhd operations.
    • The Fd() function on the named pipe that returns the raw handle to be used with other Windows syscalls.

    It fixes:

    • An a race condition where ListenPipe can fail due to concurrent DialPipe calls.

    A big Thanks! (alphabetical)

    @azdagron @CodeLingoBot @kevpar @jhowardmsft @jstarks @jterry75 @sime1 @ehotinger

    Source code(tar.gz)
    Source code(zip)
  • v0.4.12(Feb 21, 2019)

  • v0.4.11(Aug 23, 2018)

  • v0.4.10(Aug 20, 2018)

  • v0.4.9(Jul 19, 2018)

  • v0.4.8(Jun 28, 2018)

    This is a bug fix release that adds support for pipe message read mode, allowing the pipe functionality to work inside Hyper-V-isolated Windows containers.

    Source code(tar.gz)
    Source code(zip)
  • v0.4.2(May 24, 2017)

  • v0.4.1(May 19, 2017)

  • v0.4.0(May 8, 2017)

    This release adds support for go 1.8.x.

    Due to new variable liveness rules, and requirements to use runtime.KeepAlive, go 1.6.x is no longer supported (and is impossible to support along with go 1.8.x in a single codebase for this project)

    Source code(tar.gz)
    Source code(zip)
Owner
Microsoft
Open source projects and samples from Microsoft
Microsoft
dht is used by anacrolix/torrent, and is intended for use as a library in other projects both torrent related and otherwise

dht Installation Install the library package with go get github.com/anacrolix/dht, or the provided cmds with go get github.com/anacrolix/dht/cmd/....

Matt Joiner 246 Sep 26, 2022
An online book focusing on Go syntax/semantics and runtime related things

Go 101 is a book focusing on Go syntax/semantics and all kinds of runtime related things. It tries to help gophers gain a deep and thorough understanding of Go. This book also collects many details of Go and in Go programming. The book is expected to be helpful for both beginner and experienced Go programmers.

Go101 4.7k Sep 21, 2022
A small utility that aims to automate and simplify some tasks related to software release cycles.

Stork is a small utility that aims to automate and simplify some tasks related to software release cycles such as reading the current version from a f

Simone Margaritelli 43 Aug 30, 2022
A directory of hardware related libs, tools, and tutorials for Go

Go + hardware This repo is a directory of tools, packages and tutorials to let you introduce Go in your hardware projects. Why Go? Go can target platf

Jaana Dogan 1.3k Sep 24, 2022
Nebula Operator manages NebulaGraph clusters on Kubernetes and automates tasks related to operating a NebulaGraph cluster

Nebula Operator manages NebulaGraph clusters on Kubernetes and automates tasks related to operating a NebulaGraph cluster. It evolved from NebulaGraph Cloud Service, makes NebulaGraph a truly cloud-native database.

vesoft inc. 52 Aug 17, 2022
Hammond is a self hosted vehicle management system to track fuel and other expenses related to all of your vehicles.

Hammond is a self hosted vehicle management system to track fuel and other expenses related to all of your vehicles. It supports multiple users sharing multiple vehicles. It is the logical successor to Clarkson which has not been updated for quite some time now.

Akhil Gupta 315 Sep 26, 2022
gophertunnel is composed of several packages that may be of use for creating Minecraft related tools

gophertunnel is composed of several packages that may be of use for creating Minecraft related tools. A brief overview of all packages may be found here.

Sandertv 289 Sep 22, 2022
Get related domains / subdomains by looking at Google Analytics IDs

AnalyticsRelationships This script try to get related domains / subdomains by looking at Google Analytics IDs from a URL. First search for ID of Googl

Josué Encinar 129 Sep 11, 2022
🎉 An awesome version control tool for protoc and its related plugins.

❤️ PowerProto is actively maintained! Any questions in use can be directly raised issue, I will respond to you as fast as possible. If you think the p

storyicon 157 Sep 16, 2022
OpenAPI specification and related artifacts for HashiCorp Nomad

Overview This repository contains the HashiCorp Nomad OpenAPI specification and related artifacts. The OpenAPI specification defines a machine-readabl

HashiCorp 42 Sep 21, 2022
A collection of authentication Go packages related to OIDC, JWKs and Distributed Claims.

cap (collection of authentication packages) provides a collection of related packages which enable support for OIDC, JWT Verification and Distributed Claims.

HashiCorp 336 Sep 22, 2022
🌍 Package tcplisten provides a customizable TCP net.Listener with various performance-related options

Package tcplisten provides customizable TCP net.Listener with various performance-related options: SO_REUSEPORT. This option allows linear scaling ser

Spiral Scout 1 Sep 7, 2022
A program to create assembly 8086 strings to print without using any printing/strings related function but only mov-xchg-int and loops

Assembly String builder tool A program to create assembly 8086 strings to print without using any printing/strings related function but only mov-xchg-

Reg 2 Feb 1, 2022
The primary place where Optimism works on stuff related to Optimistic Ethereum

The Optimism Monorepo TL;DR This is the primary place where Optimism works on stuff related to Optimistic Ethereum. Documentation Extensive documentat

Onther Inc. Technology 1 Aug 16, 2022
A scanner for running security-related configuration checks such as CIS benchmarks

Localtoast Localtoast is a scanner for running security-related configuration checks such as CIS benchmarks in an easily configurable manner. The scan

Google 30 Sep 13, 2022
Grpc bridge to various mediabank related systems

Mediabank bridge This internal tool enables authenticated gRPC based endpoint for securely communicating with systems like: Telestream Vantage Workflo

BCC Code 0 Jan 7, 2022
cluster-api-state-metrics (CASM) is a service that listens to the Kubernetes API server and generates metrics about the state of custom resource objects related of Kubernetes Cluster API.

Overview cluster-api-state-metrics (CASM) is a service that listens to the Kubernetes API server and generates metrics about the state of custom resou

Daimler Group 60 Aug 16, 2022
Source code related to an on-site demonstration (for Ardan Labs) of packaging Go applications with Nix

Ardan Labs Nix Demo High-Level Overview We bumbled the scheduling of an earlier presentation about git worktree that a few co-workers attended. In eff

John Rinehart 4 Apr 30, 2022
A go program that relies on back-end ffmpeg to process video-related content

Video Compress A go program that relies on back-end ffmpeg to process video-related content Installation v-go You can download the corresponding v-go

JokerHeyra 0 Dec 22, 2021
Labs, solutions and related materials from the MIT 6.824 Distributed Systems course.

MIT 6.824 Distributed Systems Labs, solutions and related materials from the MIT 6.824 Distributed Systems course. Overview From the official website:

Arindam Das 44 Sep 16, 2022