Go language interface to Swift / Openstack Object Storage / Rackspace cloud files (golang)

Overview

Swift

This package provides an easy to use library for interfacing with Swift / Openstack Object Storage / Rackspace cloud files from the Go Language

See here for package docs

http://godoc.org/github.com/ncw/swift/v2

Build Status GoDoc

Install

Use go to install the library

go get github.com/ncw/swift/v2

Usage

See here for full package docs

Here is a short example from the docs

import "github.com/ncw/swift/v2"

// Create a connection
c := swift.Connection{
UserName: "user",
ApiKey:   "key",
AuthUrl:  "auth_url",
Domain:   "domain", // Name of the domain (v3 auth only)
Tenant:   "tenant", // Name of the tenant (v2 auth only)
}
// Authenticate
err := c.Authenticate()
if err != nil {
panic(err)
}
// List all the containers
containers, err := c.ContainerNames(nil)
fmt.Println(containers)
// etc...

Migrating from v1

The library has current major version v2. If you want to migrate from the first version of library github.com/ncw/swift you have to explicitly add the /v2 suffix to the imports.

Most of the exported functions were added a new context.Context parameter in the v2, which you will have to provide when migrating.

Additions

The rs sub project contains a wrapper for the Rackspace specific CDN Management interface.

Testing

To run the tests you can either use an embedded fake Swift server either use a real Openstack Swift server or a Rackspace Cloud files account.

When using a real Swift server, you need to set these environment variables before running the tests

export SWIFT_API_USER='user'
export SWIFT_API_KEY='key'
export SWIFT_AUTH_URL='https://url.of.auth.server/v1.0'

And optionally these if using v2 authentication

export SWIFT_TENANT='TenantName'
export SWIFT_TENANT_ID='TenantId'

And optionally these if using v3 authentication

export SWIFT_TENANT='TenantName'
export SWIFT_TENANT_ID='TenantId'
export SWIFT_API_DOMAIN_ID='domain id'
export SWIFT_API_DOMAIN='domain name'

And optionally these if using v3 trust

export SWIFT_TRUST_ID='TrustId'

And optionally this if you want to skip server certificate validation

export SWIFT_AUTH_INSECURE=1

And optionally this to configure the connect channel timeout, in seconds

export SWIFT_CONNECTION_CHANNEL_TIMEOUT=60

And optionally this to configure the data channel timeout, in seconds

export SWIFT_DATA_CHANNEL_TIMEOUT=60

Then run the tests with go test

License

This is free software under the terms of MIT license (check COPYING file included in this package).

Contact and support

The project website is at:

There you can file bug reports, ask for help or contribute patches.

Authors

Contributors

Issues
  • Support for Bulk Delete

    Support for Bulk Delete

    • Connection.BulkDelete()

      deletes named objects at once using bulk delete API.

    • BulkDeleteResult

      stores results of BulkDelete operation.

    The referenced specification is Rackspace Cloud Files API: http://docs.rackspace.com/files/api/v1/cf-devguide/content/Bulk_Delete-d1e2338.html

    Tested on Rackspace.

    See #8 for discussions.

    opened by ymmt2005 15
  • Go tests fail on v2.0 endpoint

    Go tests fail on v2.0 endpoint

    As requested I opened an issue because the Go tests fail on my OpenStack Mitaka backend with v2.0 authentication.

    https://github.com/restic/restic/issues/1039#issuecomment-309209227

    ~/src/github.com/ncw/swift$ source openrc.sh
    ~/src/github.com/ncw/swift$ export SWIFT_TENANT_ID=$OS_TENANT_ID
    ~/src/github.com/ncw/swift$ export SWIFT_API_KEY=$OS_PASSWORD
    ~/src/github.com/ncw/swift$ export SWIFT_AUTH_URL=$OS_AUTH_URL
    ~/src/github.com/ncw/swift$ export SWIFT_TENANT=$OS_TENANT
    ~/src/github.com/ncw/swift$ export SWIFT_API_USER=$OS_USERNAME
    ~/src/github.com/ncw/swift$ go version
    go version go1.8.3 linux/amd64
    ~/src/github.com/ncw/swift$ go test
    Waiting for server to start . Started
    --- FAIL: TestAuthenticateRace (1.10s)
    	swift_test.go:456: Auth failed Response didn't have storage url and auth token
    	swift_test.go:456: Auth failed Response didn't have storage url and auth token
    	swift_test.go:456: Auth failed Response didn't have storage url and auth token
    	swift_test.go:456: Auth failed Response didn't have storage url and auth token
    	swift_test.go:456: Auth failed Response didn't have storage url and auth token
    	swift_test.go:456: Auth failed Response didn't have storage url and auth token
    	swift_test.go:456: Auth failed Response didn't have storage url and auth token
    	swift_test.go:456: Auth failed Response didn't have storage url and auth token
    	swift_test.go:456: Auth failed Response didn't have storage url and auth token
    	swift_test.go:456: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestSerializeConnectionJson (0.11s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestSerializeConnectionXml (0.10s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestOnReAuth (0.10s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestAccount (0.11s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestAccountUpdate (0.10s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestContainerCreate (0.10s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestContainer (0.10s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestContainersAll (0.10s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestContainersAllWithLimit (0.09s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestContainerUpdate (0.09s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestContainerNames (0.09s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestContainerNamesAll (0.10s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestContainerNamesAllWithLimit (0.10s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestObjectPutString (0.11s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestObjectPut (0.11s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestObjectEmpty (0.10s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestObjectPutBytes (0.11s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestObjectPutMimeType (0.11s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestObjectCreate (0.13s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestObjectGetString (0.09s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestObjectGetBytes (0.10s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestObjectOpen (0.11s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    ^Csignal: interrupt
    FAIL	github.com/ncw/swift	4.409s
    
    opened by namsral 11
  • Rackspace v2 Auth giving 400 Bad Request on Authenticate

    Rackspace v2 Auth giving 400 Bad Request on Authenticate

    Having both authentication methods in one request causes a bad request response from Rackspace. After debugging with support, the tech instructed that the RAX-KSKEY method is the correct one and only one to use.

    opened by kayleg 11
  • Retry when getting an EOF on HEAD/GET requests because of a Golang bug

    Retry when getting an EOF on HEAD/GET requests because of a Golang bug

    When running a 30 minutes testsuite (https://github.com/lebauce/distribution), I faced a weird issue : I sometimes got a url.Error : "Head https://my.swift.server/v1/AUTH_abcdb3abcda34a1c8d618164d99eb53e/test/myobject : EOF". I'm using the 1.4.2 version of Go.

    The issue seems to be in the net/http library and is described in both tickets : https://github.com/golang/go/issues/4677 https://github.com/golang/go/issues/8946

    A fix on net/http is under review : https://golang.org/cl/3210

    A fix for this problem is simply to send the request again when the request was a HEAD or a GET

    opened by lebauce 10
  • Time difference between server and client

    Time difference between server and client

    Hi, I've noticed that when I have a wrong timezone on the client (e.g. when booted with live CD in a VM) I get this error

    "Response didn't have storage url and auth token"

    from https://github.com/ncw/swift/blob/v1/swift.go#L535, but that's not what actually happens there, I checked and compared the responses and if I correct the time everything works just fine, so at the very least the error message isn't accurate if that's intended behavior. Is it though? Is there a requirement that time is synced between client and server? Seems strange. Seems to me that the token or something gets expired right after it gets assigned an expiration date in the same function call.

    opened by xaduha 9
  • add workarounds for servers that don't paginate correctly

    add workarounds for servers that don't paginate correctly

    Some Swift API implementations (I've observed this with Ceph RADOSGW) can return fewer results than specified by the "limit" parameter, even when we have not reached the end of the listing.

    It's unclear to me from reading the API docs whether this is a violation of the API specification, but since it happens in the wild, it's best to be able to handle it.

    One way of doing this is to simply keep fetching pages until we receive an empty page. Another is to assume that pages within a certain percentage of limit are not the last page. Given the tradeoffs involved, let's support both.

    opened by jetpackdanger 8
  • Please tag a release

    Please tag a release

    For restic, we'll switch to dep shortly, and it would be very helpful if you could tag a release. Thanks!

    https://dave.cheney.net/2016/06/24/gophers-please-tag-your-releases

    enhancement 
    opened by fd0 8
  • Doing a reauth doesn't eat the first bytes in the body

    Doing a reauth doesn't eat the first bytes in the body

    This should hopefully fix #71. It uses Expect: 100-continue. The trick to get it working properly was to set TransferEncoding: []string{"chunked"} on the request object.

    opened by mr-andreas 8
  • CDN Management

    CDN Management

    I've hacked your source to do CDN enabling/disabling of containers.

    Two relatively big changes were:

    • Changed storageOpts to the more generic requestOpts name.
    • Added a "manage" method similar to "storage", but they now wrap a new "call" method so the correct storage/management URL gets called.

    There are also tests in place which pass.

    Please get back to me if there is anything I need to change.

    opened by boj 8
  • Add inspectable context parameter to API calls

    Add inspectable context parameter to API calls

    I am currently working on a product in which multiple goroutines are sharing the same swift client, and we need to be able to correlate the requests the client is making with the specific goroutine which initiated the action. To do this we pass down the request ID and make it accessible to a custom http.Transport for logging. We do this successfully with other storage clients. However, with this Swift client we've had to do some hacking. We append request ID to the Connection.UserAgent field, then extracting in our overriden http.Transport.RoundTrip method before dispatching the request.

    This seemed to us was a generic enough use-case to open an issue. We suggest Google's approach of adding a context parameter to each API function call, https://godoc.org/cloud.google.com/go/storage I understand this would mean updating the API functions to accept a context AND attach the context to the requests the client makes to allow a custom Transport to inspect the request's context. Not only would it support our use case, but make it far easier for your users to process custom data in their net/http interface implementations.

    enhancement 
    opened by zebwaredano 7
  • Allow using adminURL as storage endpoint URL

    Allow using adminURL as storage endpoint URL

    According to http://developer.openstack.org/api-ref-identity-v2.html#authenticate-v2.0 the endpoints response parameter is defined as:

    One or more endpoints objects. Each object shows the adminURL, region, internalURL, id, and publicURL for the endpoint.

    Reading the adminURL field and the ability to use it as an endpoint, was missing from this library.

    This PR changes the signature of Authenticator.StorageUrl to receive an EndpointType instead of a boolean. Also, I've added an EndpointType field to the Connection struct to allow users to specify a custom endpoint type.

    I tried to keep everything compatible so the Internal boolean field remains in the Connection struct, if the EndpointType is not set and the Internal flag is set, the chosen endpoint type will be the internal url otherwise the public url is used.

    opened by cezarsa 7
  • Implement federated (v3oidcaccesstoken) auth

    Implement federated (v3oidcaccesstoken) auth

    v3oidcaccesstoken requires two authentication requests:

    1. Retrieving an unscoped token using an oidc access token
    2. Retrieving a scoped token using the unscoped token (already implemented in v3auth)

    Change summary:

    • Extracted method for auth request handling
    • Added missing fields to Connection, e.g. IdentityProvider
      • Added mappings to the relevant openstack environment variables
    • Added interface TwoStageAuthenticator
    • Implemented PrelimRequest and PrelimResponse for v3auth

    Sadly I saw no reasonable way to test these changes, but I'm completely open to suggestions. This includes the way I extended the authentication handling, which may or may not be optimal.

    These changes are a follow up to this forum thread: https://forum.rclone.org/t/swift-env-auth-true-env-os-access-token-http-error-404-404-not-found/22520

    opened by lburgey 3
  • No support for rate-limit errors with retry-after headers.

    No support for rate-limit errors with retry-after headers.

    When receiving errors for creating/updating objects/containers the headers are thrown away, this makes it impossible to respect retry-after headers which can be present in e.g. http 503 Service Unavailable or 429 Too many requests. (Unless Ḯm missing something obvious?) A possible non breaking solution would be to add an optional field "Headers" to swift.Error and append headers to this error. Another solution which would be breaking could be to simply return both error and headers from ObjectUpdate etc.

    opened by axiserik 0
  • Document about authentication flow may be misleading?

    Document about authentication flow may be misleading?

    The Authentication method says:

    If you don't call it before calling one of the connection methods then it will be called for you on the first access.

    And the Call method says

    This will Authenticate if necessary, and re-authenticate if it receives a 401 error which means the token has expired

    But if user create the connection object and use Call method directly, program will receive SIGSEGV, because the conn.authLock member is not initialized yet. Other method like ObjectPut ObjectDelete also suffer from this problem.

    Although the sample code do call Authenticate after connection object construction. I thought the document may be a little misleading. :D

    opened by wdhongtw 0
  • slo: no headers for segments

    slo: no headers for segments

    When creating a slo, no headers are written to the segment objects: headers, err := file.conn.ObjectPut(file.segmentContainer, segmentName, segmentReader, true, "", file.contentType, nil) https://github.com/ncw/swift/blob/master/largeobjects.go#L403

    In my case I want to pass a X-Delete-At parameter also to the segments that are created. Otherwise they will be orphaned once the actual file is expired and deleted.

    Can we pass in the headers from the file itself? headers, err := file.conn.ObjectPut(file.segmentContainer, segmentName, segmentReader, true, "", file.contentType, file.headers)

    thanks!

    opened by stefanhipfel 0
  • ObjectCreate() 404 return is wrongly interpreted as ObjectNotFound instead of ContainerNotFound

    ObjectCreate() 404 return is wrongly interpreted as ObjectNotFound instead of ContainerNotFound

    According to swift doc PUT request on object (object creation) returns 404 if container is not found

    If the container for the object does not already exist, the operation returns the 404 Not Found response code.

    ObjectCreate() function is currently mapped against objectErrorMap here so ncw/swift interprets 404 on object PUT as ObjectNotFound instead of ContainerNotFound which is confusing.

    opened by sblaisot 0
Releases(v2.0.1)
  • v2.0.1(Oct 7, 2021)

    What's Changed

    • Trim double quotes from received ETag when checking hashes by @timss in https://github.com/ncw/swift/pull/175
    • Fix races related to the way how authLock is implemented and used by @yuri-per in https://github.com/ncw/swift/pull/170
    • Path unescape the data return in the X-Object-Manifest header by @houtmanj in https://github.com/ncw/swift/pull/168
    • add workarounds for servers that don't paginate correctly by @jetpackdanger in https://github.com/ncw/swift/pull/167
    • Update build process to use GitHub actions and a docker based swift All in One
    • Add golangci-lint to the build process

    New Contributors

    • @timss made their first contribution in https://github.com/ncw/swift/pull/175
    • @houtmanj made their first contribution in https://github.com/ncw/swift/pull/168
    • @jetpackdanger made their first contribution in https://github.com/ncw/swift/pull/167

    Full Changelog: https://github.com/ncw/swift/compare/v2.0.0...v2.0.1

    Source code(tar.gz)
    Source code(zip)
Owner
Nick Craig-Wood
Open source. Rclone. Family. Sleep. Pick any two. Having fun with Go.
Nick Craig-Wood
A simple image hosting script in Golang (Smart Storage with Telegram Cloud Storage)

image-upload-tg It's a simple image hosting script in Golang. It's have http server and image can be uploaded from here, also images store temporary i

Alperen Çetin 0 Jan 19, 2022
Contentrouter - Protect static content via Firebase Hosting with Cloud Run and Google Cloud Storage

contentrouter A Cloud Run service to gate static content stored in Google Cloud

G. Hussain Chinoy 0 Jan 2, 2022
Container Storage Interface driver for Synology NAS

Synology CSI Driver for Kubernetes The official Container Storage Interface driver for Synology NAS. Container Images & Kubernetes Compatibility Drive

Synology Open Source 162 Aug 2, 2022
cloud-native local storage management system

Open-Local是由多个组件构成的本地磁盘管理系统,目标是解决当前 Kubernetes 本地存储能力缺失问题。通过Open-Local,使用本地存储会像集中式存储一样简单。

null 227 Aug 9, 2022
GoDrive: A cloud storage system similar to Dropbox or Google Drive, with resilient

Cloud Storage Service Author: Marisa Tania, Ryan Tjakrakartadinata Professor: Matthew Malensek See project spec here: https://www.cs.usfca.edu/~mmalen

Ryan G Tjakrakartadinata 2 Dec 7, 2021
The extensible SQL interface to your favorite cloud APIs.

The extensible SQL interface to your favorite cloud APIs.

Turbot 2k Aug 8, 2022
Cloudpods is a cloud-native open source unified multi/hybrid-cloud platform developed with Golang

Cloudpods is a cloud-native open source unified multi/hybrid-cloud platform developed with Golang, i.e. Cloudpods is a cloud on clouds. Cloudpods is able to manage not only on-premise KVM/baremetals, but also resources from many cloud accounts across many cloud providers. It hides the differences of underlying cloud providers and exposes one set of APIs that allow programatically interacting with these many clouds.

null 1 Jan 11, 2022
Sample apps and code written for Google Cloud in the Go programming language.

Google Cloud Platform Go Samples This repository holds sample code written in Go that demonstrates the Google Cloud Platform. Some samples have accomp

Google Cloud Platform 3.5k Aug 1, 2022
Lightweight Cloud Instance Contextualizer

Flamingo Flamingo is a lightweight contextualization tool that aims to handle initialization of cloud instances. It is meant to be a replacement for c

Tamer Tas 36 Jun 18, 2022
Terraform provider for HashiCorp Cloud Platform.

HashiCorp Cloud Platform (HCP) Terraform Provider Requirements Terraform >= 0.12.x Go >= 1.14 Building The Provider Clone the repository Enter the rep

HashiCorp 27 Jul 4, 2022
The Cloud Posse Terraform Provider for various utilities (E.g. deep merging)

terraform-provider-utils Terraform provider to add additional missing functionality to Terraform This project is part of our comprehensive "SweetOps"

Cloud Posse 62 Jul 27, 2022
Cloud cost estimates for Terraform in your CLI and pull requests 💰📉

Infracost shows cloud cost estimates for Terraform projects. It helps developers, devops and others to quickly see the cost breakdown and compare different options upfront.

Infracost 7.6k Aug 1, 2022
Cloud-native way to provide elastic Jupyter Notebook services on Kubernetes

elastic-jupyter-operator: Elastic Jupyter on Kubernetes Kubernetes 原生的弹性 Jupyter 即服务 介绍 为用户按需提供弹性的 Jupyter Notebook 服务。elastic-jupyter-operator 提供以下特性

TKEStack 121 Jul 27, 2022
Google Cloud Client Libraries for Go.

Google Cloud Client Libraries for Go.

Google APIs 3k Aug 1, 2022
A Cloud Native Buildpack for Go

The Go Paketo Buildpack provides a set of collaborating buildpacks that enable the building of a Go-based application.

Paketo Buildpacks 53 Jun 6, 2022
Fleex allows you to create multiple VPS on cloud providers and use them to distribute your workload.

Fleex allows you to create multiple VPS on cloud providers and use them to distribute your workload. Run tools like masscan, puredns, ffuf, httpx or anything you need and get results quickly!

null 156 Jul 18, 2022
☁️🏃 Get up and running with Go on Google Cloud.

Get up and running with Go and gRPC on Google Cloud Platform, with this lightweight, opinionated, batteries-included service SDK.

Einride 19 Aug 1, 2022
Elkeid is a Cloud-Native Host-Based Intrusion Detection solution project to provide next-generation Threat Detection and Behavior Audition with modern architecture.

Elkeid is a Cloud-Native Host-Based Intrusion Detection solution project to provide next-generation Threat Detection and Behavior Audition with modern architecture.

Bytedance Inc. 1.3k Aug 4, 2022
Use Google Cloud KMS as an io.Reader and rand.Source.

Google Cloud KMS Go io.Reader and rand.Source This package provides a struct that implements Go's io.Reader and math/rand.Source interfaces, using Goo

Seth Vargo 4 Nov 10, 2021