⚡️ SharePoint authentication, HTTP client & fluent API wrapper for Go (Golang)

Andrew Koltyakov

Last update: Jan 2, 2023

Overview

Gosip - SharePoint authentication, HTTP client & fluent API wrapper for Go (Golang)

Main features

Unattended authentication using different strategies.
Fluent API syntax for SharePoint object model.
Simplified API consumption (REST, CSOM, SOAP).
SharePoint-aware embedded features (retries, header presets, error handling).

Supported SharePoint versions

SharePoint Online (SPO)
On-Premises (2019/2016/2013)

Supported auth strategies

SharePoint Online:
- Azure Certificate (App Only) 🔗
- Azure Username/Password 🔗
- SAML based with user credentials
- Add-In only permissions
- ADFS user credentials (automatically detects in SAML strategy)
- On-Demand auth 🔗
- Azure AD Device flow 🔗
SharePoint On-Premises 2019/2016/2013:
- User credentials (NTLM)
- ADFS user credentials (ADFS, WAP -> Basic/NTLM, WAP -> ADFS)
- Behind a reverse proxy (Forefront TMG, WAP -> Basic/NTLM, WAP -> ADFS)
- Form-based authentication (FBA)
- On-Demand auth 🔗

Installation

go get github.com/koltyakov/gosip

Usage insights

1. Understand SharePoint environment type and authentication strategy.

Let's assume it's SharePoint Online and Add-In Only permissions. Then strategy "github.com/koltyakov/gosip/auth/addin" subpackage should be used.

package main

import (
	"github.com/koltyakov/gosip"
	"github.com/koltyakov/gosip/api"
	strategy "github.com/koltyakov/gosip/auth/addin"
)

2. Initiate an authentication object.

auth := &strategy.AuthCnfg{
	SiteURL:      os.Getenv("SPAUTH_SITEURL"),
	ClientID:     os.Getenv("SPAUTH_CLIENTID"),
	ClientSecret: os.Getenv("SPAUTH_CLIENTSECRET"),
}

AuthCnfg from different strategies contains different options relevant for a specified auth type.

The authentication options can be provided explicitly or can be read from a configuration file.

configPath := "./config/private.json"
auth := &strategy.AuthCnfg{}

err := auth.ReadConfig(configPath)
if err != nil {
	fmt.Printf("Unable to get config: %v\n", err)
	return
}

3. Bind auth client with Fluent API.

client := &gosip.SPClient{AuthCnfg: auth}

sp := api.NewSP(client)

res, err := sp.Web().Select("Title").Get()
if err != nil {
	fmt.Println(err)
}

fmt.Printf("%s\n", res.Data().Title)

Usage samples

Fluent API client

Fluent API gives a simple way of constructing API endpoint calls with IntelliSense and chainable syntax.

package main

import (
	"encoding/json"
	"fmt"
	"log"

	"github.com/koltyakov/gosip"
	"github.com/koltyakov/gosip/api"
	strategy "github.com/koltyakov/gosip/auth/addin"
)

func main() {
	// Getting auth params and client
	client, err := getAuthClient()
	if err != nil {
		log.Fatalln(err)
	}

	// Binding SharePoint API
	sp := api.NewSP(client)

	// Custom headers
	headers := map[string]string{
		"Accept": "application/json;odata=minimalmetadata",
		"Accept-Language": "de-DE,de;q=0.9",
	}
	config := &api.RequestConfig{Headers: headers}

	// Chainable request sample
	data, err := sp.Conf(config).Web().Lists().Select("Id,Title").Get()
	if err != nil {
		log.Fatalln(err)
	}

	// Response object unmarshalling (struct depends on OData mode and API method)
	res := &struct {
		Value []struct {
			ID    string `json:"Id"`
			Title string `json:"Title"`
		} `json:"value"`
	}{}

	if err := json.Unmarshal(data, &res); err != nil {
		log.Fatalf("unable to parse the response: %v", err)
	}

	for _, list := range res.Value {
		fmt.Printf("%+v\n", list)
	}

}

func getAuthClient() (*gosip.SPClient, error) {
	configPath := "./config/private.spo-addin.json"
	auth := &strategy.AuthCnfg{}
	if err := auth.ReadConfig(configPath); err != nil {
		return nil, fmt.Errorf("unable to get config: %v", err)
	}
	return &gosip.SPClient{AuthCnfg: auth}, nil
}

Generic HTTP client helper

Provides generic GET/POST helpers for REST operations, reducing the amount of http.NewRequest scaffolded code, can be used for custom or not covered with Fluent API endpoints.

package main

import (
	"fmt"
	"log"

	"github.com/koltyakov/gosip"
	"github.com/koltyakov/gosip/api"
	strategy "github.com/koltyakov/gosip/auth/ntlm"
)

func main() {
	configPath := "./config/private.ntlm.json"
	auth := &strategy.AuthCnfg{}

	if err := auth.ReadConfig(configPath); err != nil {
		log.Fatalf("unable to get config: %v\n", err)
	}

	sp := api.NewHTTPClient(&gosip.SPClient{AuthCnfg: auth})

	endpoint := auth.GetSiteURL() + "/_api/web?$select=Title"

	data, err := sp.Get(endpoint, nil)
	if err != nil {
		log.Fatalf("%v\n", err)
	}

	// sp.Post(endpoint, body, nil) // generic POST
	// sp.Delete(endpoint, nil) // generic DELETE helper crafts "X-Http-Method"="DELETE" header
	// sp.Update(endpoint, nil) // generic UPDATE helper crafts "X-Http-Method"="MERGE" header
	// sp.ProcessQuery(endpoint, body) // CSOM helper (client.svc/ProcessQuery)

	fmt.Printf("response: %s\n", data)
}

Low-level HTTP client usage

Low-lever SharePoint-aware HTTP client from github.com/koltyakov/gosip package for custom or not covered with a Fluent API client endpoints with granular control for an HTTP request, response, and http.Client parameters. The client is used internally but rarely required in consumer code.

client := &gosip.SPClient{AuthCnfg: auth}

var req *http.Request
// Initiate API request
// ...

resp, err := client.Execute(req)
if err != nil {
	fmt.Printf("Unable to request api: %v", err)
	return
}

SPClient has Execute method which is a wrapper function injecting SharePoint authentication and ending up calling http.Client's Do method.

Authentication strategies

Auth strategy should be selected corresponding to your SharePoint environment and its configuration.

Import path strategy "github.com/koltyakov/gosip/auth/{strategy}". Where /{strategy} stands for a strategy auth package.

`/{strategy}`	SPO	On-Prem	Credentials sample(s)
AAD `/azurecert`	✅	❌	details
AAD `/azurecreds`	✅	❌	details
AAD `/device`	✅	❌	details
`/saml`	✅	❌	sample
`/addin`	✅	❌	sample
`/ntlm`	❌	✅	sample
`/adfs`	✅	✅	spo, on-prem, on-prem (wap)
`/fba`	❌	✅	sample
`/tmg`	❌	✅	sample

JSON and struct representations are different in terms of language notations. So credentials parameters names in private.json files and declared as structs initiators vary.

SAML Auth (SharePoint Online user credentials authentication)

This authentication option uses Microsoft Online Security Token Service https://login.microsoftonline.com/extSTS.srf and SAML tokens in order to obtain an authentication cookie.

// AuthCnfg - SAML auth config structure
type AuthCnfg struct {
	// SPSite or SPWeb URL, which is the context target for the API calls
	SiteURL string `json:"siteUrl"`
	// Username for SharePoint Online, for example `[user]@[company].onmicrosoft.com`
	Username string `json:"username"`
	// User or App password
	Password string `json:"password"`
}

AddIn Only Auth

This type of authentication uses AddIn Only policy and OAuth bearer tokens for authenticating HTTP requests.

// AuthCnfg - AddIn Only auth config structure
type AuthCnfg struct {
	// SPSite or SPWeb URL, which is the context target for the API calls
	SiteURL string `json:"siteUrl"`
	// Client ID obtained when registering the AddIn
	ClientID string `json:"clientId"`
	// Client Secret obtained when registering the AddIn
	ClientSecret string `json:"clientSecret"`
	// Your SharePoint Online tenant ID (optional)
	Realm string `json:"realm"`
}

Realm can be left empty or filled in, which will add small performance improvement. The easiest way to find the tenant is to open SharePoint Online site collection, click Site Settings -> Site App Permissions. Taking any random app, the tenant ID (realm) is the GUID part after the @.

See more details of AddIn Configuration and Permissions.

NTLM Auth (NTLM handshake)

This type of authentication uses an HTTP NTLM handshake to obtain an authentication header.

// AuthCnfg - NTML auth config structure
type AuthCnfg struct {
	// SPSite or SPWeb URL, which is the context target for the API calls
	SiteURL  string `json:"siteUrl"`
	Domain   string `json:"domain"`   // AD domain name
	Username string `json:"username"` // AD user name
	Password string `json:"password"` // AD user password
}

Gosip uses github.com/Azure/go-ntlmssp NTLM negotiator, however, a custom one also can be provided in case of demand.

ADFS Auth (user credentials authentication)

// AuthCnfg - ADFS auth config structure
type AuthCnfg struct {
	// SPSite or SPWeb URL, which is the context target for the API calls
	SiteURL      string `json:"siteUrl"`
	Username     string `json:"username"`
	Password     string `json:"password"`
	// Following are not required for SPO
	Domain       string `json:"domain"`
	RelyingParty string `json:"relyingParty"`
	AdfsURL      string `json:"adfsUrl"`
	AdfsCookie   string `json:"adfsCookie"`
}

See more details ADFS user credentials authentication.

Gosip's ADFS also supports a scenario of ADFS or NTML behind WAP (Web Application Proxy) which adds additional auth flow and EdgeAccessCookie involved into play.

FBA/TMG Auth (Form-based authentication)

FBA - Form-based authentication for SharePoint On-Premises.

TMG - Microsoft Forefront Threat Management Gateway, currently is legacy but was a popular way of exposing SharePoint into the external world back in the days.

// AuthCnfg - FBA/TMG auth config structure
type AuthCnfg struct {
	// SPSite or SPWeb URL, which is the context target for the API calls
	SiteURL string `json:"siteUrl"`
	// Username for SharePoint On-Prem, format depends in FBA/TMG settings,
	// can include domain or doesn't
	Username string `json:"username"`
	// User password
	Password string `json:"password"`
}

Secrets encoding

When storing credential in local private.json files, which can be handy in local development scenarios, we strongly recommend to encode secrets such as password or clientSecret using cpass. Class converts a secret to an encrypted representation, which can only be decrypted on the same machine where it was generated. That reduces accidental leaks, e.g. together with git commits.

Reference

Many auth flows have been "copied" from node-sp-auth library (used as a blueprint), which we intensively use in Node.js ecosystem for years.

Fluent API and wrapper syntax are inspired by PnPjs, which is also the first-class citizen on almost all our Node.js and front-end projects with SharePoint involved.

📚 Documentation

📦 Samples

License

Comments

Issue in creating list item

We are creating list item. During creation we are passing the same object received during Getting the list item.

But facing the below mentioned issue: unable to request api: 400 Bad Request :: {"error":{"code":"-1, Microsoft.SharePoint.Client.InvalidClientQueryException","message":{"lang":"en-US","value":"Did not find the required 'results' property on the object wrapping a feed."}}}

Can you provide some insight/help regarding this?
question

opened by anurag2994 12

Error when using with multiple sharepoint sites

Describe the bug Trying to use addin auth strategy across multiple sharepoint online sites. Specifically using chunked file upload. The file uploads fine to the first site, but when uploading to the second site, I receive the following error:

unable to request api: invalid character '<' looking for beginning of value

Versions Sharepoint Online github.com/koltyakov/gosip v0.0.0-20200628141644-21f34db9ce21

To Reproduce

package main

import (
	"log"
	"os"

	"github.com/koltyakov/gosip"
	"github.com/koltyakov/gosip/api"
	strategy "github.com/koltyakov/gosip/auth/addin"
)

type site struct {
	url             string
	clientID        string
	secret          string
	uploadDirectory string
}

func main() {
	demoSite := site{
		url:             "https://example.sharepoint.com/sites/MyExampleSite",
		clientID:        "myclientid",
		secret:          "mysecret",
		uploadDirectory: "Shared Documents",
	}

	secondSite := site{
		url:             "https://example.sharepoint.com/sites/MyExampleSite2",
		clientID:        "myotherclientid",
		secret:          "myothersecret",
		uploadDirectory: "Shared Documents",
	}

	err := uploadFile(demoSite)
	if err != nil {
		log.Fatal(err)
	}

	err = uploadFile(secondSite)
	if err != nil {
		log.Fatal(err)
	}
}

func uploadFile(s site) error {
	authCnfg := strategy.AuthCnfg{
		SiteURL:      s.url,
		ClientID:     s.clientID,
		ClientSecret: s.secret,
	}
	client := gosip.SPClient{AuthCnfg: &authCnfg}
	apiClient := api.NewSP(&client)
	folder := apiClient.Web().GetFolder(s.uploadDirectory)

	file, err := os.Open("testfile.pdf")
	if err != nil {
		log.Fatalf("unable to read a file: %v\n", err)
	}
	defer file.Close()

	_, err = folder.Files().AddChunked("MyTestFile.pdf", file, nil)
	if err != nil {
		return (err)
	}

	return nil
}

Expected behavior Expect for file to upload to both sites without error.

bug

opened by homebrew79 7

How to get permission for lists and items

I am trying to get the permission for each list and item. i tried getting through Roles() method for list and item but unable to get. Please provide the reference if any.

Many thanks in Advance, RV.
enhancement question

opened by vimalraghwani 7
SP Online authentication question

Hi,

We recently moved our SharePoint instance to SharePoint online. To access SharePoint online api, we had to register an app and obtain tenant-id, client-id/client-secret and then use that information to obtain a token. Once we have the bearer token we can then access the SP API. Does this library offer this type of authentication?

opened by brett0701 6
error received empty FormDigestValue

In a long running server process, after idling for about 20 hours, the request resulted in " error received empty FormDigestValue".

How should I troubleshoot this error?

Is it a known issue?
bug

opened by cis-nothize 6
Need to increase the TLSHandshakeTimeout
Is your feature request related to a problem? Please describe.

I am trying to set TLSHandshakeTimeout on the http.Client but it does not seem to be possible.

Describe the solution you'd like

I thought something like this should work:

client := &gosip.SPClient{ Client: http.Client{ Transport: &http.Transport{TLSHandshakeTimeout: 25 * time.Second}, }, AuthCnfg: authCnfg, }

However it seems (empirically) that the timeout is not respected, I guess the http.Client part of SPClient is recreated at some stage. I have tried tracing it through but I have trouble tracing through all the layers.

Describe alternatives you've considered

I have made this work by replacing http.DefaultTransport but obviously this is a global, and thus poor solution.
enhancement
opened by jh-chrysos 5

No documentation on how to download a file

I tried to follow the GIF in your README.md to get a file from a SharePoint REST API. I've been trying for hours. either SPClient gets considered an unknown field, or api.Item can't be used to actually download the file. Is gosip even able to download files??

package main

import (
	"fmt"
	"os"
	"io"
	"log"
	"github.com/koltyakov/gosip"
	"github.com/koltyakov/gosip/api"
	strategy "github.com/koltyakov/gosip/auth/adfs"
)

func main() {
	auth := &strategy.AuthCnfg{
		SiteURL: os.Getenv("https://company.sharepoint.com/teams/teamname"),
		Username: os.Getenv("[email protected]"),
		Password: os.Getenv("password"),
	}
	if auth == nil {
		fmt.Printf("Unable to get config")
		return
	}

	fmt.Printf("credentials loaded.")

	client := &gosip.SPClient{
		AuthCnfg: auth,
	}

	sp := api.SP{SPClient: client}

	item, _ := sp.Web().GetFile("/Main%20Folder/Sub%20Folder/Excel%20File.xlsx").GetItem()
	item.Get()

	//open a file for writing
    file, err := os.Create("test.xlsx")
    if err != nil {
        log.Fatal(err)
    }
    defer file.Close()

    // Use io.Copy to just dump the response body to the file. 
    // This supports huge files
    _, err = io.Copy(file, item.Body)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println("Success!")}

question

opened by acrovitic 5

Login to sharepoint with username/password and SiteURL

Hello,

I don't have ClientID and ClientSecret for Sharepoint, but I have username and password to login it. So, Can I use you lib to login Sharepoint by using username/password and SiteURL? Can you show me code examples for this? I only need to check the folder exist or not, create folder and upload files to this folder. Please help me.

Thank you
question

opened by nbxtruong 5
api library error

Getting the following error when getting the api module:

../../../go/src/github.com/koltyakov/gosip/api/permissions.go:199:13: invalid operation: num << perm (shift count type int64, must be unsigned integer) ../../../go/src/github.com/koltyakov/gosip/api/permissions.go:202:12: invalid operation: num << perm (shift count type int64, must be unsigned integer)

opened by brett0701 4
Issue with creating List view

We are creating list views. During creation we are passing 'ViewType' as 'CALENDER' as mentioned below createMetadata["ViewType"] = "CALENDER". But still view gets created with type HTML.

Even if we try to create view of type field, 'ViewFields are not getting created.'

If we get existing list view using library code, we can see that all fields are present in 'ListViewXML' as below sample.

<View Name="{F2B2777A-7608-4EF8-80DC-786260376474}" Type="HTML" DisplayName="All Test Documents View" Url="/sites/commsite12/Shared Documents/Forms/All Test Documents View.aspx" Level="1" BaseViewID="1" ContentTypeID="0x" ImageUrl="/_layouts/15/images/dlicon.png?rev=47" ><Query><OrderBy><FieldRef Name="FileLeafRef" /></OrderBy></Query><ViewFields><FieldRef Name="DocIcon" /><FieldRef Name="LinkFilename" /><FieldRef Name="Modified" /><FieldRef Name="Editor" /><FieldRef Name="Mylocation" /><FieldRef Name="City" /><FieldRef Name="CountryOrRegion" /><FieldRef Name="SampleTest" /></ViewFields><RowLimit Paged="TRUE">30</RowLimit><Aggregations Value="Off" /><JSLink>clienttemplates.js</JSLink><XslLink Default="TRUE">main.xsl</XslLink><ViewData /><Toolbar Type="Standard"/></View>

For e.g. If we try to create new view using above ListViewXML, 'SampleTest' is not getting added to view.

Can you please provide some help
question

opened by swapnilgaonkar 4
Digest Header
I am using the saml authentication. To be able to write new files I needed to include the Digest value in the header:

POST request to /_api/contextinfo to retrieve the value and then set the header:

req.Header.Set("X-RequestDigest", "0x4C1XXXXXXXXXXXXXXXX,20 Mar 2019 12:26:18 -0000")

Maybe this could be documented or automated somehow?
enhancement
opened by jyniybinc 4

Modern In place record management

Record management documentation: https://go.spflow.com/samples/record-management

In the summary it says that you are mimicing the CSOM calls. This means you are doing "Classic IN PLACE Records Management"

Is there any plans to impliment "Modern IN PLACE Records Management"? As there is at least an api method for it now?

example of modern api call to declare record.

POST:  https://domain.sharepoint.com/sites/{nameOfSite}/_api/web/lists/GetByTitle/{nameOfList}/items({itemID})/SetComplianceTag()

BODY:
{
    "complianceTag": "{nameOfComplianceTag}",
    "isTagPolicyHold": "False",
    "isTagPolicyRecord": "True",
    "isEventBasedTag": "False",
    "isTagSuperLock": "False"
}

example of modern api call to undeclare record.

POST:  https://domain.sharepoint.com/sites/{nameOfSite}/_api/web/lists/GetByTitle/{nameOfList}/items({itemID})/SetComplianceTag()

BODY:
{
    "complianceTag": "{nameOfComplianceTag}",
    "isTagPolicyHold": "False",
    "isTagPolicyRecord": "False",
    "isEventBasedTag": "False",
    "isTagSuperLock": "False"
}

enhancement

opened by Ballzer0 4

Copy and move some item inside document library

maybe we can improve with copy and move

this is reference https://www.sharepointed.com/2018/06/copy-or-move-folder-structure-in-sharepoint-using-csom/ in CSOM
enhancement

opened by MisterFredy 2
Support for creating Site Collection

I can't find API in your library to create new Site Collection. There is API to create sub-site under particular site but no API to create Site collection
enhancement

opened by swapnilgaonkar 7
Share how you use the library and why even Go in combination with SharePoint?

SharePoint mostly was a void in Go lands before Gosip, IMO. Of course, a DIY option was always there, but realizing that SharePoint has so many authentication ways and that SharePoint APIs have so many gotchas can be painful. It can be discouraging fighting "how to deal with the API or establish an auth" but not solving actual problems and business logic and automation. As I know some SP-nuances a bit and fond of Go I considered using them both together around a year+ ago and make it reusable and simple for others.

When somebody hears "Go+SharePoint" they ask two questions "Why?" and "How?". Fair enough. I have the answers here. Yet, the goal of the ticket is to let others say how they use or going to use the library and why. That would be great feedback and a chance to hear from you. I know that current consumption is tiny and mostly done by myself, our company pipelines and code aggregators:

But anyways there should be somebody in the wild or in the future. Your help can't be underestimated.
help wanted

opened by koltyakov 2
Strategy interactive prompts

Integrate https://github.com/AlecAivazis/survey to provide interactive creds prompts. The idea is to introduce something like node-sp-auth-config or better. ;)
enhancement

opened by koltyakov 0