Golang library for querying and parsing OFX

Overview

OFXGo

Go Report Card Build Status Coverage Status PkgGoDev

OFXGo is a library for querying OFX servers and/or parsing the responses. It also provides an example command-line client to demonstrate the use of the library.

Goals

The main purpose of this project is to provide a library to make it easier to query financial information with OFX from the comfort of Golang, without having to marshal/unmarshal to SGML or XML. The library does not intend to abstract away all of the details of the OFX specification, which would be difficult to do well. Instead, it exposes the OFX SGML/XML hierarchy as structs which mostly resemble it. Its primary goal is to enable the creation of other personal finance software in Go (as it was created to allow me to fetch OFX transactions for my own project, MoneyGo).

Because the OFX specification is rather... 'comprehensive,' it can be difficult for those unfamiliar with it to figure out where to start. To that end, I have created a sample command-line client which uses the library to do simple tasks (currently it does little more than list accounts and query for balances and transactions). My hope is that by studying its code, new users will be able to figure out how to use the library much faster than staring at the OFX specification (or this library's API documentation). The command-line client also serves as an easy way for me to test/debug the library with actual financial institutions, which frequently have 'quirks' in their implementations. The command-line client can be found in the cmd/ofx directory of this repository.

Library documentation

Documentation can be found with the go doc tool, or at https://pkg.go.dev/github.com/aclindsa/ofxgo

Example Usage

The following code snippet demonstrates how to use OFXGo to query and parse OFX code from a checking account, printing the balance and returned transactions:

client := ofxgo.BasicClient{} // Accept the default Client settings

// These values are specific to your bank
var query ofxgo.Request
query.URL = "https://secu.example.com/ofx"
query.Signon.Org = ofxgo.String("SECU")
query.Signon.Fid = ofxgo.String("1234")

// Set your username/password
query.Signon.UserID = ofxgo.String("username")
query.Signon.UserPass = ofxgo.String("hunter2")

uid, _ := ofxgo.RandomUID() // Handle error in real code
query.Bank = append(query.Bank, &ofxgo.StatementRequest{
	TrnUID: *uid,
	BankAcctFrom: ofxgo.BankAcct{
		BankID:   ofxgo.String("123456789"),   // Possibly your routing number
		AcctID:   ofxgo.String("00011122233"), // Possibly your account number
		AcctType: ofxgo.AcctTypeChecking,
	},
	Include: true, // Include transactions (instead of only balance information)
})

response, _ := client.Request(&query) // Handle error in real code

// Was there an OFX error while processing our request?
if response.Signon.Status.Code != 0 {
	meaning, _ := response.Signon.Status.CodeMeaning()
	fmt.Printf("Nonzero signon status (%d: %s) with message: %s\n", response.Signon.Status.Code, meaning, response.Signon.Status.Message)
	os.Exit(1)
}

if len(response.Bank) < 1 {
	fmt.Println("No banking messages received")
	os.Exit(1)
}

if stmt, ok := response.Bank[0].(*ofxgo.StatementResponse); ok {
	fmt.Printf("Balance: %s %s (as of %s)\n", stmt.BalAmt, stmt.CurDef, stmt.DtAsOf)
	fmt.Println("Transactions:")
	for _, tran := range stmt.BankTranList.Transactions {
		currency := stmt.CurDef
		if ok, _ := tran.Currency.Valid(); ok {
			currency = tran.Currency.CurSym
		}
		fmt.Printf("%s %-15s %-11s %s%s%s\n", tran.DtPosted, tran.TrnAmt.String()+" "+currency.String(), tran.TrnType, tran.Name, tran.Payee.Name, tran.Memo)
	}
}

Similarly, if you have an OFX file available locally, you can parse it directly:

func main() {
	f, err := os.Open("./transactions.qfx")
	if err != nil {
		fmt.Printf("can't open file: %v\n", err)
		return
	}
	defer f.Close()

	resp, err := ofxgo.ParseResponse(f)
	if err != nil {
		fmt.Printf("can't parse response: %v\n", err)
		return
	}

	// do something with resp (*ofxgo.Response)
}

Requirements

OFXGo requires go >= 1.12

Using the command-line client

To install the command-line client and test it out, you may do the following:

$ go get -v github.com/aclindsa/ofxgo/cmd/ofx && go install -v github.com/aclindsa/ofxgo/cmd/ofx

Once installed (at ~/go/bin/ofx by default, if you haven't set $GOPATH), the command's usage should help you to use it (./ofx --help for a listing of the available subcommands and their purposes, ./ofx subcommand --help for individual subcommand usage).

Comments
  • "Windows" new line required for some banks

    It looks like some banks, like USAA, don't support standard unix line endings \n:

    OFXAdapter: Failed to parse request: Unable to parse an atomic field "OFXRequest.Headers.OFXHeader": unable to find end delimiters "
    ".
    

    Looking around, I found this issue which solved the problem by adding \r: https://github.com/euforic/banking.js/issues/8

    Based on the code in the python ofxclient library, it looks like using \r\n works for all banks: https://github.com/captin411/ofxclient/blob/v2.0.4/ofxclient/client.py#L25

    In my custom client, I've hacked around the problem with this line:

    var r *ofxgo.Request
    b, _ := r.Marshal()
    b = bytes.NewBuffer(bytes.Replace(b.Bytes(), []byte{'\n'}, []byte{'\r', '\n'}, -1))
    

    What are your thoughts on changing the line ending for all requests? If you like I can try my hand at a PR too 😄

    enhancement 
    opened by JohnStarich 12
  • Added Accept http header as Citi now requires it

    Added Accept http header as Citi now requires it

    Without this header set Citi returns http 403 for every request.

    I spent far to long looking into why Citi started returning only 403s for me. I eventually found that the software pocketsense (https://pocketsense.blogspot.com/2021/08/dev-testing-continued.html) was having a similar issue and it was resolved by adding this http header to the requests. I confirmed it resolves the issue for me as well.

    opened by Johnny-Malizia 10
  • Rename ofxgo_test package to ofxgo and remove self-imports/references

    Rename ofxgo_test package to ofxgo and remove self-imports/references

    It turned out that removing the self-imports was more involved than I previously thought! This PR renames the test package to use the same name (ofxgo) so imports aren't required to run the tests. I've also un-exported ErrInvalid so it is no longer part of the API surface area.

    Luckily sed is amazing, so 99% of the renaming and import deletion effort was automatic 😅

    Since this touches so many lines in the library, I figured it'd be worth breaking this out into it's own PR, for our sanity.

    opened by JohnStarich 10
  • can't parse response: OFX SECURITY header not NONE

    can't parse response: OFX SECURITY header not NONE

    Parsing OFX responses with SECURITY setting of TYPE1 fails with a validation error:

    can't parse response: OFX SECURITY header not NONE
    

    My bank still uses OFX 1.02 OFXSGML format, but uses SECURITY:TYPE1:

    OFXHEADER:100
    DATA:OFXSGML
    VERSION:102
    SECURITY:TYPE1
    ENCODING:USASCII
    CHARSET:1252
    COMPRESSION:NONE
    OLDFILEUID:NONE
    NEWFILEUID:NONE
    

    This is parsed beautifully after I change that security setting to NONE:

    OFXHEADER:100
    DATA:OFXSGML
    VERSION:102
    SECURITY:NONE
    ENCODING:USASCII
    CHARSET:1252
    COMPRESSION:NONE
    OLDFILEUID:NONE
    NEWFILEUID:NONE
    

    Apparently, this is a common issue in financial software processing .ofx responses, with the "fix" for users to hand-edit the responses to remove the TYPE1 security value and replace it with NONE:

    When I went digging to learn more about this setting I found Open Financial Exchange Specification 2.2, Nov 26, 2017 which states in section 4.2.2.2 Type 1 Protocol Overview that:

        Type 1 applies only to the request part of a message; the server response is unaffected.
    

    Thus I think we could remove [this validation](- https://github.com/aclindsa/ofxgo/blob/2b8a79e4b74b0ca32c2b6f21147841d22c656dca/response.go#L95-L98 ) as TYPE1 security doesn't seem to impact responses.

    opened by davidalpert 7
  • Return parsed response when only failing validation

    Return parsed response when only failing validation

    Hey again Aaron! Let me know your thoughts on this more flexible version of ofxgo.ParseResponse. It still returns an error if things are not parseable, but attempts to continue decoding if it's only hitting Valid()-ation errors. This way I can write institution-specific handling of validation errors and still use the response object.

    I ran into this issue with Ally Bank where their severity in the transaction list is not uppercase like the OFX 102 spec requires. I've contacted them, but I don't expect they'll actually change anything since "it works for Quicken". 😉

    opened by JohnStarich 7
  • Marshalled requests for OFX 102 should not include element end tags

    Marshalled requests for OFX 102 should not include element end tags

    The OFX 102 spec is a bit lacking in this area, but it seems that some institutions require ofx elements not include their end tags (aggregates are ok though).

    The key rule of Open Financial Exchange syntax is that each tag is either an element or an aggregate. Data follows its element tag. An aggregate tag begins a compound tag sequence, which must end with a matching tag; for example, <AGGREGATE> ... </AGGREGATE>.

    From the reading above and my use of the python ofxclient, I think always omitting the element end tags should work for all institutions using OFX 102 (maybe all 100-series, but I haven't looked yet).

    In my case, USAA doesn't like me using any element end tags. This is my hacky work-around:

    var r *ofxgo.Request
    b, _ := r.Marshal()
    
    data := b.Bytes()
    for _, s := range []string{"DTCLIENT", "DTSTART", "DTEND", "INCLUDE", "USERID", "USERPASS", "LANGUAGE", "ORG", "FID", "APPID", "APPVER", "TRNUID", "BANKID", "ACCTID", "ACCTTYPE"} {
    	data = bytes.Replace(data, []byte("</"+s+">"), []byte{}, -1)
    }
    b = bytes.NewBuffer(data)
    

    What do you think would be a good solution to this problem? I briefly looked at your fork of go's xml package, looks like that might be a good place for it. Maybe it could be some kind of encoder option?

    enhancement 
    opened by JohnStarich 7
  • QFX downloads from BMO do not have a newline between SGML and XML section

    QFX downloads from BMO do not have a newline between SGML and XML section

    This is how the beginning of the file looks like

    OFXHEADER:100
    DATA:OFXSGML
    VERSION:102
    SECURITY:NONE
    ENCODING:USASCII
    CHARSET:1252
    COMPRESSION:NONE
    OLDFILEUID:NONE
    NEWFILEUID:NONE
    <OFX>
    <SIGNONMSGSRSV1>
    <SONRS>
    <STATUS>
    <CODE>0
    <SEVERITY>INFO
    <MESSAGE>OK
    </STATUS>
    ...
    

    I am not sure if this is technically valid OFX, but Bank of Montreal is a major Canadian Bank. Feel free to reject if you think the bank is plain wrong. I should also mention that the bank refers to the export format as QFX, which is possibly a variation of OFX.

    opened by mkobetic 7
  • BMO produces invalid XML for BANKMSGSETV1

    BMO produces invalid XML for BANKMSGSETV1

    Bank of Montreal (BMO) allows downloads of multiple accounts at once. It uses BANKMSGSETV1 element for that (with embedded BANKMSGSET inside it for some reason), but it emits start elements instead of end elements at the end 🤦‍♂️ .

    OFXHEADER:100
    DATA:OFXSGML
    VERSION:102
    SECURITY:NONE
    ENCODING:USASCII
    CHARSET:1252
    COMPRESSION:NONE
    OLDFILEUID:NONE
    NEWFILEUID:NONE
    
    <OFX>
    <SIGNONMSGSRSV1>
    <SONRS>
    <STATUS>
    <CODE>0
    <SEVERITY>INFO
    <MESSAGE>OK
    </STATUS>
    <DTSERVER>20181228000224.466[-5:EDT]
    <USERKEY>329402394029374
    <LANGUAGE>ENG
    <INTU.BID>00001
    </SONRS>
    </SIGNONMSGSRSV1>
    <BANKMSGSETV1><BANKMSGSET>
    <BANKMSGSRSV1>
    
    <STMTTRNRS>
    ...
    </STMTTRNRS>
    
    </BANKMSGSRSV1>
    <BANKMSGSET><BANKMSGSETV1>
    </OFX>
    

    It seems that the library currently doesn't support MSGSET elements at all, but even if it did the end elements would still be a problem. I could get the file through by making response parsing ignore MSGSET elements completely, as follows:

    @@ -356,7 +357,13 @@ func ParseResponse(reader io.Reader) (*Response, error) {
                            return nil, err
                    } else if ofxEnd, ok := tok.(xml.EndElement); ok && ofxEnd.Name.Local == "OFX" {
                            return &or, nil // found closing XML element, so we're done
    +               } else if ofxEnd, ok := tok.(xml.EndElement); ok && strings.Contains(ofxEnd.Name.Local, "MSGSET") {
    +                       continue // ignore MSGSET elements
                    } else if start, ok := tok.(xml.StartElement); ok {
    +                       // ignore MSGSET elements
    +                       if strings.Contains(start.Name.Local, "MSGSET") {
    +                               continue
    +                       }
                            slice, ok := messageSlices[start.Name.Local]
                            if !ok {
                                    return nil, errors.New("Invalid message set: " + start.Name.Local)
    

    Note that I had to ignore EndElements as well, to allow the parser to unwind its element stack properly. The way the sample above is parsed is as if there are two sets of MSGSET elements nested in each other and the end elements get injected from the parser stack when it encounters the closing OFX element.

    Anyway, I admit that this is outright disgusting, but it does seem to allow processing the file correctly including multiple account message sets. I absolutely understand if this is too much of a hack to allow in. I can muscle my way through outside of the ofxgo library if needed (just use some grep -v MSGSET equivalent), but I figured I'd bring it up anyway.

    opened by mkobetic 6
  • Pending transactions missing from query responses

    Pending transactions missing from query responses

    I'm currently using my personal Chase Bank checking, savings and credit accounts to play with this package. Most things are working as expected however I am not able to get my statement balances and transactions to be as up to date as I would like. It appears the responses ignore pending charges entirely regardless of whether the IncludePending flag is set to true. Have you seen this behavior before? I have to imagine retrieving pending charges would be supported by chase but I could be wrong.

    question 
    opened by jwaggs 4
  • slow go-get because of aclindsa/go dep

    slow go-get because of aclindsa/go dep

    Thanks for releasing this library.

    I'm excited to use it, but it's taking a long time to download. go get is getting stuck on downloading aclindsa/go, which seems to be a big repo.

    Is it possible to use the default Go encoding/xml or put a fork in a smaller git repo?

    enhancement 
    opened by maxhawkins 3
  • Unable to connect to any of my banks using this

    Unable to connect to any of my banks using this

    query.URL = "https://www.oasis.cfree.com/fip/genesis/prod/02601.ofx"
    query.Signon.Org = ofxgo.String("ISC")
    query.Signon.Fid = ofxgo.String("2601")
    

    I keep getting no bank messages received using the example. with my credentials and bank id and acctid in place.

    opened by ldrrp 2
Releases(v0.1.1)
  • v0.1.1(Apr 3, 2020)

    This release features many additions/updates to make it easier (or in some cases possible) for ofxgo to interact with financial institutions which have their own quirks.

    Sometimes they need carriage returns in addition to standard newlines, sometimes they don't put newlines after their headers, sometimes they fail to correctly capitalize OFX elements, and sometimes they need exactly the right HTTP headers in exactly the right order... you get the idea.

    Source code(tar.gz)
    Source code(zip)
  • v0.1.0(Nov 18, 2017)

    This is the first official release of OFXGo. It supports both fetching OFX data from financial institutions directly and parsing OFX files into Golang structs, with convenience functions for doing both at once. While it does not yet understand the entire OFX specification, it is quite comprehensive and I haven't encountered OFX in the wild that it does not understand and parse correctly.

    Source code(tar.gz)
    Source code(zip)
Owner
Aaron Lindsay
Aaron Lindsay
Golang string comparison and edit distance algorithms library, featuring : Levenshtein, LCS, Hamming, Damerau levenshtein (OSA and Adjacent transpositions algorithms), Jaro-Winkler, Cosine, etc...

Go-edlib : Edit distance and string comparison library Golang string comparison and edit distance algorithms library featuring : Levenshtein, LCS, Ham

Hugo Bollon 361 Sep 27, 2022
Golang library for reading and writing Microsoft Excel™ (XLSX) files.

Excelize Introduction Excelize is a library written in pure Go providing a set of functions that allow you to write to and read from XLSX / XLSM / XLT

360 Enterprise Security Group, Endpoint Security, inc. 13k Oct 2, 2022
Go (golang) library for reading and writing XLSX files.

XLSX Introduction xlsx is a library to simplify reading and writing the XML format used by recent version of Microsoft Excel in Go programs. Tutorial

Geoffrey J. Teale 5.4k Oct 2, 2022
Library for hashing any Golang interface

recursive-deep-hash Library for hashing any Golang interface Making huge struct comparison fast & easy How to use package main import ( "fmt" "git

Panos Petropoulos 6 Mar 3, 2022
A radix sorting library for Go (golang)

zermelo A radix sorting library for Go. Trade memory for speed! import "github.com/shawnsmithdev/zermelo" func foo(large []uint64) zermelo.Sort(l

Shawn Smith 48 Jul 30, 2022
Go native library for fast point tracking and K-Nearest queries

Geo Index Geo Index library Overview Splits the earth surface in a grid. At each cell we can store data, such as list of points, count of points, etc.

Hailo Network IP Ltd 340 Sep 26, 2022
Data structure and algorithm library for go, designed to provide functions similar to C++ STL

GoSTL English | 简体中文 Introduction GoSTL is a data structure and algorithm library for go, designed to provide functions similar to C++ STL, but more p

stirlingx 714 Sep 23, 2022
Zero allocation Nullable structures in one library with handy conversion functions, marshallers and unmarshallers

nan - No Allocations Nevermore Package nan - Zero allocation Nullable structures in one library with handy conversion functions, marshallers and unmar

Andrey Kuzmin 56 Aug 31, 2022
A feature complete and high performance multi-group Raft library in Go.

Dragonboat - A Multi-Group Raft library in Go / 中文版 News 2021-01-20 Dragonboat v3.3 has been released, please check CHANGELOG for all changes. 2020-03

lni 4.4k Sep 24, 2022
Document Indexing and Searching Library in Go

Fehrist Fehrist is a pure Go library for indexing different types of documents. Currently it supports only CSV and JSON but flexible architecture give

Adnan Siddiqi 16 May 22, 2022
A generic Go library for implementations of tries (radix trees), state commitments and proofs of inclusion

trie.go Go library for implementations of tries (radix trees), state commitments and proof of inclusion for large data sets. It implements a generic 2

IOTA 5 Aug 3, 2022
Juniper is an extension to the Go standard library using generics, including containers, iterators, and streams.

Juniper Juniper is a library of extensions to the Go standard library using generics, including containers, iterators, and streams. container/tree con

Braden Walker 481 Sep 18, 2022
A small flexible merge library in go

conjungo A merge utility designed for flexibility and customizability. The library has a single simple point of entry that works out of the box for mo

InVision 107 Sep 6, 2022
A Go library for an efficient implementation of a skip list: https://godoc.org/github.com/MauriceGit/skiplist

Fast Skiplist Implementation This Go-library implements a very fast and efficient Skiplist that can be used as direct substitute for a balanced tree o

Maurice Tollmien 227 Sep 7, 2022
Go Library [DEPRECATED]

Tideland Go Library Description The Tideland Go Library contains a larger set of useful Google Go packages for different purposes. ATTENTION: The cell

Tideland 195 Sep 23, 2022
an R-Tree library for Go

rtreego A library for efficiently storing and querying spatial data in the Go programming language. About The R-tree is a popular data structure for e

Daniel Connelly 538 Sep 23, 2022
indexing library for Go

Bluge modern text indexing in go - blugelabs.com Features Supported field types: Text, Numeric, Date, Geo Point Supported query types: Term, Phrase, M

Bluge Labs 1.4k Sep 25, 2022
Go library implementing xor filters

xorfilter: Go library implementing xor filters Bloom filters are used to quickly check whether an element is part of a set. Xor filters are a faster a

null 588 Sep 25, 2022
The Go library that will drive you to AOP world!

Beyond The Golang library that will drive you to the AOP paradigm world! Check Beyond Documentation What's AOP? In computing, aspect-oriented programm

Wesovi Labs 49 Sep 27, 2022