General purpose library for reading, writing and working with OpenStreetMap data


osm CI Go Report Card Go Reference

This package is a general purpose library for reading, writing and working with OpenStreetMap data in Go (golang). It has the ability to read OSM XML and PBF data formats available at or via the v0.6 API.

Made available by the package are the following types:

  • Node
  • Way
  • Relation
  • Changeset
  • Note
  • User

And the following “container” types:

List of sub-package utilities

  • annotate - adds lon/lat, version, changeset and orientation data to way and relation members
  • osmapi - supports all the v0.6 read/data endpoints
  • osmgeojson - OSM to GeoJSON conversion compatible with osmtogeojson
  • osmpbf - stream processing of *.osm.pbf files
  • osmxml - stream processing of *.osm xml files
  • replication - fetch replication state and change files


This package refers to the core OSM data types as Objects. The Node, Way, Relation, Changeset, Note and User types implement the osm.Object interface and can be referenced using the osm.ObjectID type. As a result it is possible to have a slice of []osm.Object that contains nodes, changesets and users.

Individual versions of the core OSM Map Data types are referred to as Elements and the set of versions for a give Node, Way or Relation is referred to as a Feature. For example, an osm.ElementID could refer to "Node with id 10 and version 3" and the osm.FeatureID would refer to "all versions of node with id 10." Put another way, features represent a road and how it's changed over time and an element is a specific version of that feature.

A number of helper methods are provided for dealing with features and elements. The idea is to make it easy to work with a Way and its member nodes, for example.

Scanning large data files

For small data it is possible to use the encoding/xml package in the Go standard library to marshal/unmarshal the data. This is typically done using the osm.OSM or osm.Change "container" structs.

For large data the package defines the Scanner interface implemented in both the osmxml and osmpbf sub-packages.

type osm.Scanner interface {
	Scan() bool
	Object() osm.Object
	Err() error
	Close() error

This interface is designed to mimic the bufio.Scanner interface found in the Go standard library.

Example usage:

f, err := os.Open("./delaware-latest.osm.pbf")
if err != nil {
defer f.Close()

scanner := osmpbf.New(context.Background(), f, 3)
defer scanner.Close()

for scanner.Scan() {
	o := scanner.Object()
	// do something

scanErr := scanner.Err()
if scanErr != nil {

Note: Scanners are not safe for parallel use. One should feed the objects into a channel and have workers read from that.

  • Add support for node locations on ways

    Add support for node locations on ways


    I would like to add support for ways with node locations on them, an extension of the OSM PBF format put forward by @joto.

    This is the approach I have taken. I started by taking over the latest version of osmformat.proto, which has delta encoded packed sint64 fields numbered 9 (lat) and 10 (lon) in addition to the field 8 (refs) containing the IDs of all the nodes in the way. Then, after running protoc and taking a look at the current code for way nodes and node coordinates, I added similar logic for inserting the way node coordinates (if present) into the already existing WayNode.Lat and WayNode.Lon fields. Testing with a file generated by osmium add-locations-to-ways, this seems to give the desired result and processing regular OSM PBF files should be unaffected.

    Although it seems to me this pull request is OK, I would appreciate if you could look into it, especially since I don't yet feel very confident working with protocol buffers. Here are a few additional remarks:

    • When taking over the latest osmformat.proto, I have taken care not to overwrite the existing changes related to gogoprotobuf and to eliminate conversions from []byte to string.
    • Before all of the steps I mentioned so far, I have just run the protoc command from generate.go with no other changes. This already modified the generated Go files, probably because I am using the most recent version. Since I don't know what version was used previously, I stuck with these changes and kept them as the first commit.
    • The most important consequence of running protoc was the change of the package name from osmpbf to OSMPBF in the generated Go files. It did not seem a good idea to manually change the generated files, so I've chosen to replace the import paths to this internal package with the uppercase verision, as can be seen in the second commit.

    I'd love to hear your feedback on this!

    opened by flowrean 9
  • Huge Memory Usage

    Huge Memory Usage

    Hi there. First off, thanks for this great library.

    When trying to scan a 700Mb *osm.pbf file, the mem usage spikes to over 145Gb with just a single go routine. I know this because I am trying this task in an AWS CodeBuild project with max memory allowed (145 Gb) and it still gets killed.

    This is surprising considering that the Osmosis CLI tool can accomplish the same task (scanning the 700Mb pbf file and filtering by a bounding box) without breaching 7Gb of memory.

    Why does this tool take over 20 times more memory than Osmosis, and is there anything we can do to improve this? I'm willing to help if it's a big task.


    opened by tyliggity 7
  • [Feature Request] Provide timestamp to replication sequence number function

    [Feature Request] Provide timestamp to replication sequence number function

    This enables us to reimplement this script in go. Your library is a great candidate to use when working with osm data.

    opened by hypervtechnics 5
  • Feature request: Specify which OSM entities should be read from the file

    Feature request: Specify which OSM entities should be read from the file

    Hello and thank you for this library.

    In libosmium, osmium::io::Reader can be initialized with osmium::osm_entities::bits to specify which OSM entities should be parsed and passed to the handler (docs, example). Specifying only the entities you want to handle does significantly improve read performance in my experience with libosmium.

    I believe it can be done without changing the current behavior by adding new initialization function, perhaps osmpbf.NewWithEntities.

    I appreciate your consideration.

    opened by invisiblefunnel 5
  • gogo/protobuf security issue

    gogo/protobuf security issue

    Thanks for a great project.

    There is a security issue detected by Dependabot

    All that is needed is to upgrade the gogo version to the latest one (v1.3.2 I believe) where this is fixed.

    As a side note, the gogo project is not really maintained anymore. Has anyone tested the performance of using the standard proto package?

    opened by iulian-gm 4
  • osmpbf: support

    osmpbf: support "sparse" dense nodes

    Not all pbf files will have all the timestamp, username, changeset, etc. data so we should be okay with skipping that. No need to stop and return an error.

    @dmholtz does this work on your file?

    opened by paulmach 4
  • How to handle dense nodes?

    How to handle dense nodes?

    Decoding OSM PBF files, which contain 'dense nodes' fails:

    panic: dense node does not include all required fields
    goroutine 1 [running]:
            /Users/[user]/repos/[repo]/cmd/importer/main.go:76 +0x468
    exit status 2

    The error occurs within the example code provided in the of the osmpbf package.

    The error does not occur with OSM PBF file, which do not contain 'dense nodes' like Antarctica from Geofabrik

    Is this a bug or have I missed anything?

    After some research, it seems to me that another library just ignores the missing fields and the decoding suceeds. Thus, the issue seems to be closely related to my OSM PBF file (which I can unfortunately not share due to its size). It would be interesting to hear if your implementation supports a less strict mode, which also just ignores these fields.

    opened by dmholtz 4
  • Question: Why output and serializer chanel

    Question: Why output and serializer chanel

    Is there a specific reason to have in osmpbf part of the project n output chanels and feeding them in order to a serializer chanel ? (decode.go function line number

    The upside is it is making testing much easier since we get predictable outputs, however in my case I do not see not a reason that my application needs a special ordering of the elements : Do I miss something important why one should process objects in order ? (In other words: why doesn't the deocoder directly push to serializer)

    opened by oflebbe 3
  • protobuf: port to google protobuf

    protobuf: port to google protobuf

    I replaced gogo protobuf by the plain google protobuf implementation.

    I had to rewrite some portions regarding nullability. Hope I got it right, there might be better way to accomplish.

    Overall I was surprised that the CPU performance drop is below 3% , I was expecting much more. The number of allocations is much higher for some benchmarks, since I had to alloc temporary objects because of nullability changes above. The only one I am concerned is the change in memory for BenchmarkLondon_relations-16

    BenchmarkDiff_Marshal-16                     33708352       33497339       -0.63%
    BenchmarkDiff_Unmarshal-16                   66501681       65426347       -1.62%
    BenchmarkElementID_Sort-16                   648905         653604         +0.72%
    BenchmarkFeatureIDs_Sort-16                  636472         631671         -0.75%
    BenchmarkChange_MarshalXML-16                1790183        1744972        -2.53%
    BenchmarkChangeSet_Marshal-16                79831          77896          -2.42%
    BenchmarkUnmarshalWays-16                    5077           6401           +26.08%
    BenchmarkUnmarshalRelations-16               5508           6938           +25.96%
    BenchmarkChangeset_Marshal_gzip-16           450378         456373         +1.33%
    BenchmarkChangeset_UnmarshalXML-16           3468850        3504131        +1.02%
    BenchmarkUnmarshalChangeset-16               73306          81317          +10.93%
    BenchmarkUnmarshalChangeset_gzip-16          137123         144377         +5.29%
    BenchmarkRelations-16                        638028         633393         -0.73%
    BenchmarkWay-16                              14600          14479          -0.83%
    BenchmarkWays-16                             23030          22524          -2.20%
    BenchmarkConvert-16                          1211337        1202652        -0.72%
    BenchmarkConvertAnnotated-16                 947576         940853         -0.71%
    BenchmarkConvert_NoID-16                     1084547        1063003        -1.99%
    BenchmarkConvert_NoMeta-16                   996524         991362         -0.52%
    BenchmarkConvert_NoRelationMembership-16     1157384        1148647        -0.75%
    BenchmarkConvert_NoIDsMetaMembership-16      747993         748889         +0.12%
    BenchmarkLondon-16                           257786472      262947900      +2.00%
    BenchmarkLondon_withFiltersTrue-16           266369759      269555246      +1.20%
    BenchmarkLondon_withFiltersFalse-16          208942287      206641137      -1.10%
    BenchmarkLondon_nodes-16                     189334304      192258210      +1.54%
    BenchmarkLondon_ways-16                      147202951      150169294      +2.02%
    BenchmarkLondon_relations-16                 90644410       93034313       +2.64%
    BenchmarkAndorra-16                          3074774010     3111769317     +1.20%
    benchmark                                    old allocs     new allocs     delta
    BenchmarkDiff_Marshal-16                     129484         129484         +0.00%
    BenchmarkDiff_Unmarshal-16                   422916         422916         +0.00%
    BenchmarkElementID_Sort-16                   1              1              +0.00%
    BenchmarkFeatureIDs_Sort-16                  1              1              +0.00%
    BenchmarkChange_MarshalXML-16                4385           4385           +0.00%
    BenchmarkChangeSet_Marshal-16                291            242            -16.84%
    BenchmarkUnmarshalWays-16                    74             97             +31.08%
    BenchmarkUnmarshalRelations-16               81             105            +29.63%
    BenchmarkChangeset_Marshal_gzip-16           315            265            -15.87%
    BenchmarkChangeset_UnmarshalXML-16           23326          23326          +0.00%
    BenchmarkUnmarshalChangeset-16               346            617            +78.32%
    BenchmarkUnmarshalChangeset_gzip-16          390            660            +69.23%
    BenchmarkRelations-16                        1417           1417           +0.00%
    BenchmarkWay-16                              126            126            +0.00%
    BenchmarkWays-16                             116            116            +0.00%
    BenchmarkConvert-16                          10114          10114          +0.00%
    BenchmarkConvertAnnotated-16                 10107          10107          +0.00%
    BenchmarkConvert_NoID-16                     8501           8501           +0.00%
    BenchmarkConvert_NoMeta-16                   6898           6898           +0.00%
    BenchmarkConvert_NoRelationMembership-16     9531           9531           +0.00%
    BenchmarkConvert_NoIDsMetaMembership-16      4702           4703           +0.02%
    BenchmarkLondon-16                           4805957        4811479        +0.11%
    BenchmarkLondon_withFiltersTrue-16           4805954        4811469        +0.11%
    BenchmarkLondon_withFiltersFalse-16          1362152        1367677        +0.41%
    BenchmarkLondon_nodes-16                     3392981        3398512        +0.16%
    BenchmarkLondon_ways-16                      1793525        1799043        +0.31%
    BenchmarkLondon_relations-16                 457585         463104         +1.21%
    BenchmarkAndorra-16                          13038438       13038444       +0.00%
    benchmark                                    old bytes     new bytes     delta
    BenchmarkDiff_Marshal-16                     11232715      11232715      +0.00%
    BenchmarkDiff_Unmarshal-16                   15975454      15975430      -0.00%
    BenchmarkElementID_Sort-16                   24            24            +0.00%
    BenchmarkFeatureIDs_Sort-16                  24            24            +0.00%
    BenchmarkChange_MarshalXML-16                593113        593113        +0.00%
    BenchmarkChangeSet_Marshal-16                68687         37545         -45.34%
    BenchmarkUnmarshalWays-16                    6888          6984          +1.39%
    BenchmarkUnmarshalRelations-16               7248          7536          +3.97%
    BenchmarkChangeset_Marshal_gzip-16           891254        860106        -3.49%
    BenchmarkChangeset_UnmarshalXML-16           1131472       1131467       -0.00%
    BenchmarkUnmarshalChangeset-16               148593        173889        +17.02%
    BenchmarkUnmarshalChangeset_gzip-16          225217        250481        +11.22%
    BenchmarkRelations-16                        245465        245441        -0.01%
    BenchmarkWay-16                              9622          9622          +0.00%
    BenchmarkWays-16                             11640         11639         -0.01%
    BenchmarkConvert-16                          913288        913285        -0.00%
    BenchmarkConvertAnnotated-16                 830581        830577        -0.00%
    BenchmarkConvert_NoID-16                     890772        890778        +0.00%
    BenchmarkConvert_NoMeta-16                   702989        703007        +0.00%
    BenchmarkConvert_NoRelationMembership-16     895883        895846        -0.00%
    BenchmarkConvert_NoIDsMetaMembership-16      663290        663296        +0.00%
    BenchmarkLondon-16                           918437913     938188674     +2.15%
    BenchmarkLondon_withFiltersTrue-16           918437519     938187697     +2.15%
    BenchmarkLondon_withFiltersFalse-16          360101121     379852203     +5.48%
    BenchmarkLondon_nodes-16                     613038341     632790024     +3.22%
    BenchmarkLondon_ways-16                      432007871     451758126     +4.57%
    BenchmarkLondon_relations-16                 178275503     198026162     +11.08%
    BenchmarkAndorra-16                          471254696     471255184     +0.00%```
    opened by ghost 3
  • osmgeojson support overpass geometry output conversion

    osmgeojson support overpass geometry output conversion

    The osmgeojson package currently doesn't support converting the output of overpass queries (outputted as geom) to geojson. An example:

    If you use overpass to request the geometry of manchester as following:

    out geom;

    The result is an osm xml with 1 relation, Manchester, containing members (interesting nodes and all ways that are needed to create the geometry of Manchester.

    Currently the library needs all ways and nodes that are member of a relation to be separately defined in the osm xml.

    If you find the time!

    opened by zeevogels 3
  • osmpbf: do not preallocate nodes

    osmpbf: do not preallocate nodes

    regarding on high memory usage.

    I noticed that pre-allocating the nodes could cause problems if you're saving some of them for later use. If just 1 node out of the whole 8000 node array is saved, the GC can not cleanup the any of them.

    Benchmarks are impacted:

    benchmark                        old ns/op     new ns/op     delta
    BenchmarkLondon-12               235836166     248139745     +5.22%
    BenchmarkLondon_nodes-12         163988036     177138150     +8.02%
    BenchmarkLondon_ways-12          136805230     135422092     -1.01%
    BenchmarkLondon_relations-12     82180166      81767765      -0.50%
    benchmark                        old allocs     new allocs     delta
    BenchmarkLondon-12               2416813        5145479        +112.90%
    BenchmarkLondon_nodes-12         1003846        3732512        +271.82%
    BenchmarkLondon_ways-12          1792714        1792718        +0.00%
    BenchmarkLondon_relations-12     456772         456772         +0.00%
    benchmark                        old bytes     new bytes     delta
    BenchmarkLondon-12               954879689     952781068     -0.22%
    BenchmarkLondon_nodes-12         649480586     647382889     -0.32%
    BenchmarkLondon_ways-12          432818482     432819399     +0.00%
    BenchmarkLondon_relations-12     179085842     179085459     -0.00%
    opened by paulmach 2
  • Unmarshaling of Overpass JSON fails on version field

    Unmarshaling of Overpass JSON fails on version field

    The version information is encoded as JSON number instead of a string by Overpass API. This causes unmarshaling it to fail with error json: cannot unmarshal number into Go struct field .version of type string

    Take, for example, the minimal JSON output of overpass ([out:json];out;), e.g.

      "version": 0.6,
      "generator": "Overpass API 0.7.59 e21c39fe",
      "osm3s": {
        "timestamp_osm_base": "2022-11-17T17:14:39Z",
        "copyright": "The data included in this document is from The data is made available under ODbL."
      "elements": []

    Minimum viable example failure:

    package main
    import (
    func main() {
    	var overpass osm.OSM
    	buf := []byte(`{
      "version": 0.6,
      "generator": "Overpass API 0.7.59 e21c39fe",
      "osm3s": {
        "timestamp_osm_base": "2022-11-17T17:14:39Z",
        "copyright": "The data included in this document is from The data is made available under ODbL."
      "elements": []
    	err := json.Unmarshal(buf, &overpass)
    	if err != nil {
    opened by fawick 3
  • pkg-config: exec:

    pkg-config: exec: "pkg-config": executable file not found in $PATH. on M1 mac

    this package has a dependency (either directly, or somewhere upstream; I didn't investigate in detail) that will lead to this error on a clean Go lang install on M1 mac: # pkg-config --cflags -- zlib

    The recommended solution seems to be to use homebrew to install the pkg-config utility brew install pkg-config

    I'm leaving this issue in case it helps future Mac users. Perhaps add an entry to the README:

    ##INSTALLATION - Apple Silicon Notes
    This package require the "pkg-config" utility which must be installed on Mac using Homebrew. Install Homebrew then install the "pkg-config" package using this comnand:
      brew install pkg-config
    opened by morandd 0
  • Tags.Find() should return nil not

    Tags.Find() should return nil not ""

    OSM nodes can contain tags which specify only a key but no value. For example "building=" is common. Tags.Find() should return [nil] if no key is found, since returning "" could mean that either the tag is missing or that it is present but empty. It is common to want to catch even empty tags.

    opened by morandd 2
  • v0.7.0(Aug 17, 2022)

    What's Changed

    • remove support for marshalling core types into custom binary format by @paulmach in

    Full Changelog:

    Source code(tar.gz)
    Source code(zip)
  • v0.6.0(Aug 16, 2022)

    What's Changed

    • json: ability to unmarshal osmjson by @paulmach in
    • json: add support for external json implementations by @paulmach in

    Full Changelog:

    Source code(tar.gz)
    Source code(zip)
  • v0.5.0(Jun 7, 2022)

    What's Changed

    • replication: ability to get changeset state by @paulmach in
    • replication: search for state/sequence number by timestamp by @paulmach in

    Full Changelog:

    Source code(tar.gz)
    Source code(zip)
  • v0.4.0(May 26, 2022)

    What's Changed

    • protobuf: port to google protobuf by @OlafFlebbeBosch in

    New Contributors

    • @OlafFlebbeBosch made their first contribution in

    Full Changelog:

    Source code(tar.gz)
    Source code(zip)
  • v0.3.0(Apr 21, 2022)

    What's Changed

    • osmpbf: preallocation node tags array by @paulmach in
    • osmpbf: support "sparse" dense nodes by @paulmach in
    • osmpbf: add filter functions by @paulmach in

    Full Changelog:

    Source code(tar.gz)
    Source code(zip)
  • v0.2.2(Apr 27, 2021)

  • v0.2.1(Feb 4, 2021)

    • Reduces memory usage when decoding a OSM PBF file, thank you @oflebbe
    • Fix some more typos, thank you @meyermarcel
    Source code(tar.gz)
    Source code(zip)
  • v0.2.0(Jan 9, 2021)

    4 changes:

    • ability to efficiently skip types when decoding osmpbf
    • the above also added protoscan for osmpbf decoding for a 10%ish performance improvement
    • use cgo/czlib to decode protobufs (if cgo enabled), 20% faster on benchmarks
    • deprecated node/ways/relations marshaling into this packages custom binary format. It could be improved and should be versioned separately.
    Source code(tar.gz)
    Source code(zip)
  • v0.1.1(Jun 22, 2020)

  • v0.1.0(Oct 2, 2019)

Paul Mach
Paul Mach
Package kml provides convenince methods for creating and writing KML documents.

go-kml Package kml provides convenience methods for creating and writing KML documents. Key Features Simple API for building arbitrarily complex KML d

Tom Payne 66 Jul 29, 2022
Types and utilities for working with 2d geometry in Golang

orb Package orb defines a set of types for working with 2d geo and planar/projected geometric data in Golang. There are a set of sub-packages that use

Paul Mach 601 Nov 22, 2022
Publish Your GIS Data(Vector Data) to PostGIS and Geoserver

GISManager Publish Your GIS Data(Vector Data) to PostGIS and Geoserver How to install: go get -v Usage: testdata fol

Hisham waleed karam 48 Sep 26, 2022
A library provides spatial data and geometric algorithms

Geoos Our organization spatial-go is officially established! The first open source project Geoos(Using Golang) provides spatial data and geometric alg

spatial-go 546 Nov 16, 2022
Go (golang) wrapper for GDAL, the Geospatial Data Abstraction Library

------------- About ------------- The gdal.go package provides a go wrapper for GDAL, the Geospatial Data Abstraction Library. More information about

null 232 Nov 12, 2022
Go package to quick and easy create json data in geojson format.

#GEOJSON Go package to easy and quick create datastructure which can be serialized to geojson format INSTALLATION $ go get

Kris Pawlik 50 Jun 6, 2022
Geometry/geography library in Go, DEPRECATED, use ->

go.geo (deprecated) Go.geo is a geometry/geography library in Go. The primary use case is GIS geometry manipulation on the server side vs. in the brow

Paul Mach 321 Aug 12, 2022
Go bindings for the Cartographic Projections Library PROJ.4

The Go package proj provides a limited interface to the Cartographic Projections Library PROJ. For PROJ version 5 and beyond, see also: https://github

Peter Kleiweg 42 Nov 10, 2022
geoserver is a Go library for manipulating a GeoServer instance via the GeoServer REST API.

Geoserver geoserver Is a Go Package For Manipulating a GeoServer Instance via the GeoServer REST API. How to install: go get -v

Hisham waleed karam 76 Sep 26, 2022
S2 geometry library in Go

Overview S2 is a library for spherical geometry that aims to have the same robustness, flexibility, and performance as the best planar geometry librar

Go 1.5k Nov 23, 2022
Efficient 2D geometry library for Go.

geometry An efficient 2D geometry library for Go. Features Point, Rect, Line, and Polygon geometry types Operations such as Intersects, Contains, and

Josh Baker 30 Aug 30, 2022
Encoding and decoding GeoJSON <-> Go

go.geojson Go.geojson is a package for encoding and decoding GeoJSON into Go structs. Supports both the json.Marshaler and json.Unmarshaler interfaces

Paul Mach 222 Nov 24, 2022
Package polyline implements a Google Maps Encoding Polyline encoder and decoder.

go-polyline Package polyline implements a Google Maps Encoding Polyline encoder and decoder. Encoding example func ExampleEncodeCoords() { coords :=

Tom Payne 78 Nov 20, 2022
Real-time Geospatial and Geofencing

Tile38 is an open source (MIT licensed), in-memory geolocation data store, spatial index, and realtime geofence. It supports a variety of object types

Josh Baker 8.4k Nov 20, 2022
General purpose library for reading, writing and working with OpenStreetMap data

osm This package is a general purpose library for reading, writing and working with OpenStreetMap data in Go (golang). It has the ability to read OSM

Paul Mach 241 Nov 25, 2022
GhostDB is a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.

GhostDB is designed to speed up dynamic database or API driven websites by storing data in RAM in order to reduce the number of times an external data source such as a database or API must be read. GhostDB provides a very large hash table that is distributed across multiple machines and stores large numbers of key-value pairs within the hash table.

Jake Grogan 729 Nov 13, 2022
A general purpose application and library for aligning text.

align A general purpose application that aligns text The focus of this application is to provide a fast, efficient, and useful tool for aligning text.

John Moore 78 Sep 27, 2022
A general-purpose bot library inspired by Hubot but written in Go. :robot:

Joe Bot ?? A general-purpose bot library inspired by Hubot but written in Go. Joe is a library used to write chat bots in the Go programming language.

Joe Bot 457 Nov 14, 2022
OpenStreetMap find - A simple command line tool to explore osm data

A small tool for playing around with osm databases (those resulting from osm2pgsql). Read to learn how to set up the required databa

Richard Ulmer 121 Nov 14, 2022
Package raw enables reading and writing data at the device driver level for a network interface. MIT Licensed.

raw Package raw enables reading and writing data at the device driver level for a network interface. MIT Licensed. For more information about using ra

Matt Layher 421 Nov 16, 2022