General purpose library for reading, writing and working with OpenStreetMap data

Overview

osm CI Go Report Card Go Reference

This package is a general purpose library for reading, writing and working with OpenStreetMap data in Go (golang). It has the ability to read OSM XML and PBF data formats available at planet.osm.org or via the v0.6 API.

Made available by the package are the following types:

  • Node
  • Way
  • Relation
  • Changeset
  • Note
  • User

And the following “container” types:

List of sub-package utilities

  • annotate - adds lon/lat, version, changeset and orientation data to way and relation members
  • osmapi - supports all the v0.6 read/data endpoints
  • osmgeojson - OSM to GeoJSON conversion compatible with osmtogeojson
  • osmpbf - stream processing of *.osm.pbf files
  • osmxml - stream processing of *.osm xml files
  • replication - fetch replication state and change files

Concepts

This package refers to the core OSM data types as Objects. The Node, Way, Relation, Changeset, Note and User types implement the osm.Object interface and can be referenced using the osm.ObjectID type. As a result it is possible to have a slice of []osm.Object that contains nodes, changesets and users.

Individual versions of the core OSM Map Data types are referred to as Elements and the set of versions for a give Node, Way or Relation is referred to as a Feature. For example, an osm.ElementID could refer to "Node with id 10 and version 3" and the osm.FeatureID would refer to "all versions of node with id 10." Put another way, features represent a road and how it's changed over time and an element is a specific version of that feature.

A number of helper methods are provided for dealing with features and elements. The idea is to make it easy to work with a Way and its member nodes, for example.

Scanning large data files

For small data it is possible to use the encoding/xml package in the Go standard library to marshal/unmarshal the data. This is typically done using the osm.OSM or osm.Change "container" structs.

For large data the package defines the Scanner interface implemented in both the osmxml and osmpbf sub-packages.

type osm.Scanner interface {
	Scan() bool
	Object() osm.Object
	Err() error
	Close() error
}

This interface is designed to mimic the bufio.Scanner interface found in the Go standard library.

Example usage:

f, err := os.Open("./delaware-latest.osm.pbf")
if err != nil {
	panic(err)
}
defer f.Close()

scanner := osmpbf.New(context.Background(), f, 3)
defer scanner.Close()

for scanner.Scan() {
	o := scanner.Object()
	// do something
}

scanErr := scanner.Err()
if scanErr != nil {
	panic(scanErr)
}

Note: Scanners are not safe for parallel use. One should feed the objects into a channel and have workers read from that.

Comments
  • Add support for node locations on ways

    Add support for node locations on ways

    Hi,

    I would like to add support for ways with node locations on them, an extension of the OSM PBF format put forward by @joto.

    This is the approach I have taken. I started by taking over the latest version of osmformat.proto, which has delta encoded packed sint64 fields numbered 9 (lat) and 10 (lon) in addition to the field 8 (refs) containing the IDs of all the nodes in the way. Then, after running protoc and taking a look at the current code for way nodes and node coordinates, I added similar logic for inserting the way node coordinates (if present) into the already existing WayNode.Lat and WayNode.Lon fields. Testing with a file generated by osmium add-locations-to-ways, this seems to give the desired result and processing regular OSM PBF files should be unaffected.

    Although it seems to me this pull request is OK, I would appreciate if you could look into it, especially since I don't yet feel very confident working with protocol buffers. Here are a few additional remarks:

    • When taking over the latest osmformat.proto, I have taken care not to overwrite the existing changes related to gogoprotobuf and to eliminate conversions from []byte to string.
    • Before all of the steps I mentioned so far, I have just run the protoc command from generate.go with no other changes. This already modified the generated Go files, probably because I am using the most recent version. Since I don't know what version was used previously, I stuck with these changes and kept them as the first commit.
    • The most important consequence of running protoc was the change of the package name from osmpbf to OSMPBF in the generated Go files. It did not seem a good idea to manually change the generated files, so I've chosen to replace the import paths to this internal package with the uppercase verision, as can be seen in the second commit.

    I'd love to hear your feedback on this!

    opened by flowrean 9
  • Huge Memory Usage

    Huge Memory Usage

    Hi there. First off, thanks for this great library.

    When trying to scan a 700Mb *osm.pbf file, the mem usage spikes to over 145Gb with just a single go routine. I know this because I am trying this task in an AWS CodeBuild project with max memory allowed (145 Gb) and it still gets killed.

    This is surprising considering that the Osmosis CLI tool can accomplish the same task (scanning the 700Mb pbf file and filtering by a bounding box) without breaching 7Gb of memory.

    Why does this tool take over 20 times more memory than Osmosis, and is there anything we can do to improve this? I'm willing to help if it's a big task.

    Thanks.

    opened by tyliggity 7
  • Unmarshaling of Overpass JSON fails on version field

    Unmarshaling of Overpass JSON fails on version field

    The version information is encoded as JSON number instead of a string by Overpass API. This causes unmarshaling it to fail with error json: cannot unmarshal number into Go struct field .version of type string

    Take, for example, the minimal JSON output of overpass (http://overpass-api.de/api/interpreter?data=[out:json];out;), e.g.

    {
      "version": 0.6,
      "generator": "Overpass API 0.7.59 e21c39fe",
      "osm3s": {
        "timestamp_osm_base": "2022-11-17T17:14:39Z",
        "copyright": "The data included in this document is from www.openstreetmap.org. The data is made available under ODbL."
      },
      "elements": []
    }
    

    Minimum viable example failure: https://go.dev/play/p/n7kBY3CCbHC

    package main
    
    import (
    	"encoding/json"
    	"fmt"
    
    	"github.com/paulmach/osm"
    )
    
    func main() {
    	var overpass osm.OSM
    	buf := []byte(`{
      "version": 0.6,
      "generator": "Overpass API 0.7.59 e21c39fe",
      "osm3s": {
        "timestamp_osm_base": "2022-11-17T17:14:39Z",
        "copyright": "The data included in this document is from www.openstreetmap.org. The data is made available under ODbL."
      },
      "elements": []
    }`)
    	err := json.Unmarshal(buf, &overpass)
    	if err != nil {
    
    		fmt.Println(err)
    	}
    }
    
    opened by fawick 5
  • [Feature Request] Provide timestamp to replication sequence number function

    [Feature Request] Provide timestamp to replication sequence number function

    This enables us to reimplement this script in go. Your library is a great candidate to use when working with osm data.

    https://github.com/openstreetmap/osm2pgsql/blob/master/scripts/osm2pgsql-replication

    opened by hypervtechnics 5
  • Feature request: Specify which OSM entities should be read from the file

    Feature request: Specify which OSM entities should be read from the file

    Hello and thank you for this library.

    In libosmium, osmium::io::Reader can be initialized with osmium::osm_entities::bits to specify which OSM entities should be parsed and passed to the handler (docs, example). Specifying only the entities you want to handle does significantly improve read performance in my experience with libosmium.

    I believe it can be done without changing the current behavior by adding new initialization function, perhaps osmpbf.NewWithEntities.

    I appreciate your consideration.

    opened by invisiblefunnel 5
  • Tags.Find() should return nil not

    Tags.Find() should return nil not ""

    OSM nodes can contain tags which specify only a key but no value. For example "building=" is common. Tags.Find() should return [nil] if no key is found, since returning "" could mean that either the tag is missing or that it is present but empty. It is common to want to catch even empty tags.

    opened by morandd 4
  • gogo/protobuf security issue

    gogo/protobuf security issue

    Thanks for a great project.

    There is a security issue detected by Dependabot https://github.com/gogo/protobuf/issues/752.

    All that is needed is to upgrade the gogo version to the latest one (v1.3.2 I believe) where this is fixed.

    As a side note, the gogo project is not really maintained anymore. Has anyone tested the performance of using the standard proto package?

    opened by iulian-gm 4
  • osmpbf: support

    osmpbf: support "sparse" dense nodes

    Not all pbf files will have all the timestamp, username, changeset, etc. data so we should be okay with skipping that. No need to stop and return an error.

    https://github.com/paulmach/osm/issues/31

    @dmholtz does this work on your file?

    opened by paulmach 4
  • How to handle dense nodes?

    How to handle dense nodes?

    Decoding OSM PBF files, which contain 'dense nodes' fails:

    panic: dense node does not include all required fields
    
    goroutine 1 [running]:
    main.main()
            /Users/[user]/repos/[repo]/cmd/importer/main.go:76 +0x468
    exit status 2
    

    The error occurs within the example code provided in the README.md of the osmpbf package.

    The error does not occur with OSM PBF file, which do not contain 'dense nodes' like Antarctica from Geofabrik

    Is this a bug or have I missed anything?

    After some research, it seems to me that another library just ignores the missing fields and the decoding suceeds. Thus, the issue seems to be closely related to my OSM PBF file (which I can unfortunately not share due to its size). It would be interesting to hear if your implementation supports a less strict mode, which also just ignores these fields.

    opened by dmholtz 4
  • Question: Why output and serializer chanel

    Question: Why output and serializer chanel

    Is there a specific reason to have in osmpbf part of the project n output chanels and feeding them in order to a serializer chanel ? (decode.go function line number

    The upside is it is making testing much easier since we get predictable outputs, however in my case I do not see not a reason that my application needs a special ordering of the elements : Do I miss something important why one should process objects in order ? (In other words: why doesn't the deocoder directly push to serializer)

    opened by oflebbe 3
  • protobuf: port to google protobuf

    protobuf: port to google protobuf

    I replaced gogo protobuf by the plain google protobuf implementation.

    I had to rewrite some portions regarding nullability. Hope I got it right, there might be better way to accomplish.

    Overall I was surprised that the CPU performance drop is below 3% , I was expecting much more. The number of allocations is much higher for some benchmarks, since I had to alloc temporary objects because of nullability changes above. The only one I am concerned is the change in memory for BenchmarkLondon_relations-16

    BenchmarkDiff_Marshal-16                     33708352       33497339       -0.63%
    BenchmarkDiff_Unmarshal-16                   66501681       65426347       -1.62%
    BenchmarkElementID_Sort-16                   648905         653604         +0.72%
    BenchmarkFeatureIDs_Sort-16                  636472         631671         -0.75%
    BenchmarkChange_MarshalXML-16                1790183        1744972        -2.53%
    BenchmarkChangeSet_Marshal-16                79831          77896          -2.42%
    BenchmarkUnmarshalWays-16                    5077           6401           +26.08%
    BenchmarkUnmarshalRelations-16               5508           6938           +25.96%
    BenchmarkChangeset_Marshal_gzip-16           450378         456373         +1.33%
    BenchmarkChangeset_UnmarshalXML-16           3468850        3504131        +1.02%
    BenchmarkUnmarshalChangeset-16               73306          81317          +10.93%
    BenchmarkUnmarshalChangeset_gzip-16          137123         144377         +5.29%
    BenchmarkRelations-16                        638028         633393         -0.73%
    BenchmarkWay-16                              14600          14479          -0.83%
    BenchmarkWays-16                             23030          22524          -2.20%
    BenchmarkConvert-16                          1211337        1202652        -0.72%
    BenchmarkConvertAnnotated-16                 947576         940853         -0.71%
    BenchmarkConvert_NoID-16                     1084547        1063003        -1.99%
    BenchmarkConvert_NoMeta-16                   996524         991362         -0.52%
    BenchmarkConvert_NoRelationMembership-16     1157384        1148647        -0.75%
    BenchmarkConvert_NoIDsMetaMembership-16      747993         748889         +0.12%
    BenchmarkLondon-16                           257786472      262947900      +2.00%
    BenchmarkLondon_withFiltersTrue-16           266369759      269555246      +1.20%
    BenchmarkLondon_withFiltersFalse-16          208942287      206641137      -1.10%
    BenchmarkLondon_nodes-16                     189334304      192258210      +1.54%
    BenchmarkLondon_ways-16                      147202951      150169294      +2.02%
    BenchmarkLondon_relations-16                 90644410       93034313       +2.64%
    BenchmarkAndorra-16                          3074774010     3111769317     +1.20%
    
    benchmark                                    old allocs     new allocs     delta
    BenchmarkDiff_Marshal-16                     129484         129484         +0.00%
    BenchmarkDiff_Unmarshal-16                   422916         422916         +0.00%
    BenchmarkElementID_Sort-16                   1              1              +0.00%
    BenchmarkFeatureIDs_Sort-16                  1              1              +0.00%
    BenchmarkChange_MarshalXML-16                4385           4385           +0.00%
    BenchmarkChangeSet_Marshal-16                291            242            -16.84%
    BenchmarkUnmarshalWays-16                    74             97             +31.08%
    BenchmarkUnmarshalRelations-16               81             105            +29.63%
    BenchmarkChangeset_Marshal_gzip-16           315            265            -15.87%
    BenchmarkChangeset_UnmarshalXML-16           23326          23326          +0.00%
    BenchmarkUnmarshalChangeset-16               346            617            +78.32%
    BenchmarkUnmarshalChangeset_gzip-16          390            660            +69.23%
    BenchmarkRelations-16                        1417           1417           +0.00%
    BenchmarkWay-16                              126            126            +0.00%
    BenchmarkWays-16                             116            116            +0.00%
    BenchmarkConvert-16                          10114          10114          +0.00%
    BenchmarkConvertAnnotated-16                 10107          10107          +0.00%
    BenchmarkConvert_NoID-16                     8501           8501           +0.00%
    BenchmarkConvert_NoMeta-16                   6898           6898           +0.00%
    BenchmarkConvert_NoRelationMembership-16     9531           9531           +0.00%
    BenchmarkConvert_NoIDsMetaMembership-16      4702           4703           +0.02%
    BenchmarkLondon-16                           4805957        4811479        +0.11%
    BenchmarkLondon_withFiltersTrue-16           4805954        4811469        +0.11%
    BenchmarkLondon_withFiltersFalse-16          1362152        1367677        +0.41%
    BenchmarkLondon_nodes-16                     3392981        3398512        +0.16%
    BenchmarkLondon_ways-16                      1793525        1799043        +0.31%
    BenchmarkLondon_relations-16                 457585         463104         +1.21%
    BenchmarkAndorra-16                          13038438       13038444       +0.00%
    
    benchmark                                    old bytes     new bytes     delta
    BenchmarkDiff_Marshal-16                     11232715      11232715      +0.00%
    BenchmarkDiff_Unmarshal-16                   15975454      15975430      -0.00%
    BenchmarkElementID_Sort-16                   24            24            +0.00%
    BenchmarkFeatureIDs_Sort-16                  24            24            +0.00%
    BenchmarkChange_MarshalXML-16                593113        593113        +0.00%
    BenchmarkChangeSet_Marshal-16                68687         37545         -45.34%
    BenchmarkUnmarshalWays-16                    6888          6984          +1.39%
    BenchmarkUnmarshalRelations-16               7248          7536          +3.97%
    BenchmarkChangeset_Marshal_gzip-16           891254        860106        -3.49%
    BenchmarkChangeset_UnmarshalXML-16           1131472       1131467       -0.00%
    BenchmarkUnmarshalChangeset-16               148593        173889        +17.02%
    BenchmarkUnmarshalChangeset_gzip-16          225217        250481        +11.22%
    BenchmarkRelations-16                        245465        245441        -0.01%
    BenchmarkWay-16                              9622          9622          +0.00%
    BenchmarkWays-16                             11640         11639         -0.01%
    BenchmarkConvert-16                          913288        913285        -0.00%
    BenchmarkConvertAnnotated-16                 830581        830577        -0.00%
    BenchmarkConvert_NoID-16                     890772        890778        +0.00%
    BenchmarkConvert_NoMeta-16                   702989        703007        +0.00%
    BenchmarkConvert_NoRelationMembership-16     895883        895846        -0.00%
    BenchmarkConvert_NoIDsMetaMembership-16      663290        663296        +0.00%
    BenchmarkLondon-16                           918437913     938188674     +2.15%
    BenchmarkLondon_withFiltersTrue-16           918437519     938187697     +2.15%
    BenchmarkLondon_withFiltersFalse-16          360101121     379852203     +5.48%
    BenchmarkLondon_nodes-16                     613038341     632790024     +3.22%
    BenchmarkLondon_ways-16                      432007871     451758126     +4.57%
    BenchmarkLondon_relations-16                 178275503     198026162     +11.08%
    BenchmarkAndorra-16                          471254696     471255184     +0.00%```
    opened by ghost 3
Releases(v0.7.1)
  • v0.7.1(Nov 29, 2022)

    What's Changed

    • osm: add Tags.FindTag and Tags.HasTag methods by @paulmach in https://github.com/paulmach/osm/pull/45
    • osm: support version as json number or string by @paulmach in https://github.com/paulmach/osm/pull/46

    Full Changelog: https://github.com/paulmach/osm/compare/v0.7.0...v0.7.1

    Source code(tar.gz)
    Source code(zip)
  • v0.7.0(Aug 17, 2022)

    What's Changed

    • remove support for marshalling core types into custom binary format by @paulmach in https://github.com/paulmach/osm/pull/40

    Full Changelog: https://github.com/paulmach/osm/compare/v0.6.0...v0.7.0

    Source code(tar.gz)
    Source code(zip)
  • v0.6.0(Aug 16, 2022)

    What's Changed

    • json: ability to unmarshal osmjson by @paulmach in https://github.com/paulmach/osm/pull/39
    • json: add support for external json implementations by @paulmach in https://github.com/paulmach/osm/pull/39

    Full Changelog: https://github.com/paulmach/osm/compare/v0.5.0...v0.6.0

    Source code(tar.gz)
    Source code(zip)
  • v0.5.0(Jun 7, 2022)

    What's Changed

    • replication: ability to get changeset state by @paulmach in https://github.com/paulmach/osm/pull/37
    • replication: search for state/sequence number by timestamp by @paulmach in https://github.com/paulmach/osm/pull/38

    Full Changelog: https://github.com/paulmach/osm/compare/v0.4.0...v0.5.0

    Source code(tar.gz)
    Source code(zip)
  • v0.4.0(May 26, 2022)

    What's Changed

    • protobuf: port to google protobuf by @OlafFlebbeBosch in https://github.com/paulmach/osm/pull/36

    New Contributors

    • @OlafFlebbeBosch made their first contribution in https://github.com/paulmach/osm/pull/36

    Full Changelog: https://github.com/paulmach/osm/compare/v0.3.0...v0.4.0

    Source code(tar.gz)
    Source code(zip)
  • v0.3.0(Apr 21, 2022)

    What's Changed

    • osmpbf: preallocation node tags array by @paulmach in https://github.com/paulmach/osm/pull/33
    • osmpbf: support "sparse" dense nodes by @paulmach in https://github.com/paulmach/osm/pull/32
    • osmpbf: add filter functions by @paulmach in https://github.com/paulmach/osm/pull/30

    Full Changelog: https://github.com/paulmach/osm/compare/v0.2.2...v0.3.0

    Source code(tar.gz)
    Source code(zip)
  • v0.2.2(Apr 27, 2021)

  • v0.2.1(Feb 4, 2021)

    • Reduces memory usage when decoding a OSM PBF file https://github.com/paulmach/osm/pull/22, thank you @oflebbe
    • Fix some more typos https://github.com/paulmach/osm/pull/23, thank you @meyermarcel
    Source code(tar.gz)
    Source code(zip)
  • v0.2.0(Jan 9, 2021)

    4 changes:

    • ability to efficiently skip types when decoding osmpbf https://github.com/paulmach/osm/pull/18
    • the above also added protoscan for osmpbf decoding for a 10%ish performance improvement
    • use cgo/czlib to decode protobufs (if cgo enabled), 20% faster on benchmarks https://github.com/paulmach/osm/pull/19
    • deprecated node/ways/relations marshaling into this packages custom binary format. It could be improved and should be versioned separately. https://github.com/paulmach/osm/commit/8fcda5dc49b4767df63eccb5a25f3e63d5b17f4d
    Source code(tar.gz)
    Source code(zip)
  • v0.1.1(Jun 22, 2020)

  • v0.1.0(Oct 2, 2019)

Owner
Paul Mach
Paul Mach
A go library for reading and creating ISO9660 images

iso9660 A package for reading and creating ISO9660, forked from https://github.com/kdomanski/iso9660. Requires Go 1.13 or newer. Joliet and Rock Ridge

Karpelès Lab Inc. 8 Mar 4, 2021
The android-go project provides a platform for writing native Android apps in Go programming language.

android-go The android-go project aims to provide a platform (namely an SDK) for writing native Android apps in Go programming language. All things he

Max Kupriianov 984 Jan 5, 2023
A toy ray tracer to practice writing go.

pbrt-go A toy ray tracer written to practice writing go. Slowly working towards an implementation based on the excellent 3rd edition of PBRT, because

Dan Bolan 0 Oct 19, 2021
Publish Your GIS Data(Vector Data) to PostGIS and Geoserver

GISManager Publish Your GIS Data(Vector Data) to PostGIS and Geoserver How to install: go get -v github.com/hishamkaram/gismanager Usage: testdata fol

Hisham waleed karam 48 Sep 26, 2022
A repository for plotting and visualizing data

Gonum Plot gonum/plot is the new, official fork of code.google.com/p/plotinum. It provides an API for building and drawing plots in Go. Note that this

null 2.3k Dec 27, 2022
An API which allows you to upload an image and responds with the same image, stripped of EXIF data

strip-metadata This is an API which allows you to upload an image and responds with the same image, stripped of EXIF data. How to run You need to have

Cristina Simionescu 0 Nov 25, 2021
Decode embedded EXIF meta data from image files.

goexif Provides decoding of basic exif and tiff encoded data. Still in alpha - no guarantees. Suggestions and pull requests are welcome. Functionality

Robert Carlsen 545 Dec 17, 2022
Efficient moving window for high-speed data processing.

Moving Window Data Structure Copyright (c) 2012. Jake Brukhman. ([email protected]). All rights reserved. See the LICENSE file for BSD-style license. I

Jake Brukhman 32 Sep 4, 2022
This is old and unmaintained code, ignore it. starfish is a simple, SDL based, 2D graphics and user input library for Go. If you intend to work on it, please fork from the 'devel' branch, not 'master'. Current release: 0.12.0

What is starfish? What starfish is: starfish is a simple 2D graphics and user input library for Go built on SDL. What starfish is not: While it is bui

Gary Talent 12 Jun 4, 2019
Image processing library and rendering toolkit for Go.

blend Image processing library and rendering toolkit for Go. (WIP) Installation: This library is compatible with Go1. go get github.com/phrozen/blend

Guillermo Estrada 61 Nov 11, 2022
A lightning fast image processing and resizing library for Go

govips A lightning fast image processing and resizing library for Go This package wraps the core functionality of libvips image processing library by

David Byttow 816 Jan 8, 2023
A library to read, write, and transform Stereolithography (.stl) files in Go.

stl A library to read, write, and transform Stereolithography (.stl) files in Go. It is used in the command line STL manipulation tool stltool. Featur

Hagen Schendel 65 Sep 26, 2022
Go cross-platform glfw library for creating an OpenGL context and receiving events.

glfw Package glfw experimentally provides a glfw-like API with desktop (via glfw) and browser (via HTML5 canvas) backends. It is used for creating a G

null 77 Sep 27, 2022
go-pix is a Go library for generating Pix transactions using Copy and Paste or QR codes. 💳 💰

go-pix go-pix is a Go library for generating Pix transactions using Copy and Paste or QR codes.

Jonnas Fonini 63 Sep 12, 2022
JPEG-MPO Decoder / Converter Library and CLI Tool

MPO Decoder Library Simple Go JPEG MPO (Multi Picture Object) Decoder - Library and CLI Tool The library and CLI tool contain the ability to convert M

Jesse Donat 10 Sep 26, 2022
geoserver is a Go library for manipulating a GeoServer instance via the GeoServer REST API.

Geoserver geoserver Is a Go Package For Manipulating a GeoServer Instance via the GeoServer REST API. How to install: go get -v gopkg.in/hishamkaram/g

Hisham waleed karam 78 Dec 22, 2022
S2 geometry library in Go

Overview S2 is a library for spherical geometry that aims to have the same robustness, flexibility, and performance as the best planar geometry librar

Go 1.5k Jan 8, 2023
Go package for fast high-level image processing powered by libvips C library

bimg Small Go package for fast high-level image processing using libvips via C bindings, providing a simple programmatic API. bimg was designed to be

Tom 2.1k Jan 2, 2023
Go bindings to OpenGL Utility Library

GLU This package offers minimal bindings for GLU functions. Usage go get github.com/go-gl-legacy/glu License Copyright 2012 The go-gl Authors. All ri

go-gl legacy 21 Aug 18, 2018