Pure-Go HBase client

Related tags

Network gohbase
Overview

Golang HBase client Build Status codecov.io GoDoc

This is a pure Go client for HBase.

Current status: beta.

Supported Versions

HBase >= 1.0

Installation

go get github.com/tsuna/gohbase

Example Usage

Create a client

client := gohbase.NewClient("localhost")

Insert a cell

// Values maps a ColumnFamily -> Qualifiers -> Values.
values := map[string]map[string][]byte{"cf": map[string][]byte{"a": []byte{0}}}
putRequest, err := hrpc.NewPutStr(context.Background(), "table", "key", values)
rsp, err := client.Put(putRequest)

Get an entire row

getRequest, err := hrpc.NewGetStr(context.Background(), "table", "row")
getRsp, err := client.Get(getRequest)

Get a specific cell

// Perform a get for the cell with key "15", column family "cf" and qualifier "a"
family := map[string][]string{"cf": []string{"a"}}
getRequest, err := hrpc.NewGetStr(context.Background(), "table", "15",
    hrpc.Families(family))
getRsp, err := client.Get(getRequest)

Get a specific cell with a filter

pFilter := filter.NewKeyOnlyFilter(true)
family := map[string][]string{"cf": []string{"a"}}
getRequest, err := hrpc.NewGetStr(context.Background(), "table", "15",
    hrpc.Families(family), hrpc.Filters(pFilter))
getRsp, err := client.Get(getRequest)

Scan with a filter

pFilter := filter.NewPrefixFilter([]byte("7"))
scanRequest, err := hrpc.NewScanStr(context.Background(), "table",
		hrpc.Filters(pFilter))
scanRsp, err := client.Scan(scanRequest)

Contributing

Any help would be appreciated. Please use Github pull requests to send changes for review. Please sign the Contributor License Agreement when you send your first change for review.

License

Copyright © 2015 The GoHBase Authors. All rights reserved. Use of this source code is governed by the Apache License 2.0 that can be found in the COPYING file.

Issues
  • ErrTableNotFound

    ErrTableNotFound

    I am doing a series (read: thousands) of PUT operations on a table that does exist (basically, I am scanning one table, and for each row, doing a small transform and putting into another table). Sometimes, the PUT operation fails and I get the error "table not found" from the client. The rows that this operation fails on are different every time the task is run.

    I have narrowed down the error to the following function:

    // metaLookup checks meta table for the region in which the given row key for the given table is.
    func (c *client) metaLookup(ctx context.Context,
    	table, key []byte) (hrpc.RegionInfo, string, uint16, error) {
    
    	metaKey := createRegionSearchKey(table, key)
    	rpc, err := hrpc.NewGetBefore(ctx, metaTableName, metaKey, hrpc.Families(infoFamily))
    	if err != nil {
    		return nil, "", 0, err
    	}
    
    	resp, err := c.Get(rpc)
    	if err != nil {
    		return nil, "", 0, err
    	}
    	if len(resp.Cells) == 0 {
    		return nil, "", 0, TableNotFound
    	}
    
    	reg, host, port, err := region.ParseRegionInfo(resp)
    	if err != nil {
    		return nil, "", 0, err
    	}
    	if !bytes.Equal(table, fullyQualifiedTable(reg)) {
    		// This would indicate a bug in HBase.
    		return nil, "", 0, fmt.Errorf("wtf: meta returned an entry for the wrong table!"+
    			"  Looked up table=%q key=%q got region=%s", table, key, reg)
    	} else if len(reg.StopKey()) != 0 &&
    		bytes.Compare(key, reg.StopKey()) >= 0 {
    		// This would indicate a hole in the meta table.
    		return nil, "", 0, fmt.Errorf("wtf: meta returned an entry for the wrong region!"+
    			"  Looked up table=%q key=%q got region=%s", table, key, reg)
    	}
    	return reg, host, port, nil
    }
    

    Specifically, the block:

    if len(resp.Cells) == 0 {
    	return nil, "", 0, TableNotFound
    }
    

    It looks like, for some reason, no results are being returned when trying to retrieve the metadata for a region. What situations might cause this issue? I'm beginning to think this is another environment related issue, but wanted to be sure before I go yelling at our ops team. :)

    opened by jonbonazza 16
  • Variable/Optional Timeouts for Zookeeper, Region Lookups, and Region Reads

    Variable/Optional Timeouts for Zookeeper, Region Lookups, and Region Reads

    In our environment we are serving web pages from data provided by the gohbase library. We feel that the hard-coded defaults for the timeouts mentioned in the title are far too long.

    I am providing this pull request to give users of the library the ability to adjust the listed timeouts to suite their needs. If I have coded this correctly, the intention is to be completely backwards compatible with no public API changes, and the same default behavior as one would have experienced before the change. The timeouts listed above are exposed as Options in the same way that other Options are already exposed for the main hbase client.

    The patch passes when calling go test on my workstation. Please review, and let me know if I can do anything to further assist, or if I should change anything to be more in-line with the project's goals.

    Thanks!

    opened by travishegner 10
  • problem with

    problem with "looking up region"

    so it connects to the client. But I came across with a problem: -level=error msg="failed looking up region" backoff=16ms err="context deadline exceeded" key=""row1"" table=""emp"" and -level=error msg="failed looking up region" backoff=16ms err="context deadline exceeded" key="""" table=""hbase:meta""

    any idea how to solve this? thanks in advance

    opened by lanmanlanman 9
  • New Proto Files

    New Proto Files

    I'd like to add Admin.proto to get access to the CompactRegion function but that requires several other files.

    It looks like HBase is preparing for some changes internally. There is no longer a Master.proto file in hbase-protocol, and there is now a hbase-protocol-shaded directory with overlapping proto files.

    Given the changes and possible conflicts, how should I handle adding or updating proto files?

    opened by barnjamin 8
  • `zkRoot` is not set to `defaultZkRoot` for `AdminClient`

    `zkRoot` is not set to `defaultZkRoot` for `AdminClient`

    @jonbonazza Looks like there's a bug introduced by #32 where it was forgotten to set zkRoot in AdminClient https://github.com/tsuna/gohbase/blob/master/admin_client.go#L36 which broke operations on hbase master.

    opened by timoha 8
  • Added Scanner.ScanCh for creating a channel to consume rows as they a re scanned.

    Added Scanner.ScanCh for creating a channel to consume rows as they a re scanned.

    ScanCh is a convenience method for retrieving a buffered channel that can be used to consume rows as they are scanned. As the comment mentions, errors are swallowed and this should only be used when you don't really care about errors.

    As a potential enhancement, ScanCh could take a variadic argument of callback functions for when an error is found. This would allow us to gain the benefit of error handling as well as the convenience of a channel. This is an idiom that I didn't see elsewhere in the codebase, however, so I wanted to check with everyone before implementing it.

    opened by jonbonazza 5
  • Infinite loop looking up regions

    Infinite loop looking up regions

    Hello,

    I've been using this library for a while and I've just updated the latest version. I am having problems with an infinite loop looking up regions. I've tried this with HBase 1.2.4 and 1.3.1 - both on a large cluster and on my local machine with the same results. It works fine simply recompiling with the older version. Any ideas?

    go version go1.8 linux/amd64

    hbase

    create 'doug','c' put 'doug','boo','c:hoo',':-('


    main.go

    package main

    import ( "context" "log" "github.com/tsuna/gohbase" "github.com/tsuna/gohbase/hrpc" )

    func main() {

    client := gohbase.NewClient("localhost")
    getRequest, err := hrpc.NewGetStr(context.Background(), "doug", "boo")
    if err != nil {
    	log.Printf("%s", err.Error())
    }
    getRsp, err := client.Get(getRequest)
    log.Printf("found results %v", getRsp)
    

    }

    opened by dougklein 5
  • metrics: add bytes written & bytes read metric

    metrics: add bytes written & bytes read metric

    This change adds two new counter vector metrics:

    • bytes_read_total - number of bytes read by a regionserver connection
    • bytes_written_total - number of bytes written to a regionserver connection

    These metrics have one label - regionserver - which indicates the address of the connection used.

    opened by devoxel 4
  • go get github.com/tsuna/gohbase/region get error

    go get github.com/tsuna/gohbase/region get error

    github.com/go-logr/logr/funcr

    /root/go/src/github.com/go-logr/logr/funcr/funcr.go:398:16: undefined: strconv.FormatComplex /root/go/src/github.com/go-logr/logr/funcr/funcr.go:400:16: undefined: strconv.FormatComplex /root/go/src/github.com/go-logr/logr/funcr/funcr.go:443:16: undefined: strconv.FormatComplex /root/go/src/github.com/go-logr/logr/funcr/funcr.go:445:16: undefined: strconv.FormatComplex

    opened by wangqiao92 4
  • using recommended lowercase reference to logrus

    using recommended lowercase reference to logrus

    Logrus authors recommend that references to the logrus package use only the lowercase variation. See https://github.com/sirupsen/logrus/blob/master/README.md

    End user code will not build unless that code and all of its dependencies follow only a single logrus case convention.

    opened by netguy204 4
  • Implement Truncate

    Implement Truncate

    There is no protocol command to do this, so it would need to be implemented by combining disable + drop + create. I have implemented this myself in a wrapper, and it works, but there are two issues that would be best if directly integrated into gobhase:

    1. The largest issue is that with the current API exposed by gohbase, you need to specify the column families when you create the table, and if a table already exists, there is no way to retrieve this information. For this to work best, gohbase should at least implement the Describe Table command to retrieve this information programmatically.
    2. Disable, Drop, and Create all take a context.Context, which is great, but there is no easy way (that I am aware of, at least) to have a context that spans all 3 of these commands. For instance, If I pass a context to the Truncate function, it should only call Done when all 3 of Disable, Drop, and Create have completed.

    It would be nice if gohbase provided this integration directly. At the very least, support for Describe Table would be great.

    opened by jonbonazza 4
  • client error

    client error

    ERRO[0030] failed looking up region backoff=16ms err="context deadline exceeded" key=""300605_1658758250_abce"" table=""student""

    DEBU[0001] region was not established, retrying backoff=1.024s err="region.NotServingRegionError: HBase Java exception org.apache.hadoop.hbase.NotServingRegionException:\norg.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1 is not online on hbase-regionserver-1,16020,1658850222232\n\tat org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2942)\n\tat org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1072)\n\tat org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2026)\n\tat org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)\n\tat org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2196)\n\tat org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)\n\tat org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)\n\tat org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)\n\tat java.lang.Thread.run(Thread.java:748)\n" region="RegionInfo{Name: "hbase:meta,,1", ID: 0, Namespace: "hbase", Table: "meta", StartKey: "", StopKey: ""}"

    /etc/hosts done

    hbase shell done

    what is problem?

    opened by XieWeiXie 4
  • Rename/document hrpc.Families to indicate that you can specify column qualifiers

    Rename/document hrpc.Families to indicate that you can specify column qualifiers

    It is not immediately clear when creating a Scan request how one should specify the columns in a row that they are interested in retrieving.

    After seeing some existing code written by others I saw that you can use the tsuna/gohbase/hrpc/query.go:Families option to specify the family as a key and the list of columns (qualifiers) in the family as a string slice in a map to get those families/columns in the result.

    We should add some comments documenting what the Families map keys and values are supposed to represent. Alternatively rename the option to something like FamiliesWithColumns or FamiliesWithQualifiers to indicate that column selection is supported.

    opened by tomasbanet 0
  • stuck in an infinite loop in lookupRegion function

    stuck in an infinite loop in lookupRegion function

    i can't connect our hbase cluster image I was expecting to exit with an error after 10 retries, but I've been looping infinitely in the lookupRegion method and I can't exit. The reason is because the timeout error of the context is overwritten by the sleepAndIncreaseBackoff method, causing the err to become nil, forming an infinite loop image

    opened by LG991103 3
  • gohbase stuck retrying RPCs after region server temporarily down

    gohbase stuck retrying RPCs after region server temporarily down

    We are experiencing an issue where Put RPCs (and possibly any other types of RPCs) for a specific region may end up continuously failing with a ServerError. This seems to be triggered by HBase moving regions after one of the region servers goes down temporarily. The current workaround is to restart the gohbase client process, which indicates that the issue is with gohbase and not HBase itself.

    Our guess is that a region client connection is closed, but it remains associated with one or more region info instances in the gohbase cache. Probably after a disconnection the region client is not properly cleaned up and may end up being reused in RPC operations.

    We have seen the issue on a production HBase cluster that was periodically having sync failures making region servers restart multiple times a day. We have tried reproducing this issue in a lab by bringing region servers down temporarily, but the cluster and client process always recover.

    The gohbase versions we have seen this failed RPCs include: 6acaa09d18ed 348991136365 which are relatively old, but there doesn’t appear to be any changes newer in the repo that could have potentially resolved the issue in question.

    Below is a breakdown of goroutines of version 348991136365 while RPCs are failing. In this setup there are 3 HBase region servers and 8000 workers issuing Put RPCs:

    3 goroutines in the "send" loop (one per server?)
    /go/src/arista/vendor/github.com/tsuna/gohbase/region/new.go:69 +0x c.Dial go c.processRPCs()
    /go/src/arista/vendor/github.com/tsuna/gohbase/region/client.go:364 +0x c.processRPCs select // wait for the next batch
    
    3 goroutines in the "receive" loop (one per server?)
    /go/src/arista/vendor/github.com/tsuna/gohbase/region/new.go:71 +0x c.Dial go c.receiveRPCs()
    /go/src/arista/vendor/github.com/tsuna/gohbase/region/client.go:444 +0x c.receiveRPCs c.receive()
    /go/src/arista/vendor/github.com/tsuna/gohbase/region/client.go:464 +0x c.receive c.readFully(sz[:])
    /go/src/arista/vendor/github.com/tsuna/gohbase/region/client.go:580 +0x c.readFully io.ReadFull
    
    3 "Put" goroutines queueing an RPC
    /go/src/arista/vendor/github.com/tsuna/gohbase/client.go:237 +0x c.Put c.mutate(p)
    /go/src/arista/vendor/github.com/tsuna/gohbase/client.go:264 +0x c.mutate c.SendRPC(m)
    /go/src/arista/vendor/github.com/tsuna/gohbase/rpc.go:77 +0x  c.SendRPC c.sendRPCToRegion(rpc, reg) <- ALL ON SAME REGION interface
    /go/src/arista/vendor/github.com/tsuna/gohbase/rpc.go:141 +0x c.sendRPCToRegion sendBlocking(client, rpc)
    /go/src/arista/vendor/github.com/tsuna/gohbase/rpc.go:111 +0x sendBlocking rc.QueueRPC(rpc)
    /go/src/arista/vendor/github.com/tsuna/gohbase/region/client.go:206 +0x c.QueueRPC <-c.done: returnResult(rpc, nil, ErrClientClosed)
    
    1 "Put" goroutines starting to mark region client as "down"
    /go/src/arista/vendor/github.com/tsuna/gohbase/client.go:237 +0x c.Put c.mutate(p)
    /go/src/arista/vendor/github.com/tsuna/gohbase/client.go:264 +0x c.mutate c.SendRPC(m)
    /go/src/arista/vendor/github.com/tsuna/gohbase/rpc.go:77 +0x  c.SendRPC c.sendRPCToRegion(rpc, reg) <- ALL ON SAME REGION interface
    /go/src/arista/vendor/github.com/tsuna/gohbase/rpc.go:167 +0x c.sendRPCToRegion ServerError: c.clientDown(client)
    /go/src/arista/vendor/github.com/tsuna/gohbase/rpc.go:177 +0x c.clientDown(client hrpc.RegionClient)
    
    7995 "Put" goroutines waiting (contending) to mark a server client as "down"
    /go/src/arista/vendor/github.com/tsuna/gohbase/client.go:237 +0x c.Put c.mutate(p)
    /go/src/arista/vendor/github.com/tsuna/gohbase/client.go:264 +0x c.mutate c.SendRPC(m)
    /go/src/arista/vendor/github.com/tsuna/gohbase/rpc.go:77 +0x  c.SendRPC c.sendRPCToRegion(rpc, reg) <- ALL ON SAME REGION interface
    /go/src/arista/vendor/github.com/tsuna/gohbase/rpc.go:167 +0x c.sendRPCToRegion ServerError: c.clientDown(client)
    /go/src/arista/vendor/github.com/tsuna/gohbase/caches.go:84 +0x rcc.clientDown rcc.m.Lock()
    
    1 "Put" goroutines finishing marking a server client as "down"
    /go/src/arista/vendor/github.com/tsuna/gohbase/client.go:237 +0x c.Put c.mutate(p)
    /go/src/arista/vendor/github.com/tsuna/gohbase/client.go:264 +0x c.mutate c.SendRPC(m)
    /go/src/arista/vendor/github.com/tsuna/gohbase/rpc.go:77 +0x  c.SendRPC c.sendRPCToRegion(rpc, reg) <- ALL ON SAME REGION interface
    /go/src/arista/vendor/github.com/tsuna/gohbase/rpc.go:167 +0x c.sendRPCToRegion ServerError: c.clientDown(client)
    /go/src/arista/vendor/github.com/tsuna/gohbase/caches.go:87 +0x rcc.clientDown rcc.m.Unlock()
    

    Some observations from the stacks:

    • The Put calls have a context timeout of 10 minutes
    • No goroutine is stuck greater than 2 minutes indicating that there is no deadlock, but rather that gohbase is retrying in a loop
    • All 8000 calls to sendRPCToRegion are referencing the same region info instance
    • Al calls to c.clients.clientDown are referencing the same region client instance

    goroutine stacks attached: gohbase-goroutines.txt

    opened by rodicus 2
  • avoid cache avalanche

    avoid cache avalanche

    At the beginning of client start, there is no table region info in cache. If there are many go routine to access a table at same time, they will miss cache and send request package to get table region info. In this case, too many duplicated requests should be avoid.

    opened by xpineal 1
Owner
Benoit Sigoure
Benoit Sigoure
🚀 gnet is a high-performance, lightweight, non-blocking, event-driven networking framework written in pure Go./ gnet 是一个高性能、轻量级、非阻塞的事件驱动 Go 网络框架。

English | ???? 中文 ?? Introduction gnet is an event-driven networking framework that is fast and lightweight. It makes direct epoll and kqueue syscalls

Andy Pan 6.8k Aug 4, 2022
Pure-Go library for cross-platform local peer discovery using UDP multicast :woman: :repeat: :woman:

peerdiscovery Pure-go library for cross-platform thread-safe local peer discovery using UDP multicast. I needed to use peer discovery for croc and eve

Zack 552 Aug 4, 2022
A QUIC implementation in pure go

A QUIC implementation in pure Go quic-go is an implementation of the QUIC protocol in Go. It implements the IETF QUIC draft-29 and draft-32. Version c

Lucas Clemente 7k Aug 1, 2022
Pure Go implementation of the WebRTC API

Pion WebRTC A pure Go implementation of the WebRTC API New Release Pion WebRTC v3.0.0 has been released! See the release notes to learn about new feat

Pion 9.7k Aug 4, 2022
A Windows named pipe implementation written in pure Go.

npipe Package npipe provides a pure Go wrapper around Windows named pipes. Windows named pipe documentation: http://msdn.microsoft.com/en-us/library/w

Nate Finch 250 Aug 3, 2022
Pure Go implementation of the WebRTC API

Pure Go implementation of the WebRTC API

Pion 9.7k Aug 8, 2022
Distributed RTC System by pure Go and Flutter

ION is a distributed real-time communication system, the goal is to chat anydevice, anytime, anywhere! Distributed Real-time Communication System ION

Pion 3.6k Aug 3, 2022
Pure Go Brotli encoder and decoder

This package is a brotli compressor and decompressor implemented in Go. It was translated from the reference implementation (https://github.com/google

Andy Balholm 415 Aug 4, 2022
Snugger is a light weight but fast network recon scanner that is written from pure golang

Snugger is a light weight but fast network recon scanner that is written from pure golang. with this scann you can ARP your network, port scan hosts and host lists, as well as scan for BSSId

RE43P3R 2 May 19, 2022
Prisma Client Go is an auto-generated and fully type-safe database client

Prisma Client Go Typesafe database access for Go Quickstart • Website • Docs • API reference • Blog • Slack • Twitter Prisma Client Go is an auto-gene

Prisma 1.3k Aug 12, 2022
The Dual-Stack Dynamic DNS client, the world's first dynamic DNS client built for IPv6.

dsddns DsDDNS is the Dual-Stack Dynamic DNS client. A dynamic DNS client keeps your DNS records in sync with the IP addresses associated with your hom

Ryan Young 11 Jul 8, 2022
Go Substrate RPC Client (GSRPC)Go Substrate RPC Client (GSRPC)

Go Substrate RPC Client (GSRPC) Substrate RPC client in Go. It provides APIs and types around Polkadot and any Substrate-based chain RPC calls. This c

Chino Chang 1 Nov 11, 2021
Server and client implementation of the grpc go libraries to perform unary, client streaming, server streaming and full duplex RPCs from gRPC go introduction

Description This is an implementation of a gRPC client and server that provides route guidance from gRPC Basics: Go tutorial. It demonstrates how to u

Joram Wambugu 0 Nov 24, 2021
Godaddy-domains-client-go - Godaddy domains api Client golang - Write automaticly from swagger codegen

Go API client for swagger Overview This API client was generated by the swagger-codegen project. By using the swagger-spec from a remote server, you c

Mickael Stanislas 0 Jan 9, 2022
Tailscale-client-go - A client implementation for the Tailscale HTTP API

tailscale-client-go A client implementation for the Tailscale HTTP API Example p

David Bond 10 Jun 29, 2022
Comunicación de envios de archivos entres cliente-servidor, client-client.

Client - Server - Client Estes es un proyecto simple de comunicacion de envios de archivos del cliente al servidor y viceversamente, y de cliente a cl

Melvin RB 1 Jul 16, 2022
CoAP Client/Server implementing RFC 7252 for the Go Language

Canopus Canopus is a client/server implementation of the Constrained Application Protocol (CoAP) Updates 25.11.2016 I've added basic dTLS Support base

Zubair Hamed 148 Jan 23, 2022
FTP client package for Go

goftp A FTP client package for Go Install go get -u github.com/jlaffaye/ftp Documentation https://pkg.go.dev/github.com/jlaffaye/ftp?tab=doc Example

Julien Laffaye 970 Aug 4, 2022
Go PowerDNS 4.x API Client

PowerDNS 4.x API bindings for Golang This community project provides bindings for the currently latest version of PowerDNS Authoritative Server. Featu

Johannes Eiglsperger 59 Jul 26, 2022