a lightweight distributed transaction management service, support xa tcc saga

Overview

license Build Status Coverage Status Go Report Card Go Reference

English

GO语言分布式事务管理服务

DTM是首款golang的开源分布式事务管理器,优雅的解决了幂等、空补偿、悬挂等分布式事务难题。在微服务架构中,提供了高性能和简单易用的分布式事务解决方案。

受邀参加中国数据库大会分享多语言环境下分布式事务实践

亮点

  • 稳定可靠
    • 经过生产环境考验,单元测试覆盖率90%以上
  • 使用简单
    • 接口简单,开发者不再担心悬挂、空补偿、幂等各类问题,框架层代为处理
  • 跨语言
    • 可适合多语言栈的公司使用。协议支持http。方便go、python、php、nodejs、ruby各类语言使用。
  • 社区活跃
    • 任何问题都快速响应
  • 易部署、易扩展
    • 仅依赖mysql,部署简单,易集群化,易水平扩展
  • 多种分布式事务协议支持
    • TCC: Try-Confirm-Cancel
    • SAGA:
    • 可靠消息
    • XA

文档与介绍(更新中)

与其他框架对比

目前开源的分布式事务框架,有阿里的SEATA、华为的ServiceComb-Pack,京东的shardingsphere,以及himly,tcc-transaction,ByteTCC等等,其中以seata应用最为广泛。

这些框架基本都是Java语言,非Java语言的,暂未看到有成熟的框架。

下面将dtm和seata的主要特性做一下对比:

特性 DTM SEATA 备注
支持语言 Golang、python、node及其他 Java dtm可轻松接入一门新语言
异常处理 子事务屏障技术 手动处理 dtm解决了幂等、悬挂、空补偿
TCC事务
XA事务
AT事务 AT事务与XA事务类似
SAGA事务 简单模式 状态机复杂模式 dtm的状态机模式在规划中
事务消息 dtm提供类似rocketmq的事务消息
通信协议 HTTP dubbo等协议,无HTTP dtm后续将支持grpc类协议
star数量 github stars github stars dtm从20210604发布0.1,发展快

从上面对比的特性来看,如果您的语言栈包含了Java之外的语言,那么dtm是您的首选。如果您的语言栈是Java,您也可以选择接入dtm,使用子事务屏障技术,简化您的业务编写。

快速开始

安装

git clone github.com/yedf/dtm

dtm依赖于mysql

配置mysql:

cp conf.sample.yml conf.yml # 修改conf.yml

启动并运行saga示例

go run app/main.go saga

开始使用

使用

  // 具体业务微服务地址
  const qsBusi = "http://localhost:8081/api/busi_saga"
	req := &gin.H{"amount": 30} // 微服务的载荷
	// DtmServer为DTM服务的地址,是一个url
	saga := dtmcli.NewSaga("http://localhost:8080/api/dtmsvr").
		// 添加一个TransOut的子事务,正向操作为url: qsBusi+"/TransOut", 补偿操作为url: qsBusi+"/TransOutCompensate"
		Add(qsBusi+"/TransOut", qsBusi+"/TransOutCompensate", req).
		// 添加一个TransIn的子事务,正向操作为url: qsBusi+"/TransOut", 补偿操作为url: qsBusi+"/TransInCompensate"
		Add(qsBusi+"/TransIn", qsBusi+"/TransInCompensate", req)
	// 提交saga事务,dtm会完成所有的子事务/回滚所有的子事务
  err := saga.Submit()

完整示例

参考examples/quick_start.go

交流群

请加 yedf2008 好友或者扫码加好友,验证回复 dtm 按照指引进群

yedf2008

如果您觉得此项目不错,或者对您有帮助,请赏颗星吧!

其他语言客户端

python

客户端sdk(当前只支持TCC): https://github.com/yedf/dtmcli-python

示例: https://github.com/yedf/dtmcli-python-sample

node

客户端sdk(当前只支持TCC): https://github.com/yedf/dtmcli-node

示例: https://github.com/yedf/dtmcli-node-sample

谁在使用

常青藤爸爸 镜小二
Issues
  • saga事务进入无限补偿重试,1个补偿大致每秒重试80次,请问是什么原因?

    saga事务进入无限补偿重试,1个补偿大致每秒重试80次,请问是什么原因?

    问题:saga事务进入无限补偿重试,1个补偿大致每秒重试80次,请问是什么原因? 事务:

    • /goods.service.v1.Goods/StockDeduct商品库存扣减,/goods.service.v1.Goods/StockDeductRevert商品库存扣减补偿
    • /order.service.v1.Order/Create订单创建,/order.service.v1.Order/CreateRevert订单创建补偿

    在任务的server端添加了auth jwt 验证,dtm 未添加对应的 auth header token,在这种情况下,商品库存扣减商品库存扣减补偿都无法成功,dtm进入无限重试商品库存扣减补偿状态。

    文档注明会进入 指数退避算法重试 状态,但实际没有,请问是什么原因呢?文档参考

    opened by mowocc 15
  • Seeking advice for usage of dtm

    Seeking advice for usage of dtm

    I build a service which connected to multiple MySQLs,and has a logic that will start an XA transaction on these dbs. On the consideration of distributed transaction, should I set up the grpc server for the logic and use DTM to control the XA transaction, or i just need to let my service to be the TM and control the XA transaction?

    Looking forward to suggestions!

    opened by SgtDaJim 15
  • postman grpc  调用 dtm server 导致dtm server 停止

    postman grpc 调用 dtm server 导致dtm server 停止

    dtm version:1.14.4 docker 部署 dtm env:

    STORE_DRIVER: mysql
    STORE_HOST: xxxxx.com
    STORE_USER: root
    STORE_PASSWORD: 'pwd'
    STORE_PORT: 3306
    
    MICRO_SERVICE_DRIVER: dtm-driver-gozero
    MICRO_SERVICE_TARGET: "consul://host.docker.internal:8500/dtm-server"
    MICRO_SERVICE_END_POINT: "host.docker.internal:36790"
    

    postman version:v9.24.2 grpc Request:
    method: Prepare message: postman 自动生成的

        "BinPayloads": [
            "u1NVblZ8IRIrJN==",
            "jaUYjyap9Mhf2h1A3Zbs1WRv",
            "iENGtAdeW4VHxlMcrjqIFHiDaUR"
        ],
        "CustomedData": "consectetur non esse sit",
        "Gid": "incididunt",
        "QueryPrepared": "aliquip eu",
        "ReqExtra": {},
        "RollbackReason": "est esse ut",
        "Steps": "amet proident nisi",
        "TransOptions": {
            "PassthroughHeaders": [
                "magna proident fugiat exercitation",
                "adipisicing voluptate ullamco",
                "amet ut dolor culpa qui"
            ],
            "RequestTimeout": "18686",
            "RetryInterval": "1313",
            "TimeoutToFail": "862415057",
            "WaitResult": false
        },
        "TransType": "in adipisicing ex fugiat aliquip"
    }
    

    dtm 报错信息

    panic: invalid character 'a' looking for beginning of value
    
    goroutine 67 [running]:
    github.com/dtm-labs/dtm/dtmcli/dtmimp.E2P(...)
            /app/dtm/dtmcli/dtmimp/utils.go:62
    github.com/dtm-labs/dtm/dtmcli/dtmimp.MustUnmarshal(...)
            /app/dtm/dtmcli/dtmimp/utils.go:122
    github.com/dtm-labs/dtm/dtmcli/dtmimp.MustUnmarshalString(...)
            /app/dtm/dtmcli/dtmimp/utils.go:127
    github.com/dtm-labs/dtm/dtmsvr.TransFromDtmRequest(0x2781578, 0xc00017bf80, 0xc0003f3d60, 0x20bc3c0)
            /app/dtm/dtmsvr/trans_class.go:113 +0x469
    github.com/dtm-labs/dtm/dtmsvr.(*dtmServer).Prepare(0x369ad80, 0x2781578, 0xc00017bf80, 0xc0003f3d60, 0x369ad80, 0x0, 0x0)
            /app/dtm/dtmsvr/api_grpc.go:33 +0x45
    github.com/dtm-labs/dtm/dtmgrpc/dtmgpb._Dtm_Prepare_Handler.func1(0x2781578, 0xc00017bf80, 0x218b660, 0xc0003f3d60, 0x2, 0x2, 0x1f1fa00, 0xc0009e7988)
            /app/dtm/dtmgrpc/dtmgpb/dtmgimp_grpc.pb.go:179 +0x89
    github.com/dtm-labs/dtm/dtmgrpc/dtmgimp.GrpcServerLog(0x2781578, 0xc00017bf80, 0x218b660, 0xc0003f3d60, 0xc00041d600, 0xc000a7eb28, 0x49ba06, 0x62cf8585, 0x1c06170d, 0x1c9266e5f0a)
            /app/dtm/dtmgrpc/dtmgimp/types.go:27 +0x1d6
    google.golang.org/grpc.chainUnaryInterceptors.func1.1(0x2781578, 0xc00017bf80, 0x218b660, 0xc0003f3d60, 0x203000, 0x0, 0x0, 0x7f2be3976698)
            /go/pkg/mod/google.golang.org/[email protected]/server.go:1117 +0x8c
    github.com/dtm-labs/dtm/dtmsvr.grpcMetrics(0x2781578, 0xc00017bf80, 0x218b660, 0xc0003f3d60, 0xc00041d600, 0xc00053b300, 0x0, 0x0, 0x0, 0x0)
            /app/dtm/dtmsvr/metrics.go:86 +0x182
    google.golang.org/grpc.chainUnaryInterceptors.func1.1(0x2781578, 0xc00017bf80, 0x218b660, 0xc0003f3d60, 0xc0009e7b48, 0x40e398, 0x18, 0x2054a20)
            /go/pkg/mod/google.golang.org/[email protected]/server.go:1120 +0x11c
    google.golang.org/grpc.chainUnaryInterceptors.func1(0x2781578, 0xc00017bf80, 0x218b660, 0xc0003f3d60, 0xc00041d600, 0xc000a7eb28, 0xc000a6abb8, 0x51d766, 0x214e1a0, 0xc00017bf80)
            /go/pkg/mod/google.golang.org/[email protected]/server.go:1122 +0xeb
    github.com/dtm-labs/dtm/dtmgrpc/dtmgpb._Dtm_Prepare_Handler(0x20bc3c0, 0x369ad80, 0x2781578, 0xc00017bf80, 0xc00012d080, 0xc000468400, 0x2781578, 0xc00017bf80, 0xc0001fe5a0, 0x118)
            /app/dtm/dtmgrpc/dtmgpb/dtmgimp_grpc.pb.go:181 +0x150
    google.golang.org/grpc.(*Server).processUnaryRPC(0xc0009cc000, 0x279b5b8, 0xc000601520, 0xc000160480, 0xc000766a20, 0x364bf30, 0x0, 0x0, 0x0)
            /go/pkg/mod/google.golang.org/[email protected]/server.go:1283 +0x544
    google.golang.org/grpc.(*Server).handleStream(0xc0009cc000, 0x279b5b8, 0xc000601520, 0xc000160480, 0x0)
            /go/pkg/mod/google.golang.org/[email protected]/server.go:1620 +0xd0c
    google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc000334f40, 0xc0009cc000, 0x279b5b8, 0xc000601520, 0xc000160480)
            /go/pkg/mod/google.golang.org/[email protected]/server.go:922 +0xab
    created by google.golang.org/grpc.(*Server).serveStreams.func1
            /go/pkg/mod/google.golang.org/[email protected]/server.go:920 +0x1fd
    
    opened by imythu 8
  • 可否支持多通信协议混用?

    可否支持多通信协议混用?

    粗略的看了一些示例,配置一组事务时只需指定一个 DTM Server,通过其 URL 区分子事务的通信协议是 HTTP、gRPC 或 JSON-RPC。(补充,用到的 cli 也是不同的) 而在实际使用上,我其实并不太关心与 DTM Server 的通信方式,而是关心每个子事务的通信协议。 我的理想使用方式是配置事务时可以给每个子事务分别指定其通信协议,从而支持多协议混用。

    opened by ostatsu 8
  • windows版  http://localhost:36789 无法显示

    windows版 http://localhost:36789 无法显示

    image 配置了redis 和端口 也还是原来默认的 image

    #####################################################################

    dtm can be run without any config.

    all config in this file is optional. the default value is as specified in each line

    all configs can be specified from env. for example:

    MicroService.EndPoint => MICRO_SERVICE_END_POINT

    #####################################################################

    Store: # specify which engine to store trans status

    Driver: 'mysql'

    Host: 'localhost'

    User: 'root'

    Password: ''

    Port: 3306

    Db: 'dtm'

    Driver: 'boltdb' # default store engine

    Driver: 'redis' Host: 'localhost' User: '' Password: '' Port: 6379

    Driver: 'postgres'

    Host: 'localhost'

    User: 'postgres'

    Password: 'mysecretpassword'

    Port: '5432'

    following config is for only Driver postgres/mysql

    MaxOpenConns: 500

    MaxIdleConns: 500

    ConnMaxLifeTime: 5 # default value is 5 (minutes)

    flollowing config is only for some Driver

    DataExpire: 604800 # Trans data will expire in 7 days. only for redis/boltdb.

    FinishedDataExpire: 86400 # finished Trans data will expire in 1 days. only for redis.

    RedisPrefix: '{a}' # default value is '{a}'. Redis storage prefix. store data to only one slot in cluster

    MicroService: # grpc based microservice config

    Driver: 'dtm-driver-gozero' # name of the driver to handle register/discover

    Target: 'etcd://localhost:2379/dtmservice' # register dtm server to this url

    EndPoint: 'localhost:36790'

    HttpMicroService: # http based microservice config

    Driver: 'dtm-driver-http' # name of the driver to handle register/discover

    RegistryType: 'nacos'

    RegistryAddress: '127.0.0.1:8848,127.0.0.1:8848'

    RegistryOptions: '{"UserName":"nacos","Password":"nacos","NotLoadCacheAtStart":true}'

    Target: '{"ServiceName":"dtmService","Enable":true,"Healthy":true,"Weight":10}' # target and options

    EndPoint: '127.0.0.1:36789'

    the unit of following configurations is second

    TransCronInterval: 3 # the interval to poll unfinished global transaction for every dtm process

    TimeoutToFail: 35 # timeout for XA, TCC to fail. saga's timeout default to infinite, which can be overwritten in saga options

    RetryInterval: 10 # the subtrans branch will be retried after this interval

    RequestTimeout: 3 # the timeout of HTTP/gRPC request in dtm

    LogLevel: 'info' # default: info. can be debug|info|warn|error

    Log:

    Outputs: 'stderr' # default: stderr, split by ",", you can append files to Outputs if need. example:'stderr,/tmp/test.log'

    RotationEnable: 0 # default: 0

    RotationConfigJSON: '{}' # example: '{"maxsize": 100, "maxage": 0, "maxbackups": 0, "localtime": false, "compress": false}'

    HttpPort: 36798 GrpcPort: 36799

    JsonRpcPort: 36791

    advanced options

    UpdateBranchAsyncGoroutineNum: 1 # num of async goroutine to update branch status

    opened by Rockny 8
  • consul服务疑问

    consul服务疑问

    docker是虚拟机,内部搭建consul/dtm服务 虚拟机ip 192.168.38.128 两个服务都是开启并可用的 grpc服务器在本地 192.168.0.112:8001 这个服务是一定可用的

    配置中心 配置 MicroService: Driver: 'dtm-driver-gozero' # Target: 'consul://192.168.38.128:2379/dtmservice' # EndPoint: '192.168.38.128:36790'

    具体代码

    `var dtmServer = "consul://192.168.38.128:2379/dtmservice"

    orderRpcBusiServer, err := l.svcCtx.Config.UserRpcConf.BuildTarget()
    if err != nil {
    	//这个地方没报错
    	return nil, fmt.Errorf("下单异常超时")
    }
    //	orderRpcBusiServer = consul://192.168.38.128:8500/user.rpc 服务绝对可用
    fmt.Println(orderRpcBusiServer)
    
    gid := dtmgrpc.MustGenGid(dtmServer)
    saga := dtmgrpc.NewSagaGrpc(dtmServer, gid).
    	//测试过
    	//consul://192.168.38.128:8500/user.rpc
    	//192.168.38.128:8500
    	Add(orderRpcBusiServer+"/usercenter/QueryUser", orderRpcBusiServer"+"/usercenter/QueryUser", &grpc_.UniversalUserRequest{
    		Pattern: 1,
    		Data:    nil,
    	})
    err = saga.Submit()
    dtmimp.FatalIfError(err)`
    

    最终报错

    {"@timestamp":"2022-06-18T22:00:36.758+08:00","caller":"handler/loghandler.go:174","content":"[HTTP] 503 - POST - /order/quickCreate 127.0.0.1:2510 - PostmanRuntime/7.26.2 - slowcall(slowcall(3000.7ms))","duration":"3000.7ms","level":"slow","span":"8c7601ab80ff29ff","trace":"1de87c93118a14f1ec8cc806c7506795"} {"@timestamp":"2022-06-18T22:00:36.758+08:00","caller":"handler/loghandler.go:199","content":"[HTTP] 503 - POST /order/quickCreate - 127.0.0.1:2510 - PostmanRuntime/7.26.2\nPOST /order/quickCreate HTTP/1.1\r\nHost: 127.0.0.1:8889\r\nAccept: /\r\nAccept-Encoding: gzip, deflate, br\r\nConnection: keep-alive\r\nContent-Length: 55\r\nContent-Type: application/json\r\nPostman-Token: 79c6e169-cc47-4ccf-b244-e6f6b041d165\r\nUser-Agent: PostmanRuntime/7.26.2\r\n\r\n{\r\n "userId": 1,\r\n "goodsId": 1,\r\n "num": 1\r\n}","duration":"3000.7ms","level":"error","span":"8c7601ab80ff29ff","trace":"1de87c93118a14f1ec8cc806c7506795"} {"@timestamp":"2022-06-18T22:00:54.764+08:00","caller":"[email protected]/resolver.go:68","content":"[Consul resolver] Couldn't fetch endpoints. target={service='dtmservice' healthy='false' tag=''}; error={Get "http://192.168.38.128:2379/v1/health/service/dtmservice?near=_agent": dial tcp 192.168.38.128:2379: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.}","level":"error"}

    //这个时候我以为是端口错误, consul://192.168.38.128:8500/dtmservice 更换为这个端口,都更换了 报错 {"level":"error","ts":"2022-06-18T22:03:23.036+0800","caller":"dtmgimp/types.go:46","msg":"grpc client called: consul://192.168.38.128:8500/dtmservice/dtmgimp.Dtm/NewGid {} result: {} err: rpc error: code = Unavailable desc = name resolver error: produced zero addresses","stacktrace":"github.com/dtm-labs/dtmgrpc/dtmgimp.GrpcClientLog\n\tH:/z包/Golang/go1.18.1/pkg/mod/github.com/dtm-labs/[email protected]/dtmgimp/types.go:46\ngoogle.golang.org/grpc.(*ClientConn).Invoke\n\tH:/z包/Golang/go1.18.1/pkg/mod/google.golang.org/[email protected]/call.go:35\ngithub.com/dtm-labs/dtmgrpc/dtmgpb.(*dtmClient).NewGid\n\tH:/z包/Golang/go1.18.1/pkg/mod/github.com/dtm-labs/[email protected]/dtmgpb/dtmgimp_grpc.pb.go:43\ngithub.com/dtm-labs/dtmgrpc.MustGenGid\n\tH:/z包/Golang/go1.18.1/pkg/mod/github.com/dtm-labs/[email protected]/type.go:51\nawesomeProject9/internal/logic.(*CreateLogic).Create\n\tC:/Users/z/go/src/awesomeProject9/internal/logic/createLogic.go:44\nawesomeProject9/internal/handler.createHandler.func1\n\tC:/Users/z/go/src/awesomeProject9/internal/handler/createHandler.go:22\nnet/http.HandlerFunc.ServeHTTP\n\tH:/z包/Golang/go1.18.1/go/src/net/http/server.go:2084\ngithub.com/zeromicro/go-zero/rest/handler.GunzipHandler.func1\n\tH:/z包/Golang/go1.18.1/pkg/mod/github.com/zeromicro/[email protected]/rest/handler/gunziphandler.go:26\nnet/http.HandlerFunc.ServeHTTP\n\tH:/z包/Golang/go1.18.1/go/src/net/http/server.go:2084\ngithub.com/zeromicro/go-zero/rest/handler.MaxBytesHandler.func2.1\n\tH:/z包/Golang/go1.18.1/pkg/mod/github.com/zeromicro/[email protected]/rest/handler/maxbyteshandler.go:24\nnet/http.HandlerFunc.ServeHTTP\n\tH:/z包/Golang/go1.18.1/go/src/net/http/server.go:2084\ngithub.com/zeromicro/go-zero/rest/handler.MetricHandler.func1.1\n\tH:/z包/Golang/go1.18.1/pkg/mod/github.com/zeromicro/[email protected]/rest/handler/metrichandler.go:21\nnet/http.HandlerFunc.ServeHTTP\n\tH:/z包/Golang/go1.18.1/go/src/net/http/server.go:2084\ngithub.com/zeromicro/go-zero/rest/handler.RecoverHandler.func1\n\tH:/z包/Golang/go1.18.1/pkg/mod/github.com/zeromicro/[email protected]/rest/handler/recoverhandler.go:21\nnet/http.HandlerFunc.ServeHTTP\n\tH:/z包/Golang/go1.18.1/go/src/net/http/server.go:2084\ngithub.com/zeromicro/go-zero/rest/handler.(*timeoutHandler).ServeHTTP.func1\n\tH:/z包/Golang/go1.18.1/pkg/mod/github.com/zeromicro/[email protected]/rest/handler/timeouthandler.go:79"}

    dtm服务报错

    {"level":"info","ts":"2022-06-18T14:04:25.458Z","caller":"dtmsvr/trans_status.go:27","msg":"TouchCronTime for: {"ID":0,"create_time":"2022-06-18T08:18:28.28577398Z","update_time":"2022-06-18T14:04:25.457628306Z","gid":"gJTkRAxRbomhKBccRqUcxM","trans_type":"tcc","status":"submitted","protocol":"grpc","next_cron_interval":20,"next_cron_time":"2022-06-18T14:04:45.457628Z","wait_result":true}"} {"level":"error","ts":"2022-06-18T14:04:25.458Z","caller":"dtmsvr/trans_process.go:52","msg":"processInner got error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup host.docker.internal on 192.168.38.2:53: no such host"","stacktrace":"github.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).processInner.func1\n\t/app/dtm/dtmsvr/trans_process.go:52\ngithub.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).processInner\n\t/app/dtm/dtmsvr/trans_process.go:63\ngithub.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).process\n\t/app/dtm/dtmsvr/trans_process.go:38\ngithub.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).Process\n\t/app/dtm/dtmsvr/trans_process.go:20\ngithub.com/dtm-labs/dtm/dtmsvr.CronTransOnce\n\t/app/dtm/dtmsvr/cron.go:35\ngithub.com/dtm-labs/dtm/dtmsvr.CronExpiredTrans\n\t/app/dtm/dtmsvr/cron.go:42"} {"level":"info","ts":"2022-06-18T14:04:25.459Z","caller":"dtmsvr/cron.go:54","msg":"cron job return a trans: {"ID":0,"create_time":"2022-06-18T10:57:47.975316619Z","update_time":"2022-06-18T14:04:13.858789044Z","gid":"tCPN73GUWBSohXjEuQmpzW","trans_type":"msg","steps":[{"action":"127.0.0.1:8001/usercenter/QueryUser"}],"status":"submitted","protocol":"grpc","next_cron_interval":10,"next_cron_time":"2022-06-18T14:04:35.458458466Z","wait_result":true}"} {"level":"error","ts":"2022-06-18T14:04:25.459Z","caller":"dtmgimp/types.go:46","msg":"grpc client called: 127.0.0.1:8001/usercenter/QueryUser "IAEqensicGFzc1dvcmQiOiIxMjM0NTYiLCJtb2JpbGUiOjEzNTY4NjU0NDIsIndlQ2hhdFRva2VuIjoiIiwibW9kZWwiOiJwNTAiLCJicmFuZCI6IuWNjuS4uiIsImlkZW50aWZpY2F0aW9uQ29kZSI6Ijk4NzY1NDMyMSJ9" result: "" err: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:8001: connect: connection refused"","stacktrace":"github.com/dtm-labs/dtm/dtmgrpc/dtmgimp.GrpcClientLog\n\t/app/dtm/dtmgrpc/dtmgimp/types.go:46\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryClient.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:72\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryClient.func1\n\t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:81\ngoogle.golang.org/grpc.(*ClientConn).Invoke\n\t/go/pkg/mod/google.golang.org/[email protected]/call.go:35\ngithub.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).getURLResult\n\t/app/dtm/dtmsvr/trans_status.go:90\ngithub.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).getBranchResult\n\t/app/dtm/dtmsvr/trans_status.go:123\ngithub.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).execBranch\n\t/app/dtm/dtmsvr/trans_status.go:138\ngithub.com/dtm-labs/dtm/dtmsvr.(*transMsgProcessor).ProcessOnce\n\t/app/dtm/dtmsvr/trans_type_msg.go:69\ngithub.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).processInner\n\t/app/dtm/dtmsvr/trans_process.go:62\ngithub.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).process\n\t/app/dtm/dtmsvr/trans_process.go:38\ngithub.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).Process\n\t/app/dtm/dtmsvr/trans_process.go:20\ngithub.com/dtm-labs/dtm/dtmsvr.CronTransOnce\n\t/app/dtm/dtmsvr/cron.go:35\ngithub.com/dtm-labs/dtm/dtmsvr.CronExpiredTrans\n\t/app/dtm/dtmsvr/cron.go:42"} {"level":"info","ts":"2022-06-18T14:04:25.459Z","caller":"dtmsvr/trans_status.go:27","msg":"TouchCronTime for: {"ID":0,"create_time":"2022-06-18T10:57:47.975316619Z","update_time":"2022-06-18T14:04:25.459222335Z","gid":"tCPN73GUWBSohXjEuQmpzW","trans_type":"msg","steps":[{"action":"127.0.0.1:8001/usercenter/QueryUser"}],"status":"submitted","protocol":"grpc","next_cron_interval":20,"next_cron_time":"2022-06-18T14:04:45.459222025Z","wait_result":true}"} {"level":"error","ts":"2022-06-18T14:04:25.459Z","caller":"dtmsvr/trans_process.go:52","msg":"processInner got error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:8001: connect: connection refused"","stacktrace":"github.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).processInner.func1\n\t/app/dtm/dtmsvr/trans_process.go:52\ngithub.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).processInner\n\t/app/dtm/dtmsvr/trans_process.go:63\ngithub.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).process\n\t/app/dtm/dtmsvr/trans_process.go:38\ngithub.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).Process\n\t/app/dtm/dtmsvr/trans_process.go:20\ngithub.com/dtm-labs/dtm/dtmsvr.CronTransOnce\n\t/app/dtm/dtmsvr/cron.go:35\ngithub.com/dtm-labs/dtm/dtmsvr.CronExpiredTrans\n\t/app/dtm/dtmsvr/cron.go:42"} 这里有个很奇怪的事情 127.0.0.1:8001,但没有地方上传 grpc 服务对应的地址是192.168.0.112:8001

    opened by 309791679 6
  • dtm server数据库异常的问题

    dtm server数据库异常的问题

    image 在今天中午十二点过后,服务所有的注册事务全都出现了如上图所示的异常情况,succeed在action之前就插入了,然后根据update time看发起时间又是succeed这一条在后,目前不清楚是由于什么原因导致的,本来应该更新branch_id为1的status,结果莫名其妙插入了一条数据,导致global事务查询状态失败,然后事务所有的注册事务全部阻塞。

    opened by lzw621558 6
  • kubernets部署dtm出现错误

    kubernets部署dtm出现错误

    平台:k8s 1.21.5 部署yaml:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: dtm
      labels:
        app: dtm
    spec:
      replicas: 3
      selector:
        matchLabels:
          app: dtm
      template:
        metadata:
          labels:
            app: dtm
        spec:
          containers:
            - name: dtm
              image: yedf/dtm:1.11.0
              imagePullPolicy: IfNotPresent
              args:
                - "-c=/app/dtm/configs/config.yaml"
              volumeMounts:
                - mountPath: /app/dtm/configs
                  name: config
              ports:
                - containerPort: 36789
                  protocol: TCP
                  name: http
                - containerPort: 36790
                  protocol: TCP
                  name: grpc
              livenessProbe:
                httpGet:
                  path: /api/ping
                  port: 36789
                  scheme: HTTP
              readinessProbe:
                httpGet:
                  path: /api/ping
                  port: 36789
                  scheme: HTTP
              resources:
                requests:
                  cpu: 200m
                  memory: 200Mi
          volumes:
            - name: config
              configMap:
                name: dtm-conf
    ---
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: dtm-conf
      labels:
        app: dtm
    data:
      config.yaml: |-
        Store:
          Driver: mysql
          Host: mysqlhost
          Port: 3306
          User: mysqluser
          Password: 'mysqlpass'
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: dtm-svc
      labels:
        app: dtm
    spec:
      ports:
        - port: 8080
          targetPort: 36790
          name: grpc
          appProtocol: grpc # Kubernetes v1.20 [stable],低版本请剔除此行
        - port: 80
          targetPort: 36789
          name: http
          appProtocol: http # Kubernetes v1.20 [stable],低版本请剔除此行
      selector:
        app: dtm
      type: NodePort
    ---
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: dtm
      annotations:
        alb.ingress.kubernetes.io/scheme: internal
        alb.ingress.kubernetes.io/target-type: ip
        alb.ingress.kubernetes.io/success-codes: "200"
        alb.ingress.kubernetes.io/load-balancer-name: dtm
        alb.ingress.kubernetes.io/healthcheck-path: /api/ping
    spec:
      ingressClassName: alb
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: dtm-svc
                    port:
                      number: 80
                path: /
                pathType: Prefix
    
    

    错误日志:

    {"level":"error","ts":"2022-01-25T07:31:10.124Z","caller":"dtmsvr/cron.go:63","msg":"----recovered panic http/grpc result should be specified as in:\nhttps://dtm.pub/summary/arch.html#http\nunkown result will be retried: Post \"http://host.docker.internal:8081/api/barrierTransInConfirm?branch_id=02&gid=uL8g2XXz9U7x3ke3gnZNHJ&op=confirm&trans_type=tcc\": dial tcp: lookup host.docker.internal on 172.20.0.10:53: no such host\ngoroutine 34 [running]:\nruntime/debug.Stack(0xc000105da0, 0x1d04200, 0xc0007a2870)\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x9f\ngithub.com/dtm-labs/dtm/dtmsvr.handlePanic(0x0)\n\t/app/dtm/dtmsvr/cron.go:63 +0x5b\npanic(0x1d04200, 0xc0007a2870)\n\t/usr/local/go/src/runtime/panic.go:965 +0x1b9\ngithub.com/dtm-labs/dtm/dtmcli/dtmimp.PanicIf(...)\n\t/app/dtm/dtmcli/dtmimp/utils.go:75\ngithub.com/dtm-labs/dtm/dtmsvr.CronTransOnce(0xc0009876b0, 0x16)\n\t/app/dtm/dtmsvr/cron.go:38 +0x1e7\ngithub.com/dtm-labs/dtm/dtmsvr.CronExpiredTrans(0xffffffffffffffff)\n\t/app/dtm/dtmsvr/cron.go:45 +0x3e\ncreated by main.main\n\t/app/dtm/main.go:72 +0x210\n","stacktrace":"github.com/dtm-labs/dtm/dtmsvr.handlePanic\n\t/app/dtm/dtmsvr/cron.go:63\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:965\ngithub.com/dtm-labs/dtm/dtmcli/dtmimp.PanicIf\n\t/app/dtm/dtmcli/dtmimp/utils.go:75\ngithub.com/dtm-labs/dtm/dtmsvr.CronTransOnce\n\t/app/dtm/dtmsvr/cron.go:38\ngithub.com/dtm-labs/dtm/dtmsvr.CronExpiredTrans\n\t/app/dtm/dtmsvr/cron.go:45"}
    
    opened by xbclub 6
  • docker 使用 环境变量无效,默认连接还是 bolt

    docker 使用 环境变量无效,默认连接还是 bolt

    执行docker 运行命令:

    sudo docker run -d --name dtm -p 36789:36789 -p 36790:36790 -e DRIVER_STORE_HOST='127.0.0.1' -e DRIVER_STORE_USER='root' -e DRIVER_STORE_PASSWORD='123456' -e DRIVER_STORE_PORT="3306" -e DRIVER_STORE_DRIVER="mysql" yedf/dtm:latest 环境变量参考的是: image

    进入docker 中查看 : image

    连接的数据还是bolt ,请求是配置环境变量有问题,还是需要其他的配置消息

    opened by costa92 6
  • 通过判断NACOS_SVR这个环境变量是否有值来决定服务启动时是从nacos读取配置还是读本地配置文件

    通过判断NACOS_SVR这个环境变量是否有值来决定服务启动时是从nacos读取配置还是读本地配置文件

    通过判断NACOS_SVR这个环境变量是否有值来决定服务启动时是从nacos读取配置还是读本地配置文件 NACOS_SVR: nacos服务地址,如:192.168.1.19:8848 NACOS_NS: nacos的namespace NACOS_DATA-ID: 在nacos中的配置key NACOS_GROUP:在nacos中的配置所属的group

    除些之外,通过以下环境变量还可以进一步控制dtm服务作为nacos客户端时的行为 NACOS_TIMEOUT-MS NACOS_LOG-DIR NACOS_CACHE-DIR NACOS_ROTATE-TIME NACOS_MAX-AGE NACOS_LOG-LEVEL

    opened by nobugtodebug 6
  • 支持立即重试未完成的全局事务

    支持立即重试未完成的全局事务

    需求场景:

    假如公司某个子事务的业务经历了宕机3小时后恢复上线,那么按照dtm的指数退避重试算法,下一次重试业务子事务的时间平均要1.5个小时之后。因此业务希望在业务恢复之后,能够立即重试子事务,快速完成,因此需要dtm支持手动调整下一次的重试时间。

    设计:

    暴露一个新接口: /api/dtmsvr/resetCronTime 行为:在存储引擎中,将nextCronTime超过Now+3*TimeoutToFail的任务,选取100条,将NextCronTime改为当前时间,触发立即重试

    enhancement help wanted core 
    opened by yedf2 5
  •  INSERT IGNORE INTO barrier死锁问题

    INSERT IGNORE INTO barrier死锁问题

    业务使用二阶段消息,发现报警: 1、GET /api/dtm/msg/queryPrepared?branch_id=00&gid=ZLk5US5qQF5tMGyEvCz4Kh&op=msg&trans_type=msg 响应状态码:500

    2、死锁 SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction (SQL: INSERT IGNORE INTO barrier (trans_type, gid, branch_id, op, barrier_id, reason) values(msg,ZLk5US5qQF5tMGyEvCz4Kh,00,msg,01,rollback))

    麻烦问下,出现这个的原因是什么?需要如何解决?

    opened by yuql 4
  • Does it support app gRPC-endpoints with HTTPS?

    Does it support app gRPC-endpoints with HTTPS?

    Hi, I'm a DTM msg user.

    It seems that "action" and "queryPrepared" require no-protocol URLs for gRPC endpoints, and the DTM uses HTTP protocol by default.

    How can I customize to use HTTPS protocol? Thanks.

    opened by gdlcf88 2
  • Support rollback_reason for global transaction

    Support rollback_reason for global transaction

    There may be rollback for XA,Saga,Tcc patterns. DTM want to record the rollback reason, and then user can query the global transaction to see what cause the global transaction to rollback.

    • The column in db to record the reason can be: rollback_reason varchar(1024)
    • When XA, TCC timeout, the rollback_reason should be "Timeout after x seconds"
    help wanted core 
    opened by yedf2 0
Releases(v1.15.1)
PolarDB Cluster Manager is the cluster management component of PolarDB for PostgreSQL, responsible for topology management, high availability, configuration management, and plugin extensions.

What is PolarDB Cluster Manager PolarDB Cluster Manager is the cluster management component of PolarDB for PostgreSQL, responsible for topology manage

null 8 Dec 15, 2021
SQL transaction wrapper on golang

TxWrapper TxWrapper is a sql transaction wrapper. It helps to exclude writing code for rollback and commit commands. Usage import ( "context"

Dmitry Stoletoff 1 Mar 14, 2022
A go Library for scan database/sql rows to struct、slice、other types. And it support multiple databases connection management

ploto A go Library for scan database/sql rows to struct、slice、other types. And it support multiple databases connection management It's not an ORM. wo

solar 2 Mar 16, 2022
MySQL replication topology management and HA

orchestrator [Documentation] orchestrator is a MySQL high availability and replication management tool, runs as a service and provides command line ac

null 4.6k Jul 30, 2022
BQB is a lightweight and easy to use query builder that works with sqlite, mysql, mariadb, postgres, and others.

Basic Query Builder Why Simple, lightweight, and fast Supports any and all syntax by the nature of how it works Doesn't require learning special synta

Aaron M 39 Jul 22, 2022
Zinc Search engine. A lightweight alternative to elasticsearch that requires minimal resources, written in Go.

Zinc Zinc is a search engine that does full text indexing. It is a lightweight alternative to elasticsearch and runs in less than 100 MB of RAM. It us

Prabhat Sharma 9.8k Jul 29, 2022
Lightweight SQL database written in Go for prototyping and playing with text (CSV, JSON) data

gopicosql Lightweight SQL database written in Go for prototyping and playing wit

null 2 Jul 27, 2022
Mogo: a lightweight browser-based logs analytics and logs search platform for some datasource(ClickHouse, MySQL, etc.)

mogo Mogo is a lightweight browser-based logs analytics and logs search platform

Shimo HQ 745 Aug 5, 2022
Worker failover support for PostgreSQL Citus extension using pg_auto_failover.

citus-failover Worker failover support for citus community version using pg_auto_failover. What is this? This is a simple service to monitor changes i

Navid Zarepak 22 Aug 3, 2022
support clickhouse

Remote storage adapter This is a write adapter that receives samples via Prometheus's remote write protocol and stores them in Graphite, InfluxDB, cli

weetime 29 Jun 29, 2022
MySQL Storage engine conversion,Support mutual conversion between MyISAM and InnoDB engines.

econvert MySQL Storage engine conversion 简介 此工具用于MySQL存储引擎转换,支持CTAS和ALTER两种模式,目前只支持MyISAM和InnoDB存储引擎相互转换,其它引擎尚不支持。 注意:当对表进行引擎转换时,建议业务停止访问或者极少量访问时进行。 原

null 5 Oct 25, 2021
Clickhouse support for GORM

clickhouse Clickhouse support for GORM Quick Start package main import ( "fmt" "github.com/sweetpotato0/clickhouse" "gorm.io/gorm" ) // User

null 1 Aug 1, 2022
Support MySQL or MariaDB for gopsql/psql and gopsql/db

mysql Support MySQL or MariaDB for github.com/gopsql/psql. You can make MySQL SELECT, INSERT, UPDATE, DELETE statements with this package. NOTE: Pleas

null 0 Dec 9, 2021
A go package to add support for data at rest encryption if you are using the database/sql.

go-lockset A go package to add support for data at rest encryption if you are using the database/sql to access your database. Installation In your Gol

Bartlomiej Mika 0 Jan 30, 2022
PgSQL compatible on distributed database TiDB

TiDB for PostgreSQL Introduction TiDB for PostgreSQL is an open source launched by Digital China Cloud Base to promote and integrate into the open sou

DigitalChinaOpenSource 338 Aug 1, 2022
Distributed tracing using OpenTelemetry and ClickHouse

Distributed tracing backend using OpenTelemetry and ClickHouse Uptrace is a dist

Uptrace 774 Aug 2, 2022
Api service to serve messaging feature

Messaging Service Api service to serve messaging feature. This service running in Here Feature User can send message to another user Users can list al

M.Hakim Amransyah 0 Dec 13, 2021
This Service collects slow queries and returns them in response.

pgsql-api-go This Service collects slow queries and returns them in response. Status This service is the very first version of the project. App is up

ngh 0 Dec 30, 2021
go mysql driver, support distributed transaction

Go-MySQL-Driver A MySQL-Driver for Go's database/sql package Features Requirements Installation Usage DSN (Data Source Name) Password Protocol Address

Open Transaction 34 Jul 23, 2022
A db proxy for distributed transaction, read write splitting and sharding! Support any language! It can be deployed as a sidecar in a pod.

DBPack DBPack means a database cluster tool pack. It can be deployed as a sidecar in a pod, it shields complex basic logic, so that business developme

null 262 Aug 2, 2022
Distributed-Services - Distributed Systems with Golang to consequently build a fully-fletched distributed service

Distributed-Services This project is essentially a result of my attempt to under

Hamza Yusuff 6 Jun 1, 2022
Transaction management tool for taxable investments

Market Lot Robot Transaction management tool for taxable investments. How it works Run the web socket server with the following command: go run . Visi

theoperator.eth 0 Oct 19, 2021
PolarDB Cluster Manager is the cluster management component of PolarDB for PostgreSQL, responsible for topology management, high availability, configuration management, and plugin extensions.

What is PolarDB Cluster Manager PolarDB Cluster Manager is the cluster management component of PolarDB for PostgreSQL, responsible for topology manage

null 8 Dec 15, 2021
GTA(Go Task Async) is a lightweight reliable asynchronous task and transaction message library for Golang

GTA (Go Task Async) is a lightweight and reliable asynchronous task and transaction message library for by golang.

Kevin Su 12 Jun 4, 2022
Distributed-Log-Service - Distributed Log Service With Golang

Distributed Log Service This project is essentially a result of my attempt to un

Hamza Yusuff 6 Jun 1, 2022
a go client for distributed transaction manager DTM. go语言分布式事务管理器DTM的客户端SDK

a client for distributed transaction manager dtm dtmcli 是分布式事务管理器dtm的客户端sdk dtmcli 这个库的代码与dtm下的dtmcli代码保持完全同步,如果您需要线上使用,那么当前的dtmcli相关的依赖非常少,对于最终打包的镜像也

null 12 Apr 27, 2022
Coroutine based Deterministic 2PC Distributed Transaction

Enviorment set up off the mod go env -w GO111MODULE=off change to GPATH to project directory go env -w GOPATH=$HOME/coroprove Usage This project serve

Jack Wang 0 Oct 30, 2021
A cross-language distributed transaction manager

English Docs 跨语言分布式事务管理器 DTM是一款golang开发的分布式事务管理器,解决了跨数据库、跨服务、跨语言栈更新数据的一致性问题。 他优雅

DTM Development and Communities 7.2k Aug 7, 2022
ConsenSys Software 8 Jan 18, 2022