Dragonfly is an intelligent P2P based image and file distribution system.

Last update: Jan 9, 2023

Overview

Dragonfly

Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order to get stable binaries.

Introduction

Dragonfly is an open source intelligent P2P based image and file distribution system. Its goal is to tackle all distribution problems in cloud native scenarios. Currently Dragonfly focuses on being:

Simple: well-defined user-facing API (HTTP), non-invasive to all container engines;
Efficient: CDN support, P2P based file distribution to save enterprise bandwidth;
Intelligent: host level speed limit, intelligent flow control due to host detection;
Secure: block transmission encryption, HTTPS connection support.

Dragonfly is now hosted by the Cloud Native Computing Foundation (CNCF) as an Incubating Level Project. Originally it was born to solve all kinds of distribution at very large scales, such as application distribution, cache distribution, log distribution, image distribution, and so on.

Dragonfly has finished refactoring in Golang. Now versions > 0.4.0 are totally in Golang, while those < 0.4.0 are in Java. We encourage adopters to try Golang version first, since Java versions will be out of support in the next few releases.

Features

In details, Dragonfly has the following features:

P2P based file distribution: Using P2P technology for file transmission, which can make full use of the bandwidth resources of each peer to improve download efficiency, saves a lot of cross-IDC bandwidth, especially costly cross-board bandwidth
Non-invasive support for all kinds of container technologies: Dragonfly can seamlessly support various containers for distributing images.
Host level speed limit: Many downloading tools(wget/curl) only have rate limit for the current download task, but dragonfly also provides rate limit for the entire host.
Passive CDN: The CDN mechanism can avoid repetitive remote downloads.
Strong consistency: Dragonfly can guarantee that all downloaded files must be consistent even if users do not provide any check code(MD5).
Disk protection and high efficient IO: Precheck Disk space, delay synchronization, write file-block in the best order, split net-read / disk-write, and so on.
High performance: Cluster Manager is completely closed-loop, which means, it does not rely on any DB or distributed cache, processing requests with extremely high performance.
Exception auto isolation: Dragonfly will automatically isolate exception nodes(peer or Cluster Manager) to improve download stability.
No pressure on file source: Generally, as long as a few Cluster Managers download file from the source.
Support standard http header: Support http header, Submit authentication information through http header.
Effective concurrency control of Registry Auth: Reduce the pressure of the Registry Auth Service.
Simple and easy to use: Very few configurations are needed.

Comparison

For Dragonfly, no matter how many clients start the file downloading, the average downloading time is almost stable without increasement (12s in experiment, which means it only takes 12s in total for all client to finish downloading file/image).

And for wget, the downloading time keeps increasing when you have more clients. As the number of wget clients reaches 1200 (in following experiment), the file source will crash, then it can not serve any client.

The following table shows the testing environment and the graph shows the comparison result.

Test Environment	Statistics
Dragonfly server	2 * (24core 64GB 2000Mb/s)
File Source server	2 * (24core 64GB 2000Mb/s)
Client	4core 8GB 200Mb/s
Target file size	200MB

Roadmap

For more details about roadmap, please refer to file ROADMAP.md.

Community

You are encouraged to communicate most things via GitHub issues or pull requests.

Other active channels:

Twitter: @dragonfly_oss
Dingtalk Group(钉钉群)

Contributing

You are warmly welcomed to hack on Dragonfly. We have prepared a detailed guide CONTRIBUTING.md.

License

Dragonfly is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.

Comments

Add Continuous Fuzzing via Fuzzit

This PR adds continuous fuzzing via Fuzzit.

Description is at https://github.com/dragonflyoss/Dragonfly/issues/846

fixes https://github.com/dragonflyoss/Dragonfly/issues/846

cc @lowzj
size/L

opened by yevgenypats 18

Dragonfly does not work

Ⅰ. Issue Description

Ⅱ. Describe what happened

Follow Dragonfly Quick Start to install Dragonfly, but it does not work.

Ⅲ. Describe what you expected to happen

Ⅳ. How to reproduce it (as minimally and precisely as possible)

Env:

SuperNode: 192.168.21.103
Clients: 192.168.21.102, 192.168.21.101

Installation:

SuperNode

$ docker run --name dragonfly-supernode --restart=always -d -p 8001:8001 -p 8002:8002 registry.cn-hangzhou.aliyuncs.com/dragonflyoss/supernode:0.3.0

Clients

$ cat <<EOD >/etc/dragonfly.conf
[node]
address=192.168.21.103
EOD
$ docker run --name dragonfly-dfclient --restart=always -d -p 65001:65001 -v /root/.small-dragonfly:/root/.small-dragonfly -v /etc/dragonfly.conf:/etc/dragonfly.conf dragonflyoss/dfclient:v0.3.0 --registry=https://registry.xx.com
# Add "registry-mirrors": ["http://127.0.0.1:65001"] in /etc/docker/daemon.json
$ systemctl restart docker

Test:

I can pull images on client node, but there is on datas and logs from Dragonfly.

$ du -sh /root/.small-dragonfly/dfdaemon/data/
0	/root/.small-dragonfly/dfdaemon/data/

$ cat /root/.small-dragonfly/logs/dfdaemon.log
2019-03-15 12:31:22.357 INFO sign:1 : init...
2019-03-15 12:31:22.361 INFO sign:1 : rotate log routine start...
2019-03-15 12:31:22.396 INFO sign:1 : dfget version:
2019-03-15 12:31:22.396 ERRO sign:1 : init properties failed:open /etc/dragonfly/dfdaemon.yml: no such file or directory
2019-03-15 12:31:22.397 INFO sign:1 : init properties:{"Registries":[{"Schema":"https","Host":"cargo.caicloudprivatetest.com","Certs":null,"Regx":"(^localhost$)|(^127.0.0.1$)|(^127.0.0.1$)"}]}
2019-03-15 12:31:22.397 INFO sign:1 : init finish
2019-03-15 12:31:22.397 INFO sign:1 : start dfdaemon param: &{DfPath:/dfclient/dfget DFRepo:/root/.small-dragonfly/dfdaemon/data/ RateLimit:20M CallSystem:com_ops_dragonfly URLFilter:Signature&Expires&OSSAccessKeyId Notbs:true MaxProcs:4 Version:false Verbose:false HostIP:127.0.0.1 Port:65001 Registry:https://cargo.caicloudprivatetest.com DownRule: CertFile: KeyFile: TrustHosts:[] ConfigPath:/etc/dragonfly/dfdaemon.yml}
2019-03-15 12:31:22.397 INFO sign:1 : launch dfdaemon http server on 127.0.0.1:65001
2019-03-15 12:33:22.398 INFO sign:1 : scan repo and clean expired files
2019-03-15 12:33:22.399 INFO sign:1 : ignore /root/.small-dragonfly/dfdaemon/data/: not a regular file
2019-03-15 12:35:22.400 INFO sign:1 : scan repo and clean expired files
2019-03-15 12:35:22.401 INFO sign:1 : ignore /root/.small-dragonfly/dfdaemon/data/: not a regular file
2019-03-15 12:37:22.403 INFO sign:1 : scan repo and clean expired files
2019-03-15 12:37:22.404 INFO sign:1 : ignore /root/.small-dragonfly/dfdaemon/data/: not a regular file

$ docker logs -f --tail 50 dragonfly-dfclient
# there is no log from dfclient

Ⅴ. Anything else we need to know?

Ⅵ. Environment:

dragonfly version: v0.3.0
OS (e.g. from /etc/os-release): Linux
Kernel (e.g. uname -a): Linux 3.10.0-693.el7.x86_64
Install tools:
Others:

opened by supereagle 18

Fix an issue where HTTPS is not supported

Ⅰ. Describe what this PR did

Fix bug

Ⅱ. Does this pull request fix one issue?

Issue: #1231

Ⅲ. Why don't you add test cases (unit test/integration test)? (你真的觉得不需要加测试吗？)

Need integration testing

Ⅳ. Describe how to verify it

Don't config docker mirrors，pull the images from a non-local address（eg：docker pull 192.168.1.100:8443/alpine:3.9）

Ⅴ. Special notes for reviews
kind/bug size/XS areas/security

opened by readerx 16
When dfget specifies -- node node, dfget still synchronizes old files after the supernode file is updated

Question

问题描述：想测试一下，用dfget 同步过程中，supernode（也是file server）中的文件修改后，是否下次马上同步新文件，发现用dfget 同步的依旧是修改前的老文件，使用命令： dfget --url --output /data/qiyu/indexing.data --node ** -s 50M --totallimit 50M -b dfget --version ：0.2.0 start download by dragonfly ====================start==================== progress[3092019/3092019 in 0.000s] =====================end===================== download SUCCESS(0) cost(0.191s) length:3092019 reason:0 fail to download indexing.data at Thu Jun 27 15:30:02 CST 2019 remote file length 3092024 local file length 3092019
areas/test

opened by huerlei 15

[Question] dfget fails to pull images

Question

After installed all component of dragonfly and finished all configuration according to this document, I failed to pull images via dragonfly with great possibility.

when failed, check the dfclient.log:

[2018-11-30 16:14:54,289] INFO sign:8797-1543565691.582 lineno:64 : do register to 127.0.0.1,remainder:['127.0.0.1']
[2018-11-30 16:14:54,771] INFO sign:8626-1543565671.974 lineno:110 : pull piece task result:{u'msg': u'client sucCount:2,cdn status:RUNNING,cdn sucCount:2', u'code': 602} and sleep 1.142 ...
[2018-11-30 16:14:54,916] INFO sign:8625-1543565671.979 lineno:110 : pull piece task result:{u'msg': u'client sucCount:0,cdn status:RUNNING,cdn sucCount:0', u'code': 602} and sleep 1.890 ...
[2018-11-30 16:14:55,921] INFO sign:8626-1543565671.974 lineno:60 : local http result:20971520 for path:/rate/ and cost:0.001
[2018-11-30 16:14:55,957] INFO sign:8626-1543565671.974 lineno:110 : pull piece task result:{u'msg': u'client sucCount:3,cdn status:RUNNING,cdn sucCount:3', u'code': 602} and sleep 1.064 ...
[2018-11-30 16:14:56,813] INFO sign:8625-1543565671.979 lineno:110 : pull piece task result:{u'msg': u'client sucCount:0,cdn status:RUNNING,cdn sucCount:0', u'code': 602} and sleep 1.013 ...
[2018-11-30 16:14:57,026] INFO sign:8626-1543565671.974 lineno:110 : pull piece task result:{u'msg': u'client sucCount:3,cdn status:RUNNING,cdn sucCount:3', u'code': 602} and sleep 1.577 ...
[2018-11-30 16:14:57,832] INFO sign:8625-1543565671.979 lineno:110 : pull piece task result:{u'msg': u'client sucCount:0,cdn status:RUNNING,cdn sucCount:0', u'code': 602} and sleep 1.743 ...
[2018-11-30 16:14:58,610] INFO sign:8626-1543565671.974 lineno:110 : pull piece task result:{u'msg': u'client sucCount:3,cdn status:RUNNING,cdn sucCount:3', u'code': 602} and sleep 1.613 ...
[2018-11-30 16:14:59,295] ERROR sign:8797-1543565691.582 lineno:81 : register to node:127.0.0.1 error
Traceback (most recent call last):
  File "/usr/local/bin/component/httputil.py", line 67, in register
    schema, node), data=params, timeout=(2.0, 5.0))
  File "/usr/local/bin/vendor/requests-2.18.4-py2.7.egg/requests/sessions.py", line 555, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/local/bin/vendor/requests-2.18.4-py2.7.egg/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/bin/vendor/requests-2.18.4-py2.7.egg/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/bin/vendor/requests-2.18.4-py2.7.egg/requests/adapters.py", line 521, in send
    raise ReadTimeout(e, request=request)
ReadTimeout: HTTPConnectionPool(host='127.0.0.1', port=8002): Read timed out. (read timeout=5.0)
[2018-11-30 16:14:59,300] INFO sign:8797-1543565691.582 lineno:64 : do register to 127.0.0.1,remainder:[]
[2018-11-30 16:14:59,582] INFO sign:8625-1543565671.979 lineno:110 : pull piece task result:{u'msg': u'client sucCount:0,cdn status:RUNNING,cdn sucCount:0', u'code': 602} and sleep 1.895 ...
[2018-11-30 16:15:00,084] ERROR sign:8797-1543565691.582 lineno:136 : init fail but try back down
Traceback (most recent call last):
  File "/usr/local/bin/dfget", line 128, in <module>
    register_result = env.init()
  File "/usr/local/bin/env.py", line 148, in init
    paramparser.cmdparam.identifier)
  File "/usr/local/bin/env.py", line 84, in register
    nodes, paramparser.cmdparam.url, task_url, port, http_path, md5, identifier)
  File "/usr/local/bin/component/httputil.py", line 86, in register
    raise Exception("register result:%s" % result)
Exception: register result:{u'code': 607}
[2018-11-30 16:15:00,085] INFO sign:8797-1543565691.582 lineno:599 : download fail and not back source
[2018-11-30 16:15:00,087] INFO sign:8797-1543565691.582 lineno:94 : |UNKNOWN|http://hub.c.163.com/v2/library/php/blobs/sha256:2871138c7173567814c6bda10b07b3d772bfafb53ffefe66a4e44e530f4127bc|-1|0|UNKNOWN|com_ops_dragonfly|8.505|
[2018-11-30 16:15:00,087] INFO sign:8797-1543565691.582 lineno:111 : download FAIL cost:8.505s length:-1 reason:1001
[2018-11-30 16:15:00,234] INFO sign:8626-1543565671.974 lineno:60 : local http result:20971520 for path:/rate/ and cost:0.002
[2018-11-30 16:15:00,271] INFO sign:8626-1543565671.974 lineno:110 : pull piece task result:{u'msg': u'client sucCount:4,cdn status:RUNNING,cdn sucCount:4', u'code': 602} and sleep 0.956 ...
[2018-11-30 16:15:01,240] INFO sign:8626-1543565671.974 lineno:110 : pull piece task result:{u'msg': u'client sucCount:4,cdn status:RUNNING,cdn sucCount:4', u'code': 602} and sleep 1.511 ...
[2018-11-30 16:15:01,484] INFO sign:8625-1543565671.979 lineno:110 : pull piece task result:{u'msg': u'client sucCount:0,cdn status:RUNNING,cdn sucCount:0', u'code': 602} and sleep 1.554 ...

kind/question

opened by BlueBlue-Lee 15

dfget 本地下载报错

RT，我在本地启动两个dfget进程，同时下载一个在线文件，报了如下错误，请问是什么原因呢

本地通过docker，启了supernode进程

docker run -d -p 8001:8001 -p 8002:8002 f28c73445883

我的dragonfly.conf配置如下 [node] address=127.0.0.1,127.0.0.1

dfclient 报错日志：

[2017-12-15 18:56:47,844] WARNING sign:70462-1513335389.992 lineno:167 : has not available pieceTask,maybe resource lack [2017-12-15 18:56:47,848] WARNING sign:70462-1513335389.992 lineno:167 : has not available pieceTask,maybe resource lack [2017-12-15 18:56:47,851] INFO sign:70462-1513335389.992 lineno:110 : pull piece task result:{u'msg': u'piece resource lack', u'code': 602} and sleep 1.432 ... [2017-12-15 18:56:47,938] ERROR sign:70469-1513335392.125 lineno:332 : piece range:0-5242879 error,realMd5:70461da8b94c6ca5d2fda3260c5a8c3b,expectedMd5:436a148497d25f524d222019b174f54f,dstIp:127.0.0.1,total:162 [2017-12-15 18:56:47,943] ERROR sign:70469-1513335392.125 lineno:332 : piece range:10485760-15728639 error,realMd5:70461da8b94c6ca5d2fda3260c5a8c3b,expectedMd5:7d05eb939cd37dc6f612d09a9b6319cc,dstIp:127.0.0.1,total:162 [2017-12-15 18:56:47,944] ERROR sign:70469-1513335392.125 lineno:332 : piece range:5242880-10485759 error,realMd5:70461da8b94c6ca5d2fda3260c5a8c3b,expectedMd5:d99f987ccca3083eaac61338d800da60,dstIp:127.0.0.1,total:162 [2017-12-15 18:56:47,947] WARNING sign:70469-1513335392.125 lineno:167 : has not available pieceTask,maybe resource lack [2017-12-15 18:56:47,958] WARNING sign:70469-1513335392.125 lineno:167 : has not available pieceTask,maybe resource lack [2017-12-15 18:56:47,965] ERROR sign:70469-1513335392.125 lineno:332 : piece range:10485760-15728639 error,realMd5:70461da8b94c6ca5d2fda3260c5a8c3b,expectedMd5:7d05eb939cd37dc6f612d09a9b6319cc,dstIp:127.0.0.1,total:162 [2017-12-15 18:56:47,970] INFO sign:70469-1513335392.125 lineno:110 : pull piece task result:{u'msg': u'piece resource lack', u'code': 602} and sleep 0.665 ...

求大神帮忙看下是什么原因，谢谢了^_^

opened by meglory 15
docker proxy error

when docker pull, there are some errors as follows: root@iZ2ze7tbgo9lk7jubiy6tcZ:~# df-daemon -registry https://docker.acmcoder.com -port 18001

launch df-daemon on port:18001 2017/12/12 18:21:28 http: proxy error: context canceled 2017/12/12 18:21:32 http: proxy error: context canceled 2017/12/12 18:21:34 http: proxy error: context canceled 2017/12/12 18:21:35 http: proxy error: context canceled 2017/12/12 18:21:36 http: proxy error: context canceled 2017/12/12 18:21:36 http: proxy error: dial tcp: i/o timeout 2017/12/12 18:21:37 http: proxy error: context canceled 2017/12/12 18:21:38 http: proxy error: context canceled Killed root@iZ2ze7tbgo9lk7jubiy6tcZ:~# curl https://docker.acmcoder.com
Harbor
Loading...
root@iZ2ze7tbgo9lk7jubiy6tcZ:~#

opened by lifubang 15
Using GOPROXY environment variables if set instead of default value

Ⅰ. Describe what this PR did

Using GOPROXY environment variables if set instead of default value

Ⅱ. Does this pull request fix one issue?

Ⅲ. Why don't you add test cases (unit test/integration test)? (你真的觉得不需要加测试吗？)

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews
size/S

opened by SataQiu 14
feature: support non-ipv4 format node address for Dragonfly dfclient

Signed-off-by: Starnop [email protected]

Ⅰ. Describe what this PR did

Make it work normally when specifies non-ipv4 format node address for dfclient.

Ⅱ. Does this pull request fix one issue?

#679

Ⅲ. Why don't you add test cases (unit test/integration test)? (你真的觉得不需要加测试吗？)

None.

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

BTW. update the shell scripts.
size/M kind/feature

opened by Starnop 14
feature: update API definition to be more accurate

Signed-off-by: Allen Sun [email protected]

Ⅰ. Describe what this PR did

update API deinition to be more accurate

Ⅱ. Does this pull request fix one issue?

none

Ⅲ. Why don't you add test cases (unit test/integration test)? (你真的觉得不需要加测试吗？)

just definition of API, no need to test.

Ⅳ. Describe how to verify it

none

Ⅴ. Special notes for reviews

none
size/XL kind/feature

opened by allencloud 14
fix valid ip not support ipv6

Ⅰ. Describe what this PR did

fix valid ip not support ipv6

Ⅱ. Does this pull request fix one issue?

fixes #334

Ⅲ. Why don't you add test cases (unit test/integration test)? (你真的觉得不需要加测试吗？)

unit test

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews
kind/bug size/S

opened by zgfh 13
ipv6 support
Ⅰ. Issue Description

The server bind ip '0.0.0.0' only useing for ipv4. ipv6 should be support。

Ⅱ. Describe what happened

can not support ipv6 listen & advertiseIP not support ipv6

Ⅲ. Describe what you expected to happen

export --bind-ip flags or change default bind ip ‘0.0.0.0’ to ‘::’ advertise-ip support ipv6

Ⅳ. How to reproduce it (as minimally and precisely as possible)

Ⅴ. Anything else we need to know?

Ⅵ. Environment:

dragonfly version:

OS (e.g. from /etc/os-release):

Kernel (e.g. uname -a):

Install tools:

Others:
opened by lidajia 1
chore(deps): bump github.com/valyala/fasthttp from 1.3.0 to 1.34.0
Bumps github.com/valyala/fasthttp from 1.3.0 to 1.34.0.

Release notes

Sourced from github.com/valyala/fasthttp's releases.

v1.34.0

59f94a3 Update github.com/klauspost/compress (#1237) (Mikhail Faraponov)

62c15a5 Don't reset RequestCtx.s (#1234) (Erik Dubbelboer)

7670c6e Fix windows tests (#1235) (Erik Dubbelboer)

f54ffa1 feature: Keep the memory usage of the service at a stable level (#1216) (Rennbon)

15262ec Warn about unsafe ServeFile usage (#1228) (Erik Dubbelboer)

1116d03 Fix panic while reading invalid trailers (Erik Dubbelboer)

856ca8e Update dependencies (#1230) (Mikhail Faraponov)

6b5bc7b Add windows support to normalizePath (Erik Dubbelboer)

f0b0cfe Don't log ErrBadTrailer by default (Erik Dubbelboer)

6937fee fix: (useless check), skip Response body if http method HEAD (#1224) (Pavel Burak)

b85d2a2 Fix http proxy behavior (#1221) (Aoang)

ad8a07a RequestHeader support set no default ContentType (#1218) (Jack.Ju)

c94581c support configure HostClient (#1214) (lin longhjui)

632e222 Client examples (#1208) (Sergey Ponomarev)

6a3cc23 uri_test.go use example.com for clearness (#1212) (Sergey Ponomarev)

9d665e0 Update dependencies (#1204) (Mikhail Faraponov)

8d7953e Fix scheme check for not yet parsed requests (#1203) (ArminBTVS)

v1.33.0

61aa8b1 remove redundant code (#1202) (tyltr)

4369776 fix(hijack): reuse RequestCtx (#1201) (Sergio VS)

2aca3e8 fix(hijack): reset userValues after hijack handler execution (#1199) (Sergio VS)

9123060 Updated dependencies (#1194) (Mikhail Faraponov)

v1.32.0

7eeb00e Make tests less flaky (#1189) (Erik Dubbelboer)

d19b872 Update tcpdialer.go (#1188) (Mikhail Faraponov)

c727b99 Release UseHostHeader in ReleaseRequest() (#1185) (Tolyar)

6c0518b Fix UseHostHeader for DoTimeout + tests (#1184) (Tolyar)

6b55811 Add MaxIdleWorkerDuration to Server. (#1183) (Kilos Liu)

4517204 Allow to set Host header for Client (#1169) (Tolyar)

258a4c1 fix: reset response after reset user values on keep-alive connections (#1176) (Sergio VS)

e9db537 Use %w to wrap errors (#1175) (Erik Dubbelboer)

7db0597 Fix bad request trailer panic (Erik Dubbelboer)

4aadf9a Fix parseTrailer panic (Erik Dubbelboer)

da7ff7a Add trailer support (#1165) (ichx)

017f0aa fix: reset request after reset user values on keep-alive connections (#1162) (Sergio VS)

3b117f8 feat: close idle connections when server shutdown (#1155) (ichx)

a94a2c3 Remove redundant code (#1154) (ichx)

f7c354c Fix race condition in Client.mCleaner (Erik Dubbelboer)

c078a9d Add string and bytes buffer convert trick in README (#1151) (ichx)

3ff6aaa uri: isHttps() and isHttp() (#1150) (Sergey Ponomarev)

8febad0 http.go: Request.SetURI() (Fix #1141) (#1148) (Sergey Ponomarev)

2ca01c7 fix: Status Line parsing and writing (#1135) (Shivansh Vij)

931d0a4 Fix lint (Erik Dubbelboer)

d613502 use sync.map is better (#1145) (halst)

c15e642 Don't run all race tests on windows (#1143) (Erik Dubbelboer)

6006c87 chore (#1137) (tyltr)

6d4db9b Fix race condition in getTCPAddrs (Erik Dubbelboer)

... (truncated)

Commits

59f94a3 Update github.com/klauspost/compress (#1237)

62c15a5 Don't reset RequestCtx.s (#1234)

7670c6e Fix windows tests (#1235)

f54ffa1 feature: Keep the memory usage of the service at a stable level (#1216)

15262ec Warn about unsafe ServeFile usage (#1228)

1116d03 Fix panic while reading invalid trailers

856ca8e Update dependencies (#1230)

6b5bc7b Add windows support to normalizePath

f0b0cfe Don't log ErrBadTrailer by default

6937fee fix: (useless check), skip Response body if http method HEAD (#1224)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies go
opened by dependabot[bot] 0
knative pull image by Dragonfly is error

Question

my deploy knative service yaml :

apiVersion: serving.knative.dev/v1 kind: Service metadata: name: nginx namespace: cvd spec: template: metadata: name: nginx-v3 annotations: autoscaling.knative.dev/minScale: "1" autoscaling.knative.dev/maxScale: "11" autoscaling.knative.dev/target: "122" spec: containers: - env: value: "2" image: tools/nginx:1.9.1 imagePullPolicy: IfNotPresent name: user-container

/etc/docker/daemon.json:

{ "registry-mirrors": [ "http://127.0.0.1:65001" ], "max-concurrent-downloads": 10, "log-driver": "json-file", "log-level": "warn", "log-opts": { "max-size": "10m", "max-file": "3" }, "data-root":"/apps/docker" }

error: Unable to fetch image "tools/nginx:1.9.1": failed to resolve image to digest: Get "https://index.docker.io/v2/": dial tcp: lookup index.docker.io on 10.96.0.10:53: no such host

docker info :

Client: Debug Mode: false

Server: Containers: 50 Running: 22 Paused: 0 Stopped: 28 Images: 24 Server Version: 19.03.0 Storage Driver: overlay2 Backing Filesystem: xfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f init version: fec3683 Security Options: seccomp Profile: default Kernel Version: 3.10.0-1160.el7.x86_64 Operating System: BigCloud Enterprise Linux 7.6 (Core) OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 62.42GiB Name: 10.253.173.21 ID: NZR5:RHYS:MZDG:X3SU:JDEF:ZCG7:QPPV:VYVK:PHL2:SA3D:JKKY:44NH Docker Root Dir: /apps/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: docker.dep.devops.cmit.cloud:20100 127.0.0.0/8 Registry Mirrors: http://127.0.0.1:65001/ Live Restore Enabled: false Product License: Community Engine

create pod by Dragonfly is success but knative is erro

opened by gfppower 0

dfget变僵尸进程

Ⅰ. dfget变僵尸进程

[root@700 ~]# ps aux  | grep dfget
root      5289  0.0  0.0  12132  1156 pts/1    S+   10:07   0:00 grep --color=auto dfget
root     29631  0.0  0.0      0     0 ?        Z    Dec09   0:01 [dfget] <defunct>
[root@700 ~]#

Ⅱ. 错误日志

dfclient.log

2021-12-09 22:17:45.987 INFO sign:158-1639058589.746 : do register to x.x.x.x:8002, res:null error:dialing to the given TCP address timed out
2021-12-09 22:17:46.541 ERRO sign:158-1639058589.746 : failed to report piece{taskid:5e7d99190ab19eeb996bd25ecd612fa7fbedf045b32b398d754d94ed4bff9fd8,range:352321536-358612991},err: dialing to the given TCP address timed out
2021-12-09 22:17:46.541 WARN sign:158-1639058589.746 : failed to report piece to supernode with request(&{TaskID:5e7d99190ab19eeb996bd25ecd612fa7fbedf045b32b398d754d94ed4bff9fd8 Cid:dockerip-158-1639058589.746 DstCid:cdnnode:dockerip~5e7d99190ab19eeb996bd25ecd612fa7fbedf045b32b398d754d94ed4bff9fd8 PieceRange:352321536-358612991}) for (1) times and will retry after sleep 0.359s
2021-12-09 22:17:48.388 WARN sign:158-1639058589.746 : success to report piece with request(&{TaskID:5e7d99190ab19eeb996bd25ecd612fa7fbedf045b32b398d754d94ed4bff9fd8 Cid:dockerip-158-1639058589.746 DstCid:cdnnode:dockerip~5e7d99190ab19eeb996bd25ecd612fa7fbedf045b32b398d754d94ed4bff9fd8 PieceRange:352321536-358612991}) after retrying (2) times
2021-12-09 22:17:48.988 INFO sign:158-1639058589.746 : do register to x.x.x.x:8002, res:null error:dialing to the given TCP address timed out
2021-12-09 22:17:48.988 ERRO sign:158-1639058589.746 : register fail:{"Code":-100,"Msg":"dialing to the given TCP address timed out"}
2021-12-09 22:17:48.988 ERRO sign:158-1639058589.746 : failed to download piece: {"Code":-100,"Msg":"dialing to the given TCP address timed out"}
2021-12-09 22:17:48.989 ERRO sign:158-1639058589.746 : failed to download by dragonfly: failed to download with p2p pattern, reason: 3, and start try to download from source
2021-12-09 22:17:48.989 INFO sign:158-1639058589.746 : start download b4dffbe3-7409-450f-84e1-1f1f7dfbacec from the source station
2021-12-10 00:10:22.990 INFO sign:158-1639058589.746 : download FAIL cost:7633.244s length:442008134 reason:3 error:failed to download file from source: download timeout(6754.000s)
2021-12-09 22:17:46.541 ERRO sign:158-1639058589.746 : failed to report piece{taskid:5e7d99190ab19eeb996bd25ecd612fa7fbedf045b32b398d754d94ed4bff9fd8,range:352321536-358612991},err: dialing to the given TCP address timed out
2021-12-09 22:17:46.541 WARN sign:158-1639058589.746 : failed to report piece to supernode with request(&{TaskID:5e7d99190ab19eeb996bd25ecd612fa7fbedf045b32b398d754d94ed4bff9fd8 Cid:dockerip-158-1639058589.746 DstCid:cdnnode:dockerip~5e7d99190ab19eeb996bd25ecd612fa7fbedf045b32b398d754d94ed4bff9fd8 PieceRange:352321536-358612991}) for (1) times and will retry after sleep 0.359s
2021-12-09 22:17:48.388 WARN sign:158-1639058589.746 : success to report piece with request(&{TaskID:5e7d99190ab19eeb996bd25ecd612fa7fbedf045b32b398d754d94ed4bff9fd8 Cid:dockerip-158-1639058589.746 DstCid:cdnnode:dockerip~5e7d99190ab19eeb996bd25ecd612fa7fbedf045b32b398d754d94ed4bff9fd8 PieceRange:352321536-358612991}) after retrying (2) times
2021-12-09 22:17:48.988 INFO sign:158-1639058589.746 : do register to x.x.x.x:8002, res:null error:dialing to the given TCP address timed out
2021-12-09 22:17:48.988 ERRO sign:158-1639058589.746 : register fail:{"Code":-100,"Msg":"dialing to the given TCP address timed out"}
2021-12-09 22:17:48.988 ERRO sign:158-1639058589.746 : failed to download piece: {"Code":-100,"Msg":"dialing to the given TCP address timed out"}
2021-12-09 22:17:48.989 ERRO sign:158-1639058589.746 : failed to download by dragonfly: failed to download with p2p pattern, reason: 3, and start try to download from source
2021-12-09 22:17:48.989 INFO sign:158-1639058589.746 : start download b4dffbe3-7409-450f-84e1-1f1f7dfbacec from the source station
2021-12-10 00:10:22.990 INFO sign:158-1639058589.746 : download FAIL cost:7633.244s length:442008134 reason:3 error:failed to download file from source: download timeout(6754.000s)

dfserver.log

2021-12-09 22:03:09.456 INFO sign:63-1639058328.796 : update total limit to 209715200
2021-12-09 22:03:09.526 ERRO sign:63-1639058328.796 : failed to open file:4416a668-508a-45c4-b1f3-7480a641197d-112-1639058534.194, failed to get taskPath: 4416a668-508a-45c4-b1f3-7480a641197d-112-1639058534.194
2021-12-09 22:03:09.899 INFO sign:63-1639058328.796 : update total limit to 209715200
2021-12-09 22:03:10.384 ERRO sign:63-1639058328.796 : failed to open file:b3fbc506-56b8-4b30-bff9-d01b37e22408-85-1639058322.810, failed to get taskPath: b3fbc506-56b8-4b30-bff9-d01b37e22408-85-1639058322.810
2021-12-09 22:03:10.384 ERRO sign:63-1639058328.796 : failed to open file:8e772c19-3fac-4910-b98d-3b628986ca39-86-1639058333.073, failed to get taskPath: 8e772c19-3fac-4910-b98d-3b628986ca39-86-1639058333.073
2021-12-09 22:03:10.385 ERRO sign:63-1639058328.796 : failed to open file:060b5cc5-1a6d-4425-a83f-d957b4368b1c-88-1639058325.280, failed to get taskPath: 060b5cc5-1a6d-4425-a83f-d957b4368b1c-88-1639058325.280
2021-12-09 22:03:10.385 ERRO sign:63-1639058328.796 : failed to open file:50623dd5-9c9e-4422-8e22-3cba78c5ed52-86-1639058371.813, failed to get taskPath: 50623dd5-9c9e-4422-8e22-3cba78c5ed52-86-1639058371.813
2021-12-09 22:03:10.567 ERRO sign:63-1639058328.796 : failed to open file:396baa34-edc9-4ae5-9629-1f1049a6c036-87-1639058323.241, failed to get taskPath: 396baa34-edc9-4ae5-9629-1f1049a6c036-87-1639058323.241
2021-12-09 22:03:10.729 ERRO sign:63-1639058328.796 : failed to open file:396baa34-edc9-4ae5-9629-1f1049a6c036-87-1639058323.241, failed to get taskPath: 396baa34-edc9-4ae5-9629-1f1049a6c036-87-1639058323.241
2021-12-09 22:03:10.885 ERRO sign:63-1639058328.796 : failed to open file:1f005af6-5e1d-4e72-8f79-98c1db778d1e-88-1639058333.660, failed to get taskPath: 1f005af6-5e1d-4e72-8f79-98c1db778d1e-88-1639058333.660
2021-12-09 22:03:11.045 ERRO sign:63-1639058328.796 : failed to open file:fc13b11b-7655-4fa6-9793-4e0967ed8749-87-1639058327.307, failed to get taskPath: fc13b11b-7655-4fa6-9793-4e0967ed8749-87-1639058327.307
2021-12-09 22:03:11.203 ERRO sign:63-1639058328.796 : failed to open file:fc13b11b-7655-4fa6-9793-4e0967ed8749-87-1639058327.307, failed to get taskPath: fc13b11b-7655-4fa6-9793-4e0967ed8749-87-1639058327.307
2021-12-09 22:06:23.581 INFO sign:63-1639058328.796 : server gc, delete file:/root/.small-dragonfly/data/1864bb3f-b17d-470a-bfc7-e61b2446967b-26-1639058328.552.service
2021-12-09 22:06:23.727 INFO sign:63-1639058328.796 : server gc, delete file:/root/.small-dragonfly/data/a0006b05-0c17-47ec-9e63-ee8689f12c55-146-1639058589.423.service
2021-12-09 22:20:25.299 INFO sign:63-1639058328.796 : server gc, delete file:/root/.small-dragonfly/data/cae62168-b4ca-4c50-932d-77e8a73421eb-134-1639058385.453
2021-12-09 22:20:25.303 INFO sign:63-1639058328.796 : server gc, delete file:/root/.small-dragonfly/data/cae62168-b4ca-4c50-932d-77e8a73421eb-134-1639058385.453.service
2021-12-09 22:20:55.832 INFO sign:63-1639058328.796 : server gc, delete file:/root/.small-dragonfly/data/b4dffbe3-7409-450f-84e1-1f1f7dfbacec-158-1639058589.746
2021-12-09 22:20:55.832 INFO sign:63-1639058328.796 : server gc, delete file:/root/.small-dragonfly/data/b4dffbe3-7409-450f-84e1-1f1f7dfbacec-158-1639058589.746.service
2021-12-09 22:22:00.464 INFO sign:63-1639058328.796 : no more task, peer server will stop...
2021-12-09 22:22:00.466 INFO sign:63-1639058328.796 : peer server is shutdown.

Ⅲ. Environment:

dragonfly version:

dragonflyoss/dfclient:latest
dragonflyoss/supernode:latest

OS (e.g. from /etc/os-release):

NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="8"

Kernel (e.g. uname -a):

Linux 700.link.china 4.18.0-147.5.1.el8_1.x86_64 #1 SMP Wed Feb 5 02:00:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Install tools:
Others:

supernode 3台
下载镜像1.4G
dfclient 50台

opened by divmid 0

Fix goroutine leak bug in DoDownloadTimeout

Ⅰ. Describe what this PR did

This PR fix a goroutine leak bug in DoDownloadTimeout

Ⅱ. Does this pull request fix one issue?

NONE

Ⅲ. Why don't you add test cases (unit test/integration test)?

This fix is already covered by TestDoDownloadTimeout

Ⅳ. Describe how to verify it

In the original code, DoDownloadTimeout has a potential goroutine leak in rare goroutine intervening. It can be verified by using https://github.com/uber-go/goleak if we instrument the entry of func Test(t *testing.T) in dfget/core/downloader/downloader_test.go and then run TestDoDownloadTimeout enough times:

go test github.com/dragonflyoss/Dragonfly/dfget/core/downloader -gocheck.f=TestDoDownloadTimeout

This command results in :

time="2021-11-01T13:37:27+08:00" level=warning msg="invalid download timeout(0.000s), use default:(300.000s)"
leaks.go:78: found unexpected goroutines:
    [Goroutine 8 in state chan send, with github.com/dragonflyoss/Dragonfly/dfget/core/downloader.DoDownloadTimeout.func1 on top of the stack:
    goroutine 8 [chan send]:
    github.com/dragonflyoss/Dragonfly/dfget/core/downloader.DoDownloadTimeout.func1()
            /home/yuanting/work/dev/goprojects/dragonfly/dfget/core/downloader/downloader.go:56 +0x12f
    created by github.com/dragonflyoss/Dragonfly/dfget/core/downloader.DoDownloadTimeout
            /home/yuanting/work/dev/goprojects/dragonfly/dfget/core/downloader/downloader.go:55 +0x205
        ]

Ⅴ. Special notes for reviews

opened by timmyyuan 0

Releases(v1.0.6)

v1.0.6(Sep 21, 2020)

support preheat for register. At present, it has completed the integration with the harbor
Source code(tar.gz)
Source code(zip)
checksums.txt(203 bytes)
Dragonfly_1.0.6_darwin_amd64.tar.gz(16.12 MB)
Dragonfly_1.0.6_linux_amd64.tar.gz(15.47 MB)
v1.0.5(Aug 26, 2020)

Changelog

5c732d5 Merge pull request #1456 from ansinlee/master 9aafd32 pass unit test f8946b1 preheat support manifests list ce14eb4 update go version to 1.13.15 of circleci 3dbdf79 add dfget to supernode dockerfile ee974ae add test parse layers 1e77334 add implement of preheat 4906cd1 Merge pull request #1446 from pouchrobot/auto-doc-2020-07-30 4ef67f7 docs: auto generate Dragonfly cli/api docs via code 7da71ca Merge pull request #1443 from qhb1001/docs 5fda8bd docs: modify the container id to be run cd36a2f Merge pull request #1435 from zcc35357949/fix/cdn_download_slow ce65c0c bugfix: notify supernode client piece task success under cdn pattern f33568a Merge pull request #1432 from pouchrobot/auto-doc-2020-07-17 2bd05b9 docs: auto generate Dragonfly cli/api docs via code daa6ddd Merge pull request #1431 from pouchrobot/auto-doc-2020-07-16 a015475 Merge pull request #1429 from zhaxzhax/fix-#1428 5173d7d docs: auto generate Dragonfly cli/api docs via code 8ce59d0 Merge pull request #1411 from zcc35357949/crosswrite_panic 2ea9362 bugfix: fix missing error message 'Unable to autodetect advertiser ip, please set it via --advertise-ip' related to #1428 c6928f8 bugfix: fix dfget panic when link error between output file and tmp file. 12cfd62 Merge pull request #1426 from SataQiu/fix-pool-test-20200713 8d9596e fix the bug in unit tests about sync.Pool 8c716a1 Merge pull request #1414 from YesterdayxD/master 60051a9 Merge pull request #1425 from pouchrobot/auto-doc-2020-07-13 1f875bd docs: auto generate Dragonfly cli/api docs via code 911ce40 fix : spelling mistake and markdown format 68b07f6 Merge pull request #1412 from YesterdayxD/master 65f58bc fix format 88cb000 Merge pull request #1375 from lowzj/fix-ByteBuffer-gc f01641e Merge pull request #1366 from lowzj/refactor-api-register 764b4ed optimize: release allocated ByteBuffer explicitly cd9fa36 refactor: use package server/api to register supernode's API f5cc28f Merge pull request #1404 from lowzj/readme bb5482f doc(README): change "Sandbox" to "Incubating" 39aa614 Merge pull request #1398 from antsystem/update-docs 7ffa1e7 update docs f372031 Merge pull request #1382 from hs0210/work d92dfcb Merge pull request #1383 from Hellcatlk/go-lint 862a6a7 Merge pull request #1374 from fenggw-fnst/work b23a54f test: add unit test case for static_locator.go 2f07b25 Merge pull request #1381 from lowzj/fix-downloader-header 60479b2 Merge pull request #1386 from lowzj/fix-gc-directory 2559fbf Merge pull request #1385 from Komey/docs_update c58f062 fix: delete empty directory in local storage 4474dca fix: remove default registry definition in Dockerfile. fix: change dfpath default value in dfdaemon config template. e64a762 bugfix: cdn cannot re-download uncompleted task with header param 80eae52 Merge pull request #1362 from antsystem/feat/add-hash-circler-supernode-locator ea73b4f test: add some unit test cases for pkg/errortypes/errortypes.go 92ea5c8 fix go-lint warning: "exported method ... should have comment or be unexported" 0fdd9e7 optimize the hash circle, the algorithm of insert/delete/find is updated to rbtree instead of array. 07b4ad9 Merge pull request #1368 from antsystem/feat/define-clientopt-for-protocol 3b1c257 add functional options to define opts for NewProtocolClient. ed1145f Merge pull request #1372 from antsystem/feat/add-more-supernode-api-for-seed-pattern 0da63b0 add hash circler locator 9542232 add supernode api for seed pattern 5981693 Merge pull request #1369 from fenggw-fnst/work e110a40 Merge pull request #1371 from lowzj/fix-dingtalk-qr a1c850d fix: replace expired QR-code of dingtalk group f2dce8f test: add unit test case for supernode_value.go e8a6514 Merge pull request #1364 from antsystem/feat/update-clientwriter-interface 872c6e2 update interface of ClientWriter. f1bd6e8 Merge pull request #1346 from lowzj/api-register 71bde95 refactor: define api register of supernode to add api 3945cdb Merge pull request #1352 from antsystem/feat/add-implementation-for-dfget-interface 3cd96a0 add implementation of http/https protocol ef3d3c4 Merge pull request #1358 from antsystem/feat/define-schedule-peer-info 0dc8afb add schedule peer info 88cb9fa Merge pull request #1355 from antsystem/feat/update-dfget-interface-type-define cefdca4 move some types to basic package 4350001 Merge pull request #1353 from lowzj/fix-deleteTaskFiles a9f24af fix: don't delete the parent directory of the task file eb8359c Merge pull request #1351 from antsystem/feat/optimize-HTTPWithHeaders edaaa3b optimize the HTTPWithHeaders.When do a http request, share the transport and add timeout in request context to share the http client. 54f0a67 Merge pull request #1343 from antsystem/feat/reconstruct-dfget-interface d04b08c define the interface of downloader, data_scheduler, client_stream, register, uploader, data_protocol and url protocol. d499c51 Merge pull request #1345 from lowzj/locator-add-select 9864570 feature: add Select api into SupernodeLocator f131b5f Merge pull request #1305 from antsystem/feat/add-seed-cache-prepare-pr a2a6d82 add cache for seed. 10a6ac2 Merge pull request #1348 from lowzj/fix-git-depth d631f4d fix(ci): disable shadow clone in travis-ci 183a2f5 Merge pull request #1347 from pouchrobot/auto-doc-2020-05-15 7788033 docs: auto generate Dragonfly cli/api docs via code 2443cba Merge pull request #1325 from lowzj/refactor-dfget-core-with-locator 122ac3c refactor: refactor dfget/core with SupernodeLocator 8a1b37a Merge pull request #1338 from AdamKorcz/master 6e7e41d Added 2 fuzzers 164e390 Merge pull request #1272 from Hellcatlk/master 6da8759 add unit test cases 2af6a81 Merge pull request #1330 from q384566678/add-config-test 4ca7667 supernode/config: add UT for IsSuperCID 003b73d Merge pull request #1340 from lowzj/fix-docker-build e70f944 fix: pass GOPROXY to docker build 33c3f7f Merge pull request #1339 from pouchrobot/auto-doc-2020-05-12 c69bc35 docs: auto generate Dragonfly cli/api docs via code 849f36f Merge pull request #1313 from antsystem/supernode_support_seed_task_pr 0ca3333 Merge pull request #1242 from lowzj/preheat 435eb38 feature: define the interface and http api of preheat 0d3e9b2 Merge pull request #1324 from lowzj/locator-interface 359175e Merge pull request #1326 from SataQiu/fix-log-20200506 51695f3 proxy: do not log error messages when http Server is closed 854074c refactor: redefine the behavior of SupernodeLocator 4525cb4 define new apis 44506bf Merge pull request #1321 from lowzj/fix-checkOutput dbc297d bugfix: infinite loop in dfget/config/config.go#checkOutput 8237693 Merge pull request #1294 from lowzj/SupernodeLocate 7e04f05 feature: define the interface SupernodeLocator ab71c0d Merge pull request #1318 from lowzj/dfget-util ce8b258 refactor: move package dfget/util to pkg/algorithm 6cc9d86 Merge pull request #1317 from pouchrobot/auto-doc-2020-04-30 62cd952 docs: auto generate Dragonfly cli/api docs via code 8134362 bugfix: initialize dfget downloader in dfdaemon 77c995d Merge pull request #1308 from lowzj/fix-sleep 11611b5 bugfix: notify p2p downloader to pull next piece after reporting 93152ce Merge pull request #1316 from hxbaishikele/typo_v1 2c6d041 typo: fix typo cdnDource to cdnSource 4eda7c3 Merge pull request #1310 from tanjunchen/improve-code e73614c pkg/algorithm/algorithm.go:improve algorithm ca96df3 Merge pull request #1302 from lowzj/fix-GCDSlice 10cfda1 fix: GCDSlice returns 1 when slice is empty a8ade02 Merge pull request #1304 from pouchrobot/auto-doc-2020-04-27 f3fc768 docs: auto generate Dragonfly cli/api docs via code 734cde5 Merge pull request #1297 from antsystem/feat/add-seed-downloader-prepare-pr 97d20db Merge pull request #1300 from Komey/feature_expose_more_ports c84e38d feature: modify dfdaemon port validate method, allow to expose more port to listen. 1da7fba add seed downloader 06d2368 Merge pull request #1286 from antsystem/feat/define-seed-interface cb47003 define the interface of seed 7aa7efc Merge pull request #1285 from antsystem/feat/add-bitmap 9814193 add bitmap in pkg, the max bitIndex is math.MaxInt32 4c2af2b Merge pull request #1291 from fenggw-fnst/work efbf460 Merge pull request #1292 from pouchrobot/auto-doc-2020-04-18 f6f0e6f docs: auto generate Dragonfly cli/api docs via code 58e82bb test: add unit test case for super_reader.go 0e4e541 Merge pull request #1290 from antsystem/feat/add-seed-response 0996a1b add seed response in seed pattern. 9426ea9 Merge pull request #1277 from antsystem/fix/update-heart-beat-api 702937c update api definition of /peer/heartbeat and /peer/network 6681c18 Merge pull request #1287 from moyasu/master e942b10 docs: fix typo 19f1613 Merge pull request #1275 from antsystem/feat/add-lru-queue-and-mockfileserver-prepare-pr b3222f5 add lru queue and mockFileServer 7856ccd Merge pull request #1271 from hs0210/work cbfc36c Merge pull request #1267 from fenggw-fnst/work 05b2f8e Merge pull request #1270 from antsystem/feat/define-some-api-types-prepare-pr 1d160be define some api types. 3d28168 Merge pull request #1259 from lowzj/cdnPattern-support-range 0d5bc54 test: add unit test case for func GetSuperCID ef57720 fix: cdn-source pattern supports range task 9a4e7b0 Merge pull request #1268 from pouchrobot/auto-doc-2020-04-03 0ab5157 test: add unit test case for func WithConsole 9ebc1b7 docs: auto generate Dragonfly cli/api docs via code 18b652c Merge pull request #1264 from antsystem/feat/extract-dfget-lib-prepare-pr ce27e84 remove https://coreos.com/operators/ from roadmap.md in case ci/circleci: linters-check not success 101a072 add stream mode which do not up dfget progress. aa709f6 Merge pull request #1262 from fenggw-fnst/work 988e45f test: add unit test case for func ReportMetrics b184f25 Merge pull request #1261 from pouchrobot/auto-doc-2020-03-25 abe08cd docs: auto generate Dragonfly cli/api docs via code df222f9 Merge pull request #1258 from Hellcatlk/UnitTest 79fd076 Merge pull request #1260 from fenggw-fnst/work 0f41f77 test: add unit test case for core.go 7179243 Merge pull request #1257 from hs0210/work d7d4ff4 Merge pull request #1213 from Starnop/source-cdn 9abd23a add unit test for client/httputils.go d9d4c92 test: add unit test case for func ConstructRangeStr 26d9d97 feature: support source cdn pattern bae7528 Merge pull request #1255 from pouchrobot/auto-doc-2020-03-20 532c391 docs: auto generate Dragonfly cli/api docs via code f27e2e4 Merge pull request #1253 from lowzj/fix-issue1198 e399394 Merge pull request #1252 from SataQiu/add-gen-ca-20200317 e6a20fb feature: add dfdaemon sub command 'gen-ca' for CA generation d38af90 bugfix: check err returned by peerMgr.Get to prevent nil pointer reference 894eae8 Merge pull request #1251 from pouchrobot/auto-doc-2020-03-17 a4323e7 Merge pull request #1235 from antsystem/feat/support-custom-protocol 9f4f017 docs: auto generate Dragonfly cli/api docs via code 768851b update IsValidURL 523e98d add test for RegisterProtocol 92de56e support custom protocols 393bde4 Merge pull request #1249 from xiaoanyunfei/bugfix/conn_close e8c93e7 bugfix:cConn should close immediately 6de0f19 Merge pull request #1247 from pouchrobot/auto-doc-2020-03-15 15c81ab docs: auto generate Dragonfly cli/api docs via code 2022ee9 Merge pull request #1246 from antsystem/fix/json-decode-panic 5da96fe fix the concurrent race of reuse of fasthttp resp body 35d3d7e Merge pull request #1241 from fenggw-fnst/work 7ebffb3 test: add unit test case for func WithMaxSizeMB bc16744 Merge pull request #1240 from pouchrobot/auto-doc-2020-03-12 3714813 docs: auto generate Dragonfly cli/api docs via code 74ba83c Merge pull request #1232 from readerx/master 06bdaed Fix an issue where HTTPS is not supported d8e4f96 Merge pull request #1221 from q384566678/add-supernode-value-test a47017f Merge pull request #1223 from littleroad/symboliclink 0d0a606 Merge pull request #1234 from fenggw-fnst/work 17e65e3 bugfix: fix untested case due to prefix missing 5e7589b add unit test for NodeWeightSlice2StringSlice 8af6879 change Wight into Weight 7aa441b Merge pull request #1220 from allencloud/docs-data-integrity 8712de3 Merge pull request #1228 from pouchrobot/auto-doc-2020-02-20 6dd69a9 docs: auto generate Dragonfly cli/api docs via code fc56458 Merge pull request #1215 from allencloud/remove-gitter dcd35d5 Merge pull request #1225 from littleroad/construct_error_with_conventions 58ccd6a pkg/fileutils: constructing errors following conventions caad9fd Merge pull request #1194 from fenggw-fnst/work2 a88b5f0 Merge pull request #1219 from q384566678/add-fileutils-test 6dd279b Merge pull request #1226 from fenggw-fnst/work 46593a5 fix typo: fix comment for const var DefaultFailureCountLimit d1fbbd3 test: add unit test case for func GetAllPeerIDs 082e612 pkg/fileutils: add unit test for SymbolicLink 16763d6 pkg/fileutils: add verifications for SymbolicLink a392463 Merge pull request #1216 from q384566678/add-suncmap-unit-test e33b08f pkg/fileutils: add unit test for IsRegularFile 11ce918 pkg/syncmap: add unit test for GetAsAtomicInt 5e94ef2 docs: add data integrity theory of distributing files and blocks bc736ed Merge pull request #1214 from q384566678/add-test 1481da3 Merge pull request #1218 from pouchrobot/auto-doc-2020-02-13 2b004ce docs: auto generate Dragonfly cli/api docs via code 790e620 Merge pull request #1186 from antsystem/add-client-stream-writer 2d41e64 docs: remove gitter from realtime chatting tools 78504d7 pkg/constants: improve GetMsgByCode test 0508f23 Merge pull request #1212 from truongnh1992/Typo 51069cb Fix markdown syntax in documentation aa99056 Merge pull request #1209 from pouchrobot/auto-doc-2020-02-06 6692474 fix typo f292c81 docs: auto generate Dragonfly cli/api docs via code e1ae4b8 Merge pull request #1181 from YanzheL/feature-https-mitm 96c3c82 Hijack HTTPS by generating leaf TLS certs on the fly issued by user provided CA 6745cc8 Merge pull request #1197 from q384566678/add-supernode-test 4c0fac1 test: add unit tests for supernode errors 9f9b328 Merge pull request #1196 from q384566678/add-errortypes-test 3564b36 test: add unit tests for pkg/errortypes 0891364 Merge pull request #1202 from truongnh1992/typo c4af904 docs: trivial fix some typos b40524a implement limit rate for stream mode c9472bb add ClientStreamWriter unit test ec355c4 refactor Downloader interface 9de0927 Merge pull request #1193 from fenggw-fnst/work 04fcd1b Merge pull request #1192 from q384566678/add-digest-test 77f128e test: add unit test case for func GetAsTime ff38f2d add unit test for Sha1 2b587f3 add unit test for Sha256 919ff03 Merge pull request #1189 from hs0210/work 2865234 test: add unit test case for func GetAsInt64 b954004 Merge pull request #1187 from hs0210/work ce118fc Merge pull request #1188 from pouchrobot/auto-doc-2020-01-17 22c04aa docs: auto generate Dragonfly cli/api docs via code 3c375d4 feature: add client stream writer for Streaming 3fc7a73 test: add unit test case for func GetAsBitset e7332a3 Merge pull request #1185 from Starnop/retry-report-piece 5d248de bugfix: retry multi times if failed to report pieces 169ff3f Merge pull request #1184 from hs0210/work cbb4d50 Merge pull request #1145 from YanzheL/configurable-logfile-path 5ad6eb3 Merge pull request #1180 from antsystem/feat/header_range_per_task 2dee5c0 fix: checkStatusCode after download 42e81a9 test: add unit test case for func GetAsMap 72929df Merge pull request #1182 from pouchrobot/auto-doc-2020-01-15 cbcb99b Merge pull request #1183 from SataQiu/clean-up-20200115 416d583 code clean: remove tmp file 22bae4d docs: auto generate Dragonfly cli/api docs via code 0f1a3a5 feat: generate task per http range c02b0b0 Set default logfile path for dfdaemon/dfget ebfc638 Do not set default path here, rollback the change. 54b4918 Fix linter issue 1a20200 Rebase master bf81246 Configure logfile path via 'logConfig.path' property def2b21 Merge pull request #1178 from q384566678/add-util-test 4b07fc0 Merge pull request #1179 from q384566678/add-dfget-test 41ab743 add unit test for GetMsgByCode 733a6e3 add unit test for Set b936211 add unit test for Add da94e78 Merge pull request #1176 from hs0210/work e2faa4f test: add unit test case for func GetAsBool c32b58a Merge pull request #1152 from Starnop/cdn-fail-source-down c17e3ca Merge pull request #1024 from Starnop/docs-config 4b0a28e docs: optimize the config documents ed99479 Merge pull request #1120 from Starnop/deploy-machine ff0e850 Merge pull request #1028 from Starnop/df-harbor ea45899 docs: add a doc about using dragonfly with harbor registry c0abca3 docs: update docs about deploy with Physical Machines 7d00289 bugfix: update cdn fail when the content is being read and the source server is down 239454c Merge pull request #1174 from truongnh1992/secure_links 810724e Securing many http links to https links a562669 Merge pull request #1156 from truongnh1992/contribution ae01fe8 docs: add more detail steps in workspace preparation 9b9ca93 Merge pull request #1172 from hs0210/work 1243876 Merge pull request #1173 from fenggw-fnst/work d19813a test: add test case in unit test TestParseFilter 34cf285 test: add unit test case for func GetAsString 9769c2c Merge pull request #1170 from hs0210/work 2fe7047 test: add unit test case for func GetAsInt 7845c18 Merge pull request #1169 from pouchrobot/auto-doc-2020-01-10 4eb5eeb docs: auto generate Dragonfly cli/api docs via code 9d8c850 Merge pull request #1166 from xujihui1985/master 7484761 Merge pull request #1165 from fenggw-fnst/work e817bf2 Merge pull request #1167 from hs0210/work 7eb5be1 fix: use StringArrayVar to parse http header a021b8a test: add unit test case for func Remove b6bf1db fix typo: fix comment for func Print b32d1ca Merge pull request #1163 from fenggw-fnst/work a432eda fix typo: fix comment for func ParseNodesString 7894da1 Merge pull request #1160 from fenggw-fnst/work 3d768bb test: add unit test case for func ParsePieceIndex 48a4736 Merge pull request #1159 from SataQiu/move-version-pkg-20200103 6d98b90 refactor: make version subcommand as a public subcommand 9aaea24 Merge pull request #1151 from pouchrobot/auto-doc-2019-12-31 3f7ee57 Merge pull request #1155 from fenggw-fnst/work e1c58f3 fix typo: 'the the' -> 'the' 2d8dec6 docs: auto generate Dragonfly cli/api docs via code 509945a Merge pull request #1065 from yunfeiyanggzq/bridge_1 f2428a5 Merge pull request #1084 from SataQiu/add-dfdaemon-config-cmd-20191118 b552a12 Merge pull request #1148 from fenggw-fnst/work d1d7b0a test: add unit test case for func TransRate df32ce2 Merge pull request #1134 from Aresforchina/Drangonflu-update02 c3b7c6b bugfix:modify the words spelling mistakes 2d0c65a Merge pull request #1137 from pouchrobot/auto-doc-2019-12-17 00538be docs: auto generate Dragonfly cli/api docs via code 39efe64 Merge pull request #1133 from Aresforchina/Dragonfly-update 5f45806 modify 1.0 bridge d33ea4c bugfix:modify the words spelling mistakes e77febc Merge pull request #1131 from pouchrobot/auto-doc-2019-12-09 3dccb97 docs: auto generate Dragonfly cli/api docs via code 6235fe1 Merge pull request #1125 from fengzixu/master a32f2fd feature: add yamllint tool to format yaml files 20dca47 Merge pull request #1127 from yeya24/fix-link 8583da4 fix bad link 096afac Merge pull request #1121 from lowzj/fix-totallimit c10fde1 fix: unset the default value of totallimit e49d466 Merge pull request #1114 from yunfeiyanggzq/bridge_3 992496d feature: add 'config default' command to print the default configuration of components 5e3fab2 feature:modify client api d1577a8 Merge pull request #1110 from yeya24/remove-check 7da1850 Merge pull request #1116 from pouchrobot/auto-doc-2019-12-02 85663db docs: auto generate Dragonfly cli/api docs via code c053e4e Merge pull request #1113 from yunfeiyanggzq/bridge_2 a404b3d bugfix:fix swagger wrong d714d98 Merge pull request #1107 from yeya24/fix-dashboard ccd900d Merge pull request #1099 from widy21/master ee8fa00 cleanup unused scripts and Makefile c56a6d2 Merge pull request #1109 from pouchrobot/auto-doc-2019-12-01 4fb7772 docs: auto generate Dragonfly cli/api docs via code 7531128 fix broken grafana dashboard b1a405c Merge pull request #1102 from Starnop/disable-autogendoc 7803b95 fix the "index out of range" error when the length of speed is 0. 6307135 bugfix: disable displaying auto generation tag in cli docs for supernode f8ecf02 Merge pull request #1100 from Starnop/supernode-cli-docs c7ae59b [feature]: add generate doc cmd for supernode bd23388 fix the "index out of range" error when the length of speed is 0. 72b98ba Merge pull request #1088 from lowzj/optimize-server-start 84c3145 Merge pull request #1089 from lowzj/fix-invalid-timeout d03b207 optimize: reduce the waiting time for starting dfget server 5633b9e bugfix: the timeout specified by user should be used firstly 04f0ffe Merge pull request #1087 from zcc35357949/fix/server_startup 8ebbc6f bugfix: lock file before check server port
Source code(tar.gz)
Source code(zip)
checksums.txt(203 bytes)
Dragonfly_1.0.5_darwin_amd64.tar.gz(16.12 MB)
Dragonfly_1.0.5_linux_amd64.tar.gz(15.47 MB)
v1.0.4(Jul 16, 2020)

Image

Images Supernode Image Link: supernode:1.0.4 Dfclient Image Link: dfclient:1.0.4

Changelog

5eb5540 bugfix: fix missing error message 'Unable to autodetect advertiser ip, please set it via --advertise-ip' related to #1428 cc2adc4 bugfix: fix dfget panic when link error between output file and tmp file. 83cdf2b bugfix the bug in unit tests about sync.Pool 032161d bugfix: infinite loop in dfget/config/config.go#checkOutput
Source code(tar.gz)
Source code(zip)
checksums.txt(203 bytes)
Dragonfly_1.0.4_darwin_amd64.tar.gz(16.18 MB)
Dragonfly_1.0.4_linux_amd64.tar.gz(15.45 MB)
v1.0.3(Jul 2, 2020)

NOTE: This release is not recommended.

Image

Images Supernode Image Link: supernode:1.0.3 Dfclient Image Link: dfclient:1.0.3

Changelog

0122fc9 optimize: release allocated ByteBuffer explicitly a347174 bugfix: cdn cannot re-download uncompleted task with header param 4bbbc80 fix: delete empty directory in local storage
Source code(tar.gz)
Source code(zip)
checksums.txt(203 bytes)
Dragonfly_1.0.3_darwin_amd64.tar.gz(16.18 MB)
Dragonfly_1.0.3_linux_amd64.tar.gz(15.45 MB)
v1.0.2(May 22, 2020)

Image

Images Supernode Image Link: supernode:1.0.2 Dfclient Image Link: dfclient:1.0.2

Changelog

b23342a fix: don't delete the parent directory of the task file 386e25f fix: pass GOPROXY to docker build bd47d0c proxy: do not log error messages when http Server is closed
Source code(tar.gz)
Source code(zip)
checksums.txt(203 bytes)
Dragonfly_1.0.2_darwin_amd64.tar.gz(16.18 MB)
Dragonfly_1.0.2_linux_amd64.tar.gz(15.44 MB)
v1.0.1(Apr 29, 2020)
Images

Supernode Image Link: supernode:1.0.1

Dfclient Image Link: dfclient:1.0.1

Changelog

bacc34b bugfix: initialize dfget downloader in dfdaemon 8cac49e bugfix: notify p2p downloader to pull next piece after reporting 5ff4530 pkg/algorithm/algorithm.go:improve algorithm 0ba788d fix: GCDSlice returns 1 when slice is empty c84e38d feature: modify dfdaemon port validate method, allow to expose more port to listen. 1da7fba add seed downloader cb47003 define the interface of seed 9814193 add bitmap in pkg, the max bitIndex is math.MaxInt32 0996a1b add seed response in seed pattern. 702937c update api definition of /peer/heartbeat and /peer/network b3222f5 add lru queue and mockFileServer 1d160be define some api types. ef57720 fix: cdn-source pattern supports range task ce27e84 remove https://coreos.com/operators/ from roadmap.md in case ci/circleci: linters-check not success 101a072 add stream mode which do not up dfget progress. 26d9d97 feature: support source cdn pattern e6a20fb feature: add dfdaemon sub command 'gen-ca' for CA generation d38af90 bugfix: check err returned by peerMgr.Get to prevent nil pointer reference 768851b update IsValidURL 523e98d add test for RegisterProtocol 92de56e support custom protocols e8c93e7 bugfix:cConn should close immediately 5da96fe fix the concurrent race of reuse of fasthttp resp body 06bdaed Fix an issue where HTTPS is not supported 17e65e3 bugfix: fix untested case due to prefix missing 8af6879 change Wight into Weight 58ccd6a pkg/fileutils: constructing errors following conventions 16763d6 pkg/fileutils: add verifications for SymbolicLink 78504d7 pkg/constants: improve GetMsgByCode test 51069cb Fix markdown syntax in documentation 96c3c82 Hijack HTTPS by generating leaf TLS certs on the fly issued by user provided CA b40524a implement limit rate for stream mode ec355c4 refactor Downloader interface 3c375d4 feature: add client stream writer for Streaming 5d248de bugfix: retry multi times if failed to report pieces 2dee5c0 fix: checkStatusCode after download 416d583 code clean: remove tmp file 0f1a3a5 feat: generate task per http range c02b0b0 Set default logfile path for dfdaemon/dfget ebfc638 Do not set default path here, rollback the change. 54b4918 Fix linter issue 1a20200 Rebase master bf81246 Configure logfile path via 'logConfig.path' property 7d00289 bugfix: update cdn fail when the content is being read and the source server is down 810724e Securing many http links to https links 7eb5be1 fix: use StringArrayVar to parse http header 6d98b90 refactor: make version subcommand as a public subcommand c3b7c6b bugfix:modify the words spelling mistakes 5f45806 modify 1.0 bridge d33ea4c bugfix:modify the words spelling mistakes a32f2fd feature: add yamllint tool to format yaml files 8583da4 fix bad link c10fde1 fix: unset the default value of totallimit 992496d feature: add 'config default' command to print the default configuration of components 5e3fab2 feature:modify client api a404b3d bugfix:fix swagger wrong ee8fa00 cleanup unused scripts and Makefile 7531128 fix broken grafana dashboard 7803b95 fix the "index out of range" error when the length of speed is 0. c7ae59b [feature]: add generate doc cmd for supernode bd23388 fix the "index out of range" error when the length of speed is 0. d03b207 optimize: reduce the waiting time for starting dfget server 5633b9e bugfix: the timeout specified by user should be used firstly 8ebbc6f bugfix: lock file before check server port
Source code(tar.gz)
Source code(zip)
checksums.txt(203 bytes)
Dragonfly_1.0.1_darwin_amd64.tar.gz(16.18 MB)
Dragonfly_1.0.1_linux_amd64.tar.gz(15.44 MB)
v1.0.0(Nov 19, 2019)
Dragonfly 1.0.0

Welcome to the v1.0.0 release of Dragonfly!

Images

Supernode Image Link: supernode:1.0.0

Dfclient Image Link: dfclient:1.0.0

Breaking Changes

feature: add weight for node config.

feature: Optimize performance by limiting the number which download from supernode for one task

feature: support gc mechanism for supernode

feature: add grafana dashboards doc and images

feature: registry mirror support direct param.

bugfix: paas tls config to dfget and supernode.

bugfix: add piece error API for 404 error

bugfix: avoid creating dup-process of dfget server.

bugfix: support truncate the file when put

refactor: handle errors gracefully

chore: add goreleaser and docker image publish

Contributors

lowzj

Starnop

liuyuxing

Allen Sun

pouchrobot

SataQiu

yeya24

zhouchencheng

Ben Ye

ZhiQiang Gu

fengzixu

inoc603

yizheng

yunfeiyangbuaa

zhouhaibing089

岁月摧锋断吴钩

Changes

d64938d refactor: handle errors gracefully

604d7ff add goreleaser and docker image publish

c531afb feature: make the log related parameter can be configured

76f2ed4 bugfix: fix running piece concurrent update wrong

1539f8c feature: add context for dfget downloader

36eb70c feature: prohibit the IPv6 addr for supernode peer IP

f2a016c bugfix: avoid creating dup-process of dfget server.

ca93c7c docs: auto generate Dragonfly cli/api docs via code

8f18a26 feat(dfdaemon): add redirect to proxy rule

d7a8323 feature: registry mirror support direct param.

dc7e3ab feature: try to download from back source even if download timeout with dragonfly

95dd1c9 feature: prohibit the IPv6 to communicate with supernode

ba9efb0 docs: auto generate Dragonfly cli/api docs via code

f1b76f5 modify the comment

9d9d9a8 bugfix: support truncate the file when put

189a6cc add cache in circleci

e907850 Squashed commit of the following:

c8619a1 docs: auto generate Dragonfly cli/api docs via code

18edd09 feature: support init from environment variables for dfdaemon and supernode

b42a14c docs: auto generate Dragonfly cli/api docs via code

744bccf Update adopters.md

0f654c1 using path/filepath instead of path for file operations

ff8d91f docs: auto generate Dragonfly cli/api docs via code

5444a8e dfdaemon: specify context when invokes dfget

Source code(tar.gz)
Source code(zip)
checksums.txt(203 bytes)
Dragonfly_1.0.0_darwin_amd64.tar.gz(15.50 MB)
Dragonfly_1.0.0_linux_amd64.tar.gz(14.79 MB)
v1.0.0-alpha(Nov 1, 2019)
Dragonfly 1.0.0-alpha

Images

Supernode Image Link: supernode:1.0.0-alpha

Supernode Image Link: dfclient:1.0.0-alpha

Breaking Changes

feature: add weight for node config.

feature: Optimize performance by limiting the number which download from supernode for one task

feature: support gc mechanism for supernode

feature: add grafana dashboards doc and images

bugfix: paas tls config to dfget and supernode.

bugfix: add piece error API for 404 error

Contributors

Allen Sun

lowzj

pouchrobot

yeya24

hwdef

Starnop

yunfeiyangbuaa

SataQiu

yuxiaobo

chentanjun

zhouchencheng

Starnop

AllenZMC

Guangming Wang

ethan

inoc603

AdamDang

Ben Ye

LiuDui

alan

dzzg

fengzixu

wangweiweir

wybzju

xichengliudui

宋文杰

Changes

dd99a67 feature: add weight for node config

fa9b6d2 feature: support set home dir in dfget config file.

a595663 bug fix dfget client timeout set error when fileLength is 0

0ec8e8d bugfix: delete the default value for supernodes from dfdaemon

856dbb9 feature: add error handling for dfget

a5f213d using GOPROXY environment variable if set instead of the default value

0381327 add supernode config

fe078b1 make config file load before init log

f1a4882 feature: add a warn log for long gc duration

1ecd2cf update pouchlinter to 0.2.4

4d50854 version: add error handling

3bf86b8 feature: upgrade golang version to 1.12.10

7d1b904 feature: enable viper for supernode

8b8c747 bugfix: remove duplicate rate.MB

6ca6927 dfget&&supernode: cleanup code based on staticcheck warnings

047e1a6 fix(dfdaemon): add error handling

349313b make cli better

a6670c7 fix(supernode/config): add error handling

1bb19a6 bugfix: set tls config if existed when dfget downloads from the source station

435631e make the struct Properties embedded into config Ref#933

e08e1a2 supernode: cleanup code based on static check warning

63bac9f clinte, dfget, supernode: delete unused functions based on static check

da13b91 update stale metrics docs

16de4be add gc metrics

b600e5d Improve the comments: modify the words spelling mistakes

c5e3339 fix(version) : add some punctuation

f43961e Improve the quality of comments in the documents

75e6931 cleanup code base: reformat some code based on staticcheck

99626a4 bugfix: gc disk will produce error logs continuously when download home dir not exist

11fdece add grafana dashboards doc and images

f72ba18 add grafana dashboard

db32623 add FAQ

de025b7 fix(surpernode/server): add error handling

8c6feee add docs about other exporters

5bf9871 feature: add supernode disk gc feature

7609dc7 fix gc bug

4fc4fea feature: add 'help' target to Makefile and make it the default

41006e9 feature: add gc count metrics

4186e93 add dot and conjunction

d037f31 bugfix: paas registry mirror tls config into dfget config.

ae7bdba doc: add instructions how to join Dragonfly as a meber

7745895 Adjust the word order of comments

3b9db8c fix: ineffective viper alias

f0dd3fb Grammatical correction

40c9b2e fix(client): add error handling in client

e317d52 fix(dfget): add err handling in dfget

e47947b feature: use constant value to replace literals in supernode config.

8a35330 test: add err handling in test

0811961 remove useless return value of gc task function

389bd6a cmd: add err handling

1fcd86a feature: add piece error API

de058d6 log: not rotate log file when startup

a91b0c8 perfect document

131435c add type rate

c3183a9 remove beego submodules and add it as a gitignore item

fcd8ec7 modify document

049eea0 perfect the document

2f03037 improve standard of the document

28b9f15 fix-up-document-format

f1e9bd6 feature: make supernode compatible with the 0.2.x dfget

3b2743f fix dfget panic

d737c54 s

a370bcf remove beego submodules

9de053d add dfget metrics

0b5656f bugfix: create home dir first when runSupernode

7f55443 fix blanklist bug

4e08341 perfect document for use

7f8fd2a rm the same func IsEmptyStr of different package

6b3f761 add gc config

7f2fc6d add supernode piece downloaded bytes

c27c70d add gc config

c6a29e0 add some cdn metrics

44c1993 delete os.O_SYNC flag when supernode write piece to local

1fe5e94 Modify the Chinese punctuation mark to the English punctuation mark

3fd6f8c Modify the Chinese punctuation mark to the English punctuation mark

6f6931e delete ertra spaces

718f2fc feature: limit the number which download from supernode for one task

701474d bugfix: add timeout for download piece

bdb42d3 delete extra slash

655c205 refactor: update some code and output logs for supernode

37ca9e4 fix staticcheck failures

f108b62 feature: enable staticcheck

63f099d refactor: new api for dflog and output log to console by default

1cbf78d fix-typo

03bc42e remove trailing spaces in dfget example

852a262 modify dfget output

0c79930 check is supernode dfget task before delete

66969d2 modify gc err

d5d5ebf bugfix: dfdaemon's config supernodes as an alias do not work.

c8c81cb add boilerplate-check into circleci

28ce102 cleanup some useless dfget config fields

f13417b feature: update the roadmap for a new stage

564290c feature: add gc mgr for supernode

3149d9d support run golangci-lint locally and in docker

d6375c1 fix mis-spelling in README.md

a1a4d20 update dfgettask status to success

fb1dbae update to golangci-lint

7174701 Use constant blocks to define constants instead of discrete definitions. Signed-off-by: xichengliudui [email protected]

29dce41 Update indentation of comments Signed-off-by: xichengliudui [email protected]

f091250 add register peers,tasks,dfgettasks counter metrics

4053d7a add peer label

aba48ed add register parameter when register metrics

3dc9888 add some supernode metrics

aa9a41e fix wrong spells in config.go

20aaa82 fix mis-spelling in multi_machines_deployment.md

0ea6869 feature: add boilerplate check script

d94e094 http_util.go: fix word in error message

9dace51 feature: supernode support download remote file by skipping secure verify or setting root ca certs.

8f8be55 storage_driver.go: property name be "offset"

853ceac feature: not return when failed to get http file Length

4c17e18 fix bug unknown profile

a0331aa bugfix: the dfget will wait forever when the final piece failed

ed5a3ef feature: add tasks create actions in client pkg of golang sdk

Source code(tar.gz)
Source code(zip)
v0.4.3(Aug 5, 2019)
Dragonfly 0.4.3

Images

Supernode Image Link: supernode:0.4.3

Supernode Image Link: dfclient:0.4.3

Breaking Changes

feature: add a dfget params item in the dfdaemon config

refactor: make common/util to be specific pkgs

feature: support non-ipv4 format node address for dfclient

feature: return supernode ip according to the value specified by peer

feature: add support for prometheus metrics

Contributors

Allen Sun

liuyuxing

lowzj

Starnop

pouchrobot

yeya24

Alan

alan

BingGuanqi

928234269

AllenZMC

Haines Chan

cd1989

dzzg

[email protected]

refname 'v0.4.2' is ambiguous.

wyb

wybzju

yunfeiyangbuaa

Changes

1599fa2 feature: add golbal dfget params in the dfdaemon config

144060c move some log level to debug

fc4099b bugfix: return error when the url is unreachable

892ad7a refactor: make common/util to be specific pkgs

6f03ff7 feature: install the swagger if it has not been installed

0da3300 feature: add access time for taskID

7113e4d optimize error description

fed30ad bugfix: the dfget will always receive fail result when change supernode

d4973f1 refactor: make p2p downloader more readable

9eb571b feature: add task update actions in client pkg of golang sdk

f01f2cb feature: add tasks delete actions in client pkg of golang sdk

10fa2be feature: add tasks info actions in client pkg of golang sdk

8d2e165 bugfix: Use a local file instead of a remote file as the url to be downloaded

621893d feature: support non-ipv4 format node address for Dragonfly

cbe57dc bugfix:modify supernode's response to dfget to resolve the json err

8526900 bugfix: fix overflow for rate limit

574eb08 add callsystem when registering

2042d74 bugfix: use access time to determine whether the file is expired

5ce8009 feature: return supernode ip according to the value specified by peer

48de7ef print help message when check parameter error

835ba13 Add options to bash scripts

1462972 bugfix: use rawUrl to interact with source server

ddfcaf4 chore: fix script to check code format

f142f4e add /metrics to swagger api

89c7518 Add support for arm32

5b0a435 add a part about targets ip address in doc

505d10b add how to add metrics and refact metrics to package common

09594c5 add build_info metrics

82ccf80 add integration-test and docs

745a06b add support for prometheus metrics

Source code(tar.gz)
Source code(zip)
df-client_0.4.3_darwin_amd64.tar.gz(16.09 MB)
df-client_0.4.3_linux_amd64.tar.gz(16.26 MB)
v0.4.2(Jul 18, 2019)
Dragonfly 0.4.2

Images

Supernode Image Link: supernode:0.4.2

Supernode Image Link: dfclient:0.4.2

Contributors

liuyuxing

Starnop

lowzj

yeya24

Allen Sun

Yizheng Jiao

inoc603

pouchrobot

yunfeiyangbuaa

Changes

9ee5566 add version and build information to both cmd and router

f4035e8 bugfix: set GO111MODULE to on when use docker build

e3961b0 bugfix: initProperties after trans the param

e43e4a2 feature: pass the value of dfdaemon verbose flag to dfget

5b5bb80 feature: enable uploader server ip and port to be specified

914c88e bugfix: update supernode Dockerfile

6f76e4a add --node when starting dfclient container

e706a09 refact pprof and add test for supernode router

8ae6c74 move to go mod

0fdb4fb feature: implement minrate option for dfget when downloading a file (#664)

Source code(tar.gz)
Source code(zip)
df-client_0.4.2_darwin_amd64.tar.gz(15.29 MB)
df-client_0.4.2_linux_amd64.tar.gz(15.52 MB)
v0.4.1(Jul 11, 2019)
Dragonfly 0.4.1

Images

Supernode Image Link: supernode:0.4.1

Supernode Image Link: dfclient:0.4.1

Breaking Changes

remove code of java-version supernode.

move to go mod.

Contributors

lowzj

Allen Sun

yeya24

liuyuxing

Starnop

pouchrobot

yunfeiyangbuaa

Yizheng Jiao

cd1989

inoc603

Changes

914c88e bugfix: update supernode Dockerfile

6f76e4a add --node when starting dfclient container

e706a09 refact pprof and add test for supernode router

8ae6c74 move to go mod

0fdb4fb feature: implement minrate option for dfget when downloading a file (#664)

3b2b248 bugfix: update Dockerfile of supernode

767cf87 fix: insecure for https proxy

27113ae chore: remove commands about building java-version supernode

e058b2a refactor: remove integration testing of java-version

99a7a05 refactor: remove code of java-version supernode

1bae464 fix gofmt

3386f6c update pouchlinter

526756c fix: unit-test failed in file_util_test.go caused by user permission

0e03f12 fix: upgrade tomcat-embed-core to 9.0.21 to fix security alert

c4cda9e bump to golang 1.12.6

2a431f4 fix bug in unit test

379d175 test: add unit tests for dfdaemon/config

Source code(tar.gz)
Source code(zip)
df-client_0.4.1_darwin_amd64.tar.gz(13.67 MB)
df-client_0.4.1_linux_amd64.tar.gz(13.86 MB)
v0.4.0(Jun 20, 2019)
Dragonfly 0.4.0

Images

Supernode Image Link: dragonflyoss/supernode:0.4.0

Supernode Image Link: dragonflyoss/dfclient:0.4.0

Breaking Changes

refactor supernode with GoLang

pluggable strategy and refactor file transfering and scheduling with plugins

Contributors

Allen Sun

@AnJia

Haosdent Huang

Hui Zhao

Jerry Chong

@KevinBetterQ

Ryan McCormick

@Starnop

@cd1989

@inoc603

@lowzj

@lurenjia528

@silenceshell

@yeya24

@yuansmin

zhoulin xie

陈潮兵

Changes

6acbac4 bugfix: increase load of the peer serving for others

eca0cc0 bugfix: update the client progress after success to schedule

56db6a6 fix: send http request with timeout

99a1f03 log: reduce info logs of supernode

d9295de feature: add stat_linux_arm64.go to support for building on arm

cc2e79f bugfix: fix NPE in FileMd5NotMatchHandler#getMd5FromMeta

6e3d4eb bugfix: parse the lastmodified header to timestamp

8d4e48f feature: store piece md5s with native bytes

1fbf9e1 bugfix: do not add the superPID to pieceState

bf49501 feature: add piece length for task md5 file

cfd8990 bugfix: return http file length as the file length of register resp

287bc76 feature: add supernode Dockerfile

b0f9c03 bugfix: do not use -1 as the basis for successful conversion

a646a64 refactor: remove the global package in dfdaemon

f5a5eb3 feature: make the download port configurable

69e1cba feature: uploader will not automically exit when alivetime is set to <=0

ea0cb74 docs: remove useless contents

7180deb feature: update task register header to slice not map

32172d5 feature: parse the port of supernode from --node flag of dfget

77b62c9 fix: link to adopoter in readme

9edf174 docs: add caicloud as adopter

1eb60ed feature: dfget reports client error to supernode

ba6de46 feature: handle client error and repair

05ce112 docs: add eleme to adopters

e66e9ac feature: update integration test with supernode from java to go

7e75c51 feature: delete the repo prefix for download home path

c7fffee feature: return updateTaskInfo with cdnStatusFailed when http download gets error

c80179a feature: add interface for v0.3

36fba9a feature: implement the task mgr

6df0378 feature: use util.LockerPool instead of local storage locker

79f30f2 bugfix: add locker util and fix thread-unsafe problems for cdn

063f5b4 fix: handle redirect at most 20 times

98c7e7e bugfix: calculate realHTTPFileLength which is different realFileLength

f661e78 bugfix: use a public limiter for cdn manager

210526a feature: add dfdaemon.Server

91a1707 feature: add file server port flag

dc968e3 feature: add GetHTTPPath function for cdn mgr

b5beae1 feature: delete unused function for passing CI

8bd5534 feature: return task update info

5656e0e feature: implement the cdn manager super writer

7d583c0 feature: implement the cdn manager cdn reporter

fc3c71d feature: implement the cdn manager detect cache

9fb5911 feature: implement the cdn manager cdn reader

eddaaeb feature: implement the cdn manager file metadata manager

1e17e89 feature: implement the cdn manager path util

4883324 feature: implement the cdn manager downloader

d1802a9 feature: implement the cdn manager framework

8f880b4 feature: store mgr return localStorage as default store

7f9a7d3 bugfix: update the response code num

4e8b62c fix: BackDownloader only download file when response is less than 400

720fe25 docs: add ecosystem adopters(shopee.com) for Dragonfly

b7a8965 docs: add three adopters into project

760b6b0 docs: get all dfget logs of a task in dfclient.log

950ee65 feature: update scheduler manager GetSuperPID

0cc2d63 logs: add some debug logs with progress mgr for better debug

acce42b Configure InsecureSkipVerify for proxied https host

6a0816d Update Dockerfile for better caching

2fca259 hijack https requests with self-signed certificates

6165ffb feature: add function GetCIDByPeerIDAndTaskID for dfgetTask

bd18c86 feature: add function GetAsString for SyncMap

3eadcb5 feature: add realMd5 field for taskInfo

03e4be9 feature: use bucket as the parent dir of key

dde1467 bugfix: use advertise-ip as the supernode ip when specified

93bd8a2 feature: implement function of IsSuperCID and IsSuperPID

ed72898 feature: add a function to send a head request to get file length

049bcce feature: add a function to checks if the source url support partial requests

3668427 feature: implement the http util IsExpired

e1d7980 feature: register the supernode as a peer before start the server

1252f32 feature: add AdvertiseIP flag for supernode

18740da doc: add proxy doc

379df68 docs: correct some abused phrase

dd7a4e6 fix: fix ineffectual assignments

30c42ff bugfix: fix the bug of local storage

626f42d fix: remove redundant code in supernode/store/loacl_storage.go

e416a43 docs: add more adopters of Dragonfly

55df158 docs: add CII for project to add CII Best Practices Badge

33cc6aa feature: add interfaces about pieces md5

5bdd16a bugfix: update the fileds of store.Raw to uppercase

4e82e8e feature: add httpFileLength field for taskInfo

cf3c601 feature: implement the scheculer mgr

7bd00c4 fix: upgrade org.apache.tomcat.embed:tomcat-embed-core to 9.0.17

3c6bfd3 feature: add mock mgr for the convenience of testing

d4c3fba bugfix: no need to return error if the result is nil when get pieces

580bf38 feature: implement an empty scheduler and update the interface

d006c38 feature: add transparent proxy to extend dfdaemon's proxy mode

6d4b4ac feature: implement the progress mgr for get info

8e61a85 doc: config multiple supernodes for dfget

aeeacf5 bugfix: unify the port type to int32

31f5e13 refactor: update the interface of task mgr

1731891 Revert "doc: config multiple supernodes for dfget"

a37f861 doc: config multiple supernodes for dfget

d627112 refactor: refactor the supernode daemon dir

a4cb482 feature: implement progress mgr

ee21c90 refactor: refactor the http utils as a common code

4848e30 comments: update the comments of ConfigFiles of dfget

5b226ec feature: Add an empty implementation of the CDN manager

b88cc5d feature: make supernode configurable for dfget with dfdaemon

fad984d refactor: make errors in common folder

035d304 fix: fix the default download path of supernode

8abd78c feature: implement plugin framework of Dragonfly

6ed5d4d added docker run desc

6bb4262 feature: implement the dfgetTaskMgr

781c956 feature: implement the peer mgr

55bcc20 bugfix: add context for the interface of storage driver

ce376d7 interface ensuring

001bc1f refactor: extract the common const code to the common folder

828cddb fix supernode default port doc

980e1e1 feature: add storage driver

1de7ee2 feature: add some utils that supernode will use

b552340 refactor: Extract the common utils to the common folder

Source code(tar.gz)
Source code(zip)
df-client_0.4.0_darwin_amd64.tar.gz(8.65 MB)
df-client_0.4.0_linux_amd64.tar.gz(8.65 MB)
v0.3.1(Jun 6, 2019)
Image

Supernode: dragonflyoss/supernode:0.3.1 Dfclient: dragonflyoss/dfclient:0.3.1

Release

feature: parse the port of supernode from --node flag of dfget

feature: dfget reports client error to supernode

feature: handle client error and repair

feature: make supernode configurable for dfget with dfdaemon

feature: add transparent proxy to extend dfdaemon's proxy mode

feature: hijack https requests with self-signed certificates

fix: handle redirect at most 20 times

fix: BackDownloader only download file when response is less than 400

Contributors

@inoc603 @lowzj @Starnop
Source code(tar.gz)
Source code(zip)
df-client_0.3.1_darwin_amd64.tar.gz(7.83 MB)
df-client_0.3.1_linux_amd64.tar.gz(7.89 MB)
v0.3.0(Mar 4, 2019)
Supernode Image

China: registry.cn-hangzhou.aliyuncs.com/dragonflyoss/supernode:0.3.0 USA: registry.us-west-1.aliyuncs.com/dragonflyoss/supernode:0.3.0

Release

Dragonfly client written by GoLang is released

supports multi-registries, https and private registry

build rpm and deb packages for dragonfly client

integration testing framework

unify the developing and testing environment

generate API/CLI docs automatically

Contributors

@allencloud @chenchaobing @fengzixu @godliness @inoc603 @JoeWrightss @liuhengloveyou @lowzj @Shawnpku @silenceshell @Starnop @zousheng @ZYecho
Source code(tar.gz)
Source code(zip)
df-client_0.3.0_darwin_amd64.tar.gz(7.77 MB)
df-client_0.3.0_linux_amd64.tar.gz(7.83 MB)
v0.2.1(Feb 13, 2019)
Supernode Image

China: registry.cn-hangzhou.aliyuncs.com/dragonflyoss/supernode:0.2.1 USA: registry.us-west-1.aliyuncs.com/dragonflyoss/supernode:0.2.1

Bugfix

#364 Fix rotating log makes dfdaemon.log file huge size.

#351 Fix cannot use Dragonfly to download an empty file.

Source code(tar.gz)
Source code(zip)
df-client_0.2.1_darwin_amd64.tar.gz(5.72 MB)
df-client_0.2.1_linux_amd64.tar.gz(5.76 MB)
v0.2.0(Aug 3, 2018)
SuperNode container image

China: registry.cn-hangzhou.aliyuncs.com/alidragonfly/supernode:0.2.0

USA: registry.us-west-1.aliyuncs.com/alidragonfly/supernode:0.2.0

Client binary package

You could download the binary package according to your system from the attachment.
Source code(tar.gz)
Source code(zip)
df-client_0.2.0_darwin_amd64.tar.gz(5.72 MB)
df-client_0.2.0_linux_amd64.tar.gz(5.76 MB)
v0.1.1(Jun 27, 2018)

bugfix: #93
Source code(tar.gz)
Source code(zip)
v0.1.0(Jun 6, 2018)

change log: Milestone v0.1.0
Source code(tar.gz)
Source code(zip)
v0.0.1(May 23, 2018)

Source code(tar.gz)
Source code(zip)