SeaweedFS is a distributed storage system for blobs, objects, files, and data warehouse, to store and serve billions of files fast! Blob store has O(1) disk seek, local tiering, cloud tiering. Filer supports cross-cluster active-active replication, Kubernetes, POSIX, S3 API, encryption, Erasure Coding for warm storage, FUSE mount, Hadoop, WebDAV.

Overview

SeaweedFS

Slack Twitter Build Status GoDoc Wiki Docker Pulls

SeaweedFS Logo

Sponsor SeaweedFS via Patreon

SeaweedFS is an independent Apache-licensed open source project with its ongoing development made possible entirely thanks to the support of these awesome backers. If you'd like to grow SeaweedFS even stronger, please consider joining our sponsors on Patreon.

Your support will be really appreciated by me and other supporters!

Gold Sponsors

shuguang


Table of Contents

Quick Start

  • Download the latest binary from https://github.com/chrislusf/seaweedfs/releases and unzip a single binary file weed or weed.exe
  • Run weed server -dir=/some/data/dir -s3 to start one master, one volume server, one filer, and one S3 gateway.

Also, to increase capacity, just add more volume servers by running weed volume -dir="/some/data/dir2" -mserver="<master_host>:9333" -port=8081 locally, or on a different machine, or on thoudsands of machines. That is it!

Introduction

SeaweedFS is a simple and highly scalable distributed file system. There are two objectives:

  1. to store billions of files!
  2. to serve the files fast!

SeaweedFS started as an Object Store to handle small files efficiently. Instead of managing all file metadata in a central master, the central master only manages volumes on volume servers, and these volume servers manage files and their metadata. This relieves concurrency pressure from the central master and spreads file metadata into volume servers, allowing faster file access (O(1), usually just one disk read operation).

SeaweedFS can transparently integrate with the cloud. With hot data on local cluster, and warm data on the cloud with O(1) access time, SeaweedFS can achieve both fast local access time and elastic cloud storage capacity. What's more, the cloud storage access API cost is minimized. Faster and Cheaper than direct cloud storage! Signup for future managed SeaweedFS cluster offering at "seaweedfilesystem at gmail dot com".

There is only 40 bytes of disk storage overhead for each file's metadata. It is so simple with O(1) disk reads that you are welcome to challenge the performance with your actual use cases.

SeaweedFS started by implementing Facebook's Haystack design paper. Also, SeaweedFS implements erasure coding with ideas from f4: Facebook’s Warm BLOB Storage System, and has a lot of similarities with Facebook’s Tectonic Filesystem

On top of the object store, optional Filer can support directories and POSIX attributes. Filer is a separate linearly-scalable stateless server with customizable metadata stores, e.g., MySql, Postgres, Redis, Cassandra, HBase, Mongodb, Elastic Search, LevelDB, RocksDB, MemSql, TiDB, Etcd, CockroachDB, etc.

For any distributed key value stores, the large values can be offloaded to SeaweedFS. With the fast access speed and linearly scalable capacity, SeaweedFS can work as a distributed Key-Large-Value store.

Back to TOC

Additional Features

  • Can choose no replication or different replication levels, rack and data center aware.
  • Automatic master servers failover - no single point of failure (SPOF).
  • Automatic Gzip compression depending on file mime type.
  • Automatic compaction to reclaim disk space after deletion or update.
  • Automatic entry TTL expiration.
  • Any server with some disk spaces can add to the total storage space.
  • Adding/Removing servers does not cause any data re-balancing unless triggered by admin commands.
  • Optional picture resizing.
  • Support ETag, Accept-Range, Last-Modified, etc.
  • Support in-memory/leveldb/readonly mode tuning for memory/performance balance.
  • Support rebalancing the writable and readonly volumes.
  • Customizable Multiple Storage Tiers: Customizable storage disk types to balance performance and cost.
  • Transparent cloud integration: unlimited capacity via tiered cloud storage for warm data.
  • Erasure Coding for warm storage Rack-Aware 10.4 erasure coding reduces storage cost and increases availability.

Back to TOC

Filer Features

Kubernetes

Back to TOC

Example: Using Seaweed Object Store

By default, the master node runs on port 9333, and the volume nodes run on port 8080. Let's start one master node, and two volume nodes on port 8080 and 8081. Ideally, they should be started from different machines. We'll use localhost as an example.

SeaweedFS uses HTTP REST operations to read, write, and delete. The responses are in JSON or JSONP format.

Start Master Server

> ./weed master

Start Volume Servers

> weed volume -dir="/tmp/data1" -max=5  -mserver="localhost:9333" -port=8080 &
> weed volume -dir="/tmp/data2" -max=10 -mserver="localhost:9333" -port=8081 &

Write File

To upload a file: first, send a HTTP POST, PUT, or GET request to /dir/assign to get an fid and a volume server url:

> curl http://localhost:9333/dir/assign
{"count":1,"fid":"3,01637037d6","url":"127.0.0.1:8080","publicUrl":"localhost:8080"}

Second, to store the file content, send a HTTP multi-part POST request to url + '/' + fid from the response:

> curl -F [email protected]/home/chris/myphoto.jpg http://127.0.0.1:8080/3,01637037d6
{"name":"myphoto.jpg","size":43234,"eTag":"1cc0118e"}

To update, send another POST request with updated file content.

For deletion, send an HTTP DELETE request to the same url + '/' + fid URL:

> curl -X DELETE http://127.0.0.1:8080/3,01637037d6

Save File Id

Now, you can save the fid, 3,01637037d6 in this case, to a database field.

The number 3 at the start represents a volume id. After the comma, it's one file key, 01, and a file cookie, 637037d6.

The volume id is an unsigned 32-bit integer. The file key is an unsigned 64-bit integer. The file cookie is an unsigned 32-bit integer, used to prevent URL guessing.

The file key and file cookie are both coded in hex. You can store the <volume id, file key, file cookie> tuple in your own format, or simply store the fid as a string.

If stored as a string, in theory, you would need 8+1+16+8=33 bytes. A char(33) would be enough, if not more than enough, since most uses will not need 2^32 volumes.

If space is really a concern, you can store the file id in your own format. You would need one 4-byte integer for volume id, 8-byte long number for file key, and a 4-byte integer for the file cookie. So 16 bytes are more than enough.

Read File

Here is an example of how to render the URL.

First look up the volume server's URLs by the file's volumeId:

> curl http://localhost:9333/dir/lookup?volumeId=3
{"volumeId":"3","locations":[{"publicUrl":"localhost:8080","url":"localhost:8080"}]}

Since (usually) there are not too many volume servers, and volumes don't move often, you can cache the results most of the time. Depending on the replication type, one volume can have multiple replica locations. Just randomly pick one location to read.

Now you can take the public url, render the url or directly read from the volume server via url:

 http://localhost:8080/3,01637037d6.jpg

Notice we add a file extension ".jpg" here. It's optional and just one way for the client to specify the file content type.

If you want a nicer URL, you can use one of these alternative URL formats:

 http://localhost:8080/3/01637037d6/my_preferred_name.jpg
 http://localhost:8080/3/01637037d6.jpg
 http://localhost:8080/3,01637037d6.jpg
 http://localhost:8080/3/01637037d6
 http://localhost:8080/3,01637037d6

If you want to get a scaled version of an image, you can add some params:

http://localhost:8080/3/01637037d6.jpg?height=200&width=200
http://localhost:8080/3/01637037d6.jpg?height=200&width=200&mode=fit
http://localhost:8080/3/01637037d6.jpg?height=200&width=200&mode=fill

Rack-Aware and Data Center-Aware Replication

SeaweedFS applies the replication strategy at a volume level. So, when you are getting a file id, you can specify the replication strategy. For example:

curl http://localhost:9333/dir/assign?replication=001

The replication parameter options are:

000: no replication
001: replicate once on the same rack
010: replicate once on a different rack, but same data center
100: replicate once on a different data center
200: replicate twice on two different data center
110: replicate once on a different rack, and once on a different data center

More details about replication can be found on the wiki.

You can also set the default replication strategy when starting the master server.

Allocate File Key on Specific Data Center

Volume servers can be started with a specific data center name:

 weed volume -dir=/tmp/1 -port=8080 -dataCenter=dc1
 weed volume -dir=/tmp/2 -port=8081 -dataCenter=dc2

When requesting a file key, an optional "dataCenter" parameter can limit the assigned volume to the specific data center. For example, this specifies that the assigned volume should be limited to 'dc1':

 http://localhost:9333/dir/assign?dataCenter=dc1

Other Features

Back to TOC

Architecture

Usually distributed file systems split each file into chunks, a central master keeps a mapping of filenames, chunk indices to chunk handles, and also which chunks each chunk server has.

The main drawback is that the central master can't handle many small files efficiently, and since all read requests need to go through the chunk master, so it might not scale well for many concurrent users.

Instead of managing chunks, SeaweedFS manages data volumes in the master server. Each data volume is 32GB in size, and can hold a lot of files. And each storage node can have many data volumes. So the master node only needs to store the metadata about the volumes, which is a fairly small amount of data and is generally stable.

The actual file metadata is stored in each volume on volume servers. Since each volume server only manages metadata of files on its own disk, with only 16 bytes for each file, all file access can read file metadata just from memory and only needs one disk operation to actually read file data.

For comparison, consider that an xfs inode structure in Linux is 536 bytes.

Master Server and Volume Server

The architecture is fairly simple. The actual data is stored in volumes on storage nodes. One volume server can have multiple volumes, and can both support read and write access with basic authentication.

All volumes are managed by a master server. The master server contains the volume id to volume server mapping. This is fairly static information, and can be easily cached.

On each write request, the master server also generates a file key, which is a growing 64-bit unsigned integer. Since write requests are not generally as frequent as read requests, one master server should be able to handle the concurrency well.

Write and Read files

When a client sends a write request, the master server returns (volume id, file key, file cookie, volume node url) for the file. The client then contacts the volume node and POSTs the file content.

When a client needs to read a file based on (volume id, file key, file cookie), it asks the master server by the volume id for the (volume node url, volume node public url), or retrieves this from a cache. Then the client can GET the content, or just render the URL on web pages and let browsers fetch the content.

Please see the example for details on the write-read process.

Storage Size

In the current implementation, each volume can hold 32 gibibytes (32GiB or 8x2^32 bytes). This is because we align content to 8 bytes. We can easily increase this to 64GiB, or 128GiB, or more, by changing 2 lines of code, at the cost of some wasted padding space due to alignment.

There can be 4 gibibytes (4GiB or 2^32 bytes) of volumes. So the total system size is 8 x 4GiB x 4GiB which is 128 exbibytes (128EiB or 2^67 bytes).

Each individual file size is limited to the volume size.

Saving memory

All file meta information stored on an volume server is readable from memory without disk access. Each file takes just a 16-byte map entry of <64bit key, 32bit offset, 32bit size>. Of course, each map entry has its own space cost for the map. But usually the disk space runs out before the memory does.

Tiered Storage to the cloud

The local volume servers are much faster, while cloud storages have elastic capacity and are actually more cost-efficient if not accessed often (usually free to upload, but relatively costly to access). With the append-only structure and O(1) access time, SeaweedFS can take advantage of both local and cloud storage by offloading the warm data to the cloud.

Usually hot data are fresh and warm data are old. SeaweedFS puts the newly created volumes on local servers, and optionally upload the older volumes on the cloud. If the older data are accessed less often, this literally gives you unlimited capacity with limited local servers, and still fast for new data.

With the O(1) access time, the network latency cost is kept at minimum.

If the hot/warm data is split as 20/80, with 20 servers, you can achieve storage capacity of 100 servers. That's a cost saving of 80%! Or you can repurpose the 80 servers to store new data also, and get 5X storage throughput.

Back to TOC

Compared to Other File Systems

Most other distributed file systems seem more complicated than necessary.

SeaweedFS is meant to be fast and simple, in both setup and operation. If you do not understand how it works when you reach here, we've failed! Please raise an issue with any questions or update this file with clarifications.

SeaweedFS is constantly moving forward. Same with other systems. These comparisons can be outdated quickly. Please help to keep them updated.

Back to TOC

Compared to HDFS

HDFS uses the chunk approach for each file, and is ideal for storing large files.

SeaweedFS is ideal for serving relatively smaller files quickly and concurrently.

SeaweedFS can also store extra large files by splitting them into manageable data chunks, and store the file ids of the data chunks into a meta chunk. This is managed by "weed upload/download" tool, and the weed master or volume servers are agnostic about it.

Back to TOC

Compared to GlusterFS, Ceph

The architectures are mostly the same. SeaweedFS aims to store and read files fast, with a simple and flat architecture. The main differences are

  • SeaweedFS optimizes for small files, ensuring O(1) disk seek operation, and can also handle large files.
  • SeaweedFS statically assigns a volume id for a file. Locating file content becomes just a lookup of the volume id, which can be easily cached.
  • SeaweedFS Filer metadata store can be any well-known and proven data stores, e.g., Redis, Cassandra, HBase, Mongodb, Elastic Search, MySql, Postgres, MemSql, TiDB, CockroachDB, Etcd etc, and is easy to customized.
  • SeaweedFS Volume server also communicates directly with clients via HTTP, supporting range queries, direct uploads, etc.
System File Metadata File Content Read POSIX REST API Optimized for large number of small files
SeaweedFS lookup volume id, cacheable O(1) disk seek Yes Yes
SeaweedFS Filer Linearly Scalable, Customizable O(1) disk seek FUSE Yes Yes
GlusterFS hashing FUSE, NFS
Ceph hashing + rules FUSE Yes
MooseFS in memory FUSE No
MinIO separate meta file for each file Yes No

Back to TOC

Compared to GlusterFS

GlusterFS stores files, both directories and content, in configurable volumes called "bricks".

GlusterFS hashes the path and filename into ids, and assigned to virtual volumes, and then mapped to "bricks".

Back to TOC

Compared to MooseFS

MooseFS chooses to neglect small file issue. From moosefs 3.0 manual, "even a small file will occupy 64KiB plus additionally 4KiB of checksums and 1KiB for the header", because it "was initially designed for keeping large amounts (like several thousands) of very big files"

MooseFS Master Server keeps all meta data in memory. Same issue as HDFS namenode.

Back to TOC

Compared to Ceph

Ceph can be setup similar to SeaweedFS as a key->blob store. It is much more complicated, with the need to support layers on top of it. Here is a more detailed comparison

SeaweedFS has a centralized master group to look up free volumes, while Ceph uses hashing and metadata servers to locate its objects. Having a centralized master makes it easy to code and manage.

Same as SeaweedFS, Ceph is also based on the object store RADOS. Ceph is rather complicated with mixed reviews.

Ceph uses CRUSH hashing to automatically manage the data placement, which is efficient to locate the data. But the data has to be placed according to the CRUSH algorithm. Any wrong configuration would cause data loss. SeaweedFS places data by assigning them to any writable volumes. If writes to one volume failed, just pick another volume to write. Adding more volumes are also as simple as it can be.

SeaweedFS is optimized for small files. Small files are stored as one continuous block of content, with at most 8 unused bytes between files. Small file access is O(1) disk read.

SeaweedFS Filer uses off-the-shelf stores, such as MySql, Postgres, Mongodb, Redis, Elastic Search, Cassandra, HBase, MemSql, TiDB, CockroachCB, Etcd, to manage file directories. These stores are proven, scalable, and easier to manage.

SeaweedFS comparable to Ceph advantage
Master MDS simpler
Volume OSD optimized for small files
Filer Ceph FS linearly scalable, Customizable, O(1) or O(logN)

Back to TOC

Compared to MinIO

MinIO follows AWS S3 closely and is ideal for testing for S3 API. It has good UI, policies, versionings, etc. SeaweedFS is trying to catch up here. It is also possible to put MinIO as a gateway in front of SeaweedFS later.

MinIO metadata are in simple files. Each file write will incur extra writes to corresponding meta file.

MinIO does not have optimization for lots of small files. The files are simply stored as is to local disks. Plus the extra meta file and shards for erasure coding, it only amplifies the LOSF problem.

MinIO has multiple disk IO to read one file. SeaweedFS has O(1) disk reads, even for erasure coded files.

MinIO has full-time erasure coding. SeaweedFS uses replication on hot data for faster speed and optionally applies erasure coding on warm data.

MinIO does not have POSIX-like API support.

MinIO has specific requirements on storage layout. It is not flexible to adjust capacity. In SeaweedFS, just start one volume server pointing to the master. That's all.

Dev Plan

  • More tools and documentation, on how to manage and scale the system.
  • Read and write stream data.
  • Support structured data.

This is a super exciting project! And we need helpers and support!

Back to TOC

Installation Guide

Installation guide for users who are not familiar with golang

Step 1: install go on your machine and setup the environment by following the instructions at:

https://golang.org/doc/install

make sure you set up your $GOPATH

Step 2: checkout this repo:

git clone https://github.com/chrislusf/seaweedfs.git

Step 3: download, compile, and install the project by executing the following command

make install

Once this is done, you will find the executable "weed" in your $GOPATH/bin directory

Back to TOC

Disk Related Topics

Hard Drive Performance

When testing read performance on SeaweedFS, it basically becomes a performance test of your hard drive's random read speed. Hard drives usually get 100MB/s~200MB/s.

Solid State Disk

To modify or delete small files, SSD must delete a whole block at a time, and move content in existing blocks to a new block. SSD is fast when brand new, but will get fragmented over time and you have to garbage collect, compacting blocks. SeaweedFS is friendly to SSD since it is append-only. Deletion and compaction are done on volume level in the background, not slowing reading and not causing fragmentation.

Back to TOC

Benchmark

My Own Unscientific Single Machine Results on Mac Book with Solid State Disk, CPU: 1 Intel Core i7 2.6GHz.

Write 1 million 1KB file:

Concurrency Level:      16
Time taken for tests:   66.753 seconds
Complete requests:      1048576
Failed requests:        0
Total transferred:      1106789009 bytes
Requests per second:    15708.23 [#/sec]
Transfer rate:          16191.69 [Kbytes/sec]

Connection Times (ms)
              min      avg        max      std
Total:        0.3      1.0       84.3      0.9

Percentage of the requests served within a certain time (ms)
   50%      0.8 ms
   66%      1.0 ms
   75%      1.1 ms
   80%      1.2 ms
   90%      1.4 ms
   95%      1.7 ms
   98%      2.1 ms
   99%      2.6 ms
  100%     84.3 ms

Randomly read 1 million files:

Concurrency Level:      16
Time taken for tests:   22.301 seconds
Complete requests:      1048576
Failed requests:        0
Total transferred:      1106812873 bytes
Requests per second:    47019.38 [#/sec]
Transfer rate:          48467.57 [Kbytes/sec]

Connection Times (ms)
              min      avg        max      std
Total:        0.0      0.3       54.1      0.2

Percentage of the requests served within a certain time (ms)
   50%      0.3 ms
   90%      0.4 ms
   98%      0.6 ms
   99%      0.7 ms
  100%     54.1 ms

Back to TOC

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

The text of this page is available for modification and reuse under the terms of the Creative Commons Attribution-Sharealike 3.0 Unported License and the GNU Free Documentation License (unversioned, with no invariant sections, front-cover texts, or back-cover texts).

Back to TOC

Stargazers over time

Stargazers over time

Issues
  • Lots of

    Lots of "volume_server_handlers_read.go:75] request /661,33289a3ecd5edd with unmaching cookie seen: 1053646557 expected: 1674074305 " error log in PROD env

    any tips to find the problem?

    when users access some files of 10TB files, they will get 404 error, to check the logs:

    I0223 06:37:10 12031 topology_vacuum.go:90] check vacuum on collection:web volume:769
    I0223 06:37:10 12031 topology_vacuum.go:90] check vacuum on collection:web volume:770
    I0223 06:37:10 12031 topology_vacuum.go:90] check vacuum on collection:web volume:771
    I0223 06:37:10 12031 topology_vacuum.go:90] check vacuum on collection:web volume:765
    I0223 06:37:10 12031 topology_vacuum.go:90] check vacuum on collection:web volume:766
    I0223 06:37:38 12067 volume_server_handlers_read.go:69] read error: Not Found /661,33289a3ecd5edd_1
    I0223 06:37:38 12067 volume_server_handlers_read.go:75] request /661,33289a3ecd5edd with unmaching cookie seen: 1053646557 expected: 1674074305 from 192.1
    68.254.8:37448 agent Apache-HttpClient/4.5.3 (Java/1.8.0_131)
    I0223 06:37:38 12067 volume_server_handlers_read.go:75] request /661,33289a3ecd5edd with unmaching cookie seen: 1053646557 expected: 1674074305 from 127.0
    .0.1:43628 agent Dalvik/2.1.0 (Linux; U; Android 6.0.1; OPPO A57 Build/MMB29M)
    I0223 06:38:26 12067 volume_server_handlers_read.go:69] read error: Not Found /767,2b718ceb08e455_1
    I0223 06:38:26 12067 volume_server_handlers_read.go:75] request /767,2b718ceb08e455 with unmaching cookie seen: 3943228501 expected: 3983295080 from 192.1
    68.254.8:37652 agent Apache-HttpClient/4.5.3 (Java/1.8.0_131)
    I0223 06:38:26 12067 volume_server_handlers_read.go:75] request /767,2b718ceb08e455 with unmaching cookie seen: 3943228501 expected: 3983295080 from 127.0
    .0.1:43766 agent Mozilla/5.0 (Linux; Android 6.0.1; vivo Y53 Build/MMB29M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.2987.132 MQQ
    Browser/6.2 TBS/043906 Mobile Safari/537.36 MicroMessenger/6.6.3.1240(0x26060339) NetType/4G Language/zh_CN
    I0223 06:39:02 12067 volume_server_handlers_read.go:69] read error: Not Found /661,33289a3ecd5edd_1
    I0223 06:39:02 12067 volume_server_handlers_read.go:75] request /661,33289a3ecd5edd with unmaching cookie seen: 1053646557 expected: 1674074305 from 192.1
    68.254.8:37816 agent Apache-HttpClient/4.5.3 (Java/1.8.0_131)
    I0223 06:39:02 12067 volume_server_handlers_read.go:75] request /661,33289a3ecd5edd with unmaching cookie seen: 1053646557 expected: 1674074305 from 127.0
    .0.1:43874 agent Dalvik/2.1.0 (Linux; U; Android 6.0.1; OPPO A57 Build/MMB29M)
    I0223 06:40:00 12067 volume_server_handlers_read.go:75] request /765/36b2ca065ca535.png with unmaching cookie seen: 106734901 expected: 3831334744 from 12
    7.0.0.1:44042 agent Mozilla/5.0 (iPhone; CPU iPhone OS 11_2_6 like Mac OS X) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0 Mobile/15D100 Safari/604
    .1
    I0223 06:41:07 12067 volume_server_handlers_read.go:75] request /765/2b717f6f406291.png with unmaching cookie seen: 1866490513 expected: 2413760694 from 1
    27.0.0.1:44234 agent Mozilla/5.0 (iPhone; CPU iPhone OS 11_2_5 like Mac OS X) AppleWebKit/604.5.6 (KHTML, like Gecko) Mobile/15D60 MicroMessenger/6.6.3 Ne
    tType/WIFI Language/zh_CN
    I0223 06:41:37 12067 volume_server_handlers_read.go:69] read error: Not Found /661,33289a3ecd5edd_1
    I0223 06:41:37 12067 volume_server_handlers_read.go:75] request /661,33289a3ecd5edd with unmaching cookie seen: 1053646557 expected: 1674074305 from 192.1
    68.254.8:38434 agent Apache-HttpClient/4.5.3 (Java/1.8.0_131)
    I0223 06:41:37 12067 volume_server_handlers_read.go:75] request /661,33289a3ecd5edd with unmaching cookie seen: 1053646557 expected: 1674074305 from 127.0
    .0.1:44318 agent CH999/2.9.9 CFNetwork/889.9 Darwin/17.2.0
    I0223 06:42:11 12067 volume_server_handlers_read.go:75] request /659/334073b80964a9.jpg with unmaching cookie seen: 3087623337 expected: 387060166 from 12
    7.0.0.1:44422 agent Mozilla/5.0 (Linux; Android 7.0; SM-G9508 Build/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.2987.132 MQ
    QBrowser/6.2 TBS/043906 Mobile Safari/537.36 MicroMessenger/6.6.3.1260(0x26060339) NetType/WIFI Language/zh_CN
    I0223 06:42:11 12067 volume_server_handlers_read.go:75] request /660/31c4f2ae71121f.jpg with unmaching cookie seen: 2926645791 expected: 269826002 from 12
    7.0.0.1:44422 agent Mozilla/5.0 (Linux; Android 7.0; SM-G9508 Build/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.2987.132 MQ
    QBrowser/6.2 TBS/043906 Mobile Safari/537.36 MicroMessenger/6.6.3.1260(0x26060339) NetType/WIFI Language/zh_CN
    I0223 06:42:31 12067 volume_server_handlers_read.go:75] request /765/36b2ca065ca535.png with unmaching cookie seen: 106734901 expected: 3831334744 from 12
    7.0.0.1:44480 agent Mozilla/5.0 (iPhone; CPU iPhone OS 11_2_6 like Mac OS X) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0 Mobile/15D100 Safari/604
    .1
    I0223 06:42:34 12067 volume_server_handlers_read.go:75] request /659/334073b80964a9.jpg with unmaching cookie seen: 3087623337 expected: 387060166 from 12
    7.0.0.1:44480 agent Mozilla/5.0 (Linux; Android 7.0; SM-G9508 Build/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.2987.132 MQ
    QBrowser/6.2 TBS/043906 Mobile Safari/537.36 MicroMessenger/6.6.3.1260(0x26060339) NetType/WIFI Language/zh_CN
    I0223 06:42:34 12067 volume_server_handlers_read.go:75] request /660/31c4f2ae71121f.jpg with unmaching cookie seen: 2926645791 expected: 269826002 from 12
    7.0.0.1:44480 agent Mozilla/5.0 (Linux; Android 7.0; SM-G9508 Build/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.2987.132 MQ
    QBrowser/6.2 TBS/043906 Mobile Safari/537.36 MicroMessenger/6.6.3.1260(0x26060339) NetType/WIFI Language/zh_CN
    I0223 06:44:28 12067 volume_server_handlers_read.go:75] request /656/375439177051ee.jpg with unmaching cookie seen: 393236974 expected: 3021359726 from 12
    7.0.0.1:44804 agent Mozilla/5.0 (Linux; Android 6.0; PLK-TL01H Build/HONORPLK-TL01H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.0.0 Mobile Safari/
    537.36
    I0223 06:44:44 12067 volume_server_handlers_read.go:75] request /661/33289a3ecd5edd.jpg with unmaching cookie seen: 1053646557 expected: 1674074305 from 1
    27.0.0.1:44804 agent Mozilla/5.0 (iPhone; CPU iPhone OS 11_1_2 like Mac OS X) AppleWebKit/604.3.5 (KHTML, like Gecko) Mobile/15B202 9ji/2.5.0/iPhone 6s
    I0223 06:47:22 12067 volume_server_handlers_read.go:75] request /657/38381fbac1a584.jpg with unmaching cookie seen: 3133252996 expected: 1543146961 from 1
    27.0.0.1:45338 agent Mozilla/5.0 (Linux; U; Android 7.1.2; zh-CN; MI 5X Build/N2G47H) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.2987.
    108 UCBrowser/11.8.8.968 Mobile Safari/537.36
    I0223 06:47:42 12067 volume_server_handlers_read.go:75] request /765/2b717f6f406291.png with unmaching cookie seen: 1866490513 expected: 2413760694 from 1
    27.0.0.1:45390 agent Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_3 like Mac OS X) AppleWebKit/603.3.8 (KHTML, like Gecko) Mobile/14G60 MicroMessenger/6.6.3 Ne
    tType/WIFI Language/zh_CN
    I0223 06:47:44 12067 volume_server_handlers_read.go:75] request /655/3469e5e28fd023.jpg with unmaching cookie seen: 3801075747 expected: 234264574 from 12
    7.0.0.1:45390 agent Mozilla/5.0 (iPhone 6sp; CPU iPhone OS 11_1 like Mac OS X) AppleWebKit/604.3.5 (KHTML, like Gecko) Version/11.0 MQQBrowser/8.0.2 Mobil
    e/15B87 Safari/8536.25 MttCustomUA/2 QBWebViewType/1 WKType/1
    I0223 06:47:52 12067 volume_server_handlers_read.go:75] request /657/38381fbac1a584.jpg with unmaching cookie seen: 3133252996 expected: 1543146961 from 1
    27.0.0.1:45390 agent Mozilla/5.0 (Linux; U; Android 7.1.2; zh-CN; MI 5X Build/N2G47H) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.2987.
    108 UCBrowser/11.8.8.968 Mobile Safari/537.36
    I0223 06:47:52 12067 volume_server_handlers_read.go:75] request /655/3469e5e28fd023.jpg with unmaching cookie seen: 3801075747 expected: 234264574 from 12
    7.0.0.1:45390 agent Mozilla/5.0 (iPhone 6sp; CPU iPhone OS 11_1 like Mac OS X) AppleWebKit/604.3.5 (KHTML, like Gecko) Version/11.0 MQQBrowser/8.0.2 Mobil
    e/15B87 Safari/8536.25 MttCustomUA/2 QBWebViewType/1 WKType/1
    I0223 06:48:04 12067 volume_server_handlers_read.go:75] request /659/334073b80964a9.jpg with unmaching cookie seen: 3087623337 expected: 387060166 from 12
    7.0.0.1:45390 agent Mozilla/5.0 (Linux; U; Android 7.0; zh-Hans-CN; SM-G9500 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.
    2987.108 Quark/2.4.1.985 Mobile Safari/537.36
    I0223 06:48:04 12067 volume_server_handlers_read.go:75] request /660/31c4f2ae71121f.jpg with unmaching cookie seen: 2926645791 expected: 269826002 from 12
    7.0.0.1:45390 agent Mozilla/5.0 (Linux; U; Android 7.0; zh-Hans-CN; SM-G9500 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.
    2987.108 Quark/2.4.1.985 Mobile Safari/537.36
    I0223 06:48:11 12067 volume_server_handlers_read.go:75] request /661/33289a3ecd5edd.jpg with unmaching cookie seen: 1053646557 expected: 1674074305 from 1
    27.0.0.1:45390 agent Mozilla/5.0 (Linux; Android 6.0.1; OPPO R9s Plus Build/MMB29M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/48.0.256
    4.116 Mobile Safari/537.36 T7/10.3 baiduboxapp/10.3.6.13 (Baidu; P1 6.0.1)
    I0223 06:48:12 12067 volume_server_handlers_read.go:75] request /659/334073b80964a9.jpg with unmaching cookie seen: 3087623337 expected: 387060166 from 12
    7.0.0.1:45390 agent Mozilla/5.0 (Linux; U; Android 7.0; zh-Hans-CN; SM-G9500 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.
    2987.108 Quark/2.4.1.985 Mobile Safari/537.36
    I0223 06:48:12 12067 volume_server_handlers_read.go:75] request /660/31c4f2ae71121f.jpg with unmaching cookie seen: 2926645791 expected: 269826002 from 12
    7.0.0.1:45390 agent Mozilla/5.0 (Linux; U; Android 7.0; zh-Hans-CN; SM-G9500 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.
    2987.108 Quark/2.4.1.985 Mobile Safari/537.36
    I0223 06:48:22 12067 volume_server_handlers_read.go:75] request /661/33289a3ecd5edd.jpg with unmaching cookie seen: 1053646557 expected: 1674074305 from 1
    27.0.0.1:45390 agent Mozilla/5.0 (Linux; Android 6.0.1; OPPO R9s Plus Build/MMB29M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/48.0.256
    4.116 Mobile Safari/537.36 T7/10.3 baiduboxapp/10.3.6.13 (Baidu; P1 6.0.1)
    I0223 06:48:22 12067 volume_server_handlers_read.go:75] request /659/334073b80964a9.jpg with unmaching cookie seen: 3087623337 expected: 387060166 from 12
    7.0.0.1:45390 agent Mozilla/5.0 (Linux; U; Android 7.0; zh-Hans-CN; SM-G9500 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.
    2987.108 Quark/2.4.1.985 Mobile Safari/537.36
    I0223 06:48:22 12067 volume_server_handlers_read.go:75] request /660/31c4f2ae71121f.jpg with unmaching cookie seen: 2926645791 expected: 269826002 from 12
    7.0.0.1:45390 agent Mozilla/5.0 (Linux; U; Android 7.0; zh-Hans-CN; SM-G9500 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.
    2987.108 Quark/2.4.1.985 Mobile Safari/537.36
    I0223 06:48:32 12067 volume_server_handlers_read.go:75] request /659/334073b80964a9.jpg with unmaching cookie seen: 3087623337 expected: 387060166 from 12
    7.0.0.1:45390 agent Mozilla/5.0 (Linux; U; Android 7.0; zh-Hans-CN; SM-G9500 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.
    2987.108 Quark/2.4.1.985 Mobile Safari/537.36
    I0223 06:48:32 12067 volume_server_handlers_read.go:75] request /660/31c4f2ae71121f.jpg with unmaching cookie seen: 2926645791 expected: 269826002 from 12
    7.0.0.1:45390 agent Mozilla/5.0 (Linux; U; Android 7.0; zh-Hans-CN; SM-G9500 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.
    2987.108 Quark/2.4.1.985 Mobile Safari/537.36
    I0223 06:48:41 12067 volume_server_handlers_read.go:75] request /659/334073b80964a9.jpg with unmaching cookie seen: 3087623337 expected: 387060166 from 12
    7.0.0.1:45390 agent Mozilla/5.0 (Linux; U; Android 7.0; zh-Hans-CN; SM-G9500 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.
    2987.108 Quark/2.4.1.985 Mobile Safari/537.36
    I0223 06:48:41 12067 volume_server_handlers_read.go:75] request /660/31c4f2ae71121f.jpg with unmaching cookie seen: 2926645791 expected: 269826002 from 12
    7.0.0.1:45390 agent Mozilla/5.0 (Linux; U; Android 7.0; zh-Hans-CN; SM-G9500 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.
    2987.108 Quark/2.4.1.985 Mobile Safari/537.36
    I0223 06:49:16 12067 volume_server_handlers_read.go:69] read error: Not Found /661,33289a3ecd5edd_1
    I0223 06:49:16 12067 volume_server_handlers_read.go:75] request /661,33289a3ecd5edd with unmaching cookie seen: 1053646557 expected: 1674074305 from 192.1
    68.254.8:40632 agent Apache-HttpClient/4.5.3 (Java/1.8.0_131)
    I0223 06:49:16 12067 volume_server_handlers_read.go:75] request /661,33289a3ecd5edd with unmaching cookie seen: 1053646557 expected: 1674074305 from 127.0
    .0.1:45652 agent Dalvik/2.1.0 (Linux; U; Android 8.0.0; MHA-AL00 Build/HUAWEIMHA-AL00)
    I0223 06:49:23 12067 volume_server_handlers_read.go:69] read error: Not Found /661,33289a3ecd5edd_1
    I0223 06:49:23 12067 volume_server_handlers_read.go:75] request /661,33289a3ecd5edd with unmaching cookie seen: 1053646557 expected: 1674074305 from 192.1
    68.254.8:40642 agent Apache-HttpClient/4.5.3 (Java/1.8.0_131)
    I0223 06:49:23 12067 volume_server_handlers_read.go:75] request /661,33289a3ecd5edd with unmaching cookie seen: 1053646557 expected: 1674074305 from 127.0
    .0.1:45652 agent Dalvik/2.1.0 (Linux; U; Android 8.0.0; MHA-AL00 Build/HUAWEIMHA-AL00)
    I0223 06:49:36 12067 volume_server_handlers_read.go:75] request /659/334073b80964a9.jpg with unmaching cookie seen: 3087623337 expected: 387060166 from 12
    7.0.0.1:45706 agent Mozilla/5.0 (Linux; U; Android 7.0; zh-Hans-CN; SM-G9500 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.
    2987.108 Quark/2.4.1.985 Mobile Safari/537.36
    I0223 06:49:36 12067 volume_server_handlers_read.go:75] request /660/31c4f2ae71121f.jpg with unmaching cookie seen: 2926645791 expected: 269826002 from 12
    7.0.0.1:45706 agent Mozilla/5.0 (Linux; U; Android 7.0; zh-Hans-CN; SM-G9500 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.
    2987.108 Quark/2.4.1.985 Mobile Safari/537.36
    I0223 06:49:43 12067 volume_server_handlers_read.go:75] request /659/2af2e996acd9ce.jpg with unmaching cookie seen: 2527910350 expected: 810789430 from 12
    7.0.0.1:45706 agent Mozilla/5.0 (Linux; U; Android 7.0; zh-cn; MI 5s Plus Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/53.0.278
    5.146 Mobile Safari/537.36 XiaoMi/MiuiBrowser/9.4.11
    I0223 06:49:51 12067 volume_server_handlers_read.go:69] read error: Not Found /661,33289a3ecd5edd_1
    I0223 06:49:51 12067 volume_server_handlers_read.go:75] request /661,33289a3ecd5edd with unmaching cookie seen: 1053646557 expected: 1674074305 from 192.1
    68.254.8:40772 agent Apache-HttpClient/4.5.3 (Java/1.8.0_131)
    I0223 06:49:51 12067 volume_server_handlers_read.go:75] request /661,33289a3ecd5edd with unmaching cookie seen: 1053646557 expected: 1674074305 from 127.0
    .0.1:45706 agent Dalvik/2.1.0 (Linux; U; Android 8.0.0; MHA-AL00 Build/HUAWEIMHA-AL00)
    I0223 06:49:52 12067 volume_server_handlers_read.go:69] read error: Not Found /661,33289a3ecd5edd_1
    I0223 06:49:52 12067 volume_server_handlers_read.go:75] request /661,33289a3ecd5edd with unmaching cookie seen: 1053646557 expected: 1674074305 from 192.1
    68.254.8:40776 agent Apache-HttpClient/4.5.3 (Java/1.8.0_131)
    I0223 06:49:52 12067 volume_server_handlers_read.go:75] request /661,33289a3ecd5edd with unmaching cookie seen: 1053646557 expected: 1674074305 from 127.0
    .0.1:45706 agent Dalvik/2.1.0 (Linux; U; Android 8.0.0; MHA-AL00 Build/HUAWEIMHA-AL00)
    I0223 06:50:16 12067 volume_server_handlers_read.go:75] request /655/2f47f8dc2842e1.jpg with unmaching cookie seen: 3693626081 expected: 590474922 from 12
    7.0.0.1:45834 agent Dalvik/2.1.0 (Linux; U; Android 7.1.2; M6 Note Build/N2G47H)
    I0223 06:50:21 12067 volume_server_handlers_read.go:69] read error: Not Found /655,29509f2e850bb0_1
    I0223 06:50:21 12067 volume_server_handlers_read.go:75] request /655,29509f2e850bb0 with unmaching cookie seen: 780471216 expected: 773671554 from 192.168
    [[email protected] seaweedfs]# 
    
    
    opened by yourchanges 68
  • Possible performance problems on some platforms

    Possible performance problems on some platforms

    As i said, we develop .NET client for seaweedFS but we see a pretty low performance on our testing deployment - so we did some tests.

    We tried some testing environments, but mostly the test results are almost same. We tried launching everything on localhost with:

    .\weed.exe master -mdir="D:\SeaweedFs\Master" .\weed.exe volume -mserver="localhost:9333" -dir="D:\SeaweedFs\VData2" -port=9101 .\weed.exe volume -mserver="localhost:9333" -dir="D:\SeaweedFs\VData2" -port=9102 .\weed.exe volume -mserver="localhost:9333" -dir="D:\SeaweedFs\VData2" -port=9103

    Also we, tried simmilar setup on 3 Windows computers (one master and volume, two just volumes) Also we, tried simmilar setup on 7 Linux computers (one master and volume, two just volumes)

    and we did tried benchmarking it:

    The results are

    Concurrency Level: 16 Time taken for tests: 486.478 seconds Complete requests: 1048576 Failed requests: 0 Total transferred: 1106793636 bytes Requests per second: 2155.44 [#/sec] Transfer rate: 2221.79 [Kbytes/sec]

    Connection Times (ms) min avg max std Total: 1.4 7.4 94.5 2.7

    Percentage of the requests served within a certain time (ms) 50% 6.9 ms 66% 8.0 ms 75% 8.5 ms 80% 9.0 ms 90% 11.0 ms 95% 12.5 ms 98% 14.5 ms 99% 16.0 ms 100% 94.5 ms

    And the readtest:

    Concurrency Level: 16 Time taken for tests: 237.319 seconds Complete requests: 1048576 Failed requests: 0 Total transferred: 1106781421 bytes Requests per second: 4418.42 [#/sec] Transfer rate: 4554.38 [Kbytes/sec]

    We do have major problem on that 7 linux machines setup since the write speed was ~320 items/sec and reading was almost ~700 items/sec. Its 2core machines, but we did some monitoring and... CPU is idling, HDD is idling, RAM is idling, Networking is idling... there is no reason for that slow performance... but yet it is that slow. This does not corresopnd with benchmark presented here on a single computer (not even close), so we must be doing something wrong.

    We tried plenty of parameters of -c and -size in benchmark also we did some experiments with -max in volumes but the results are almost same.

    We also notice, that pretty much every volume except one is doing totally nothing during benchmark, which is, if I understand it correctly, the bad thing because the volume server should be targeted by random. We are using latest 0.70 version. We can reproduce this results almost everytime. Is there some kind of utility to see whats blocking SeaweedFs going more... faster?

    opened by LightCZ 49
  • Slow performance in replication mode 010 while executing volume.fix.replication

    Slow performance in replication mode 010 while executing volume.fix.replication

    Hello We have a cluster in replication mode 010 (before that there was one rack with 000, which we decided to expand). At the moment, there are about 150,000 volumes, with a limit of 1000 megabytes. Now we are going through the fix.replication stage to replicate the data from the first rack to the second. We have huge problems with the speed of data access, especially during replication. Also, with a large data flow on the second rack, errors occur: requests.exceptions.ReadTimeout: HTTPConnectionPool(host='node2.example.com', port=11009): Read timed out Our cluster: 3 master nodes 2 racks of 18 raid arrays (12 terabytes each), each 70% full on the first rack The wizards are launched with the command: weed master -defaultReplication="010" -port="9333" -peers=node1.example.com:9333,node2.example.com:9333,node3.example.com:9333 -volumeSizeLimitMB="1024" -ip="node2.example.com" -metrics.address="node2.example.com:9091" Volumes are launched with the command: weed volume -dataCenter="netrack" -rack="node2" -port="11001" -port.public="11001" -mserver="node1.example.com:9333,node2.example.com:9333,node3.example.com:9333" -ip="node2.example.com" -max=10000 -dir=/mnt/disk1/swfs

    PS: At the moment, about 40% are replicated and it is impossible to work further

    opened by vankosa 47
  • Lots of /5,1001e1b02c1b01" errors in our production server log files">

    Lots of "volume_server_handlers.go:75] read error: /5,1001e1b02c1b01" errors in our production server log files

    Here is the

    Deploy structure

    10.252.130.159:9333(master)         10.252.130.159:5088(haproxy)
                                       /                                      \ 
                        10.252.133.22:5083 (volume1)               10.252.135.207:5084(volume2)  
    
    Replication Stratage: 001
    

    So, it always gets the content via haproxy from the volume server.

    But after running few days, we get a 404 error sometimes from the 10.252.130.159:5088 for a special fid such as :5,1001e1b02c1b01.

    And we did a check and found that one of volume server(different fid on the random different server) always output the following logs:

    I0303 15:44:23 21828 volume_server_handlers.go:75] read error: <nil> /5,1001e1b02c1b01
    I0303 15:45:14 21828 volume_server_handlers.go:75] read error: <nil> /5,1001e1b02c1b01
    I0303 15:52:52 21828 volume_server_handlers.go:75] read error: <nil> /5,1001e1b02c1b01
    I0303 15:52:52 21828 volume_server_handlers.go:75] read error: <nil> /5,1001e1b02c1b01
    I0303 15:53:05 21828 volume_server_handlers.go:75] read error: <nil> /5,1001e1b02c1b01
    I0303 15:53:06 21828 volume_server_handlers.go:75] read error: <nil> /5,1001e1b02c1b01
    I0303 15:53:07 21828 volume_server_handlers.go:75] read error: <nil> /5,1001e1b02c1b01
    I0303 15:53:07 21828 volume_server_handlers.go:75] read error: <nil> /5,1001e1b02c1b01
    I0303 15:53:18 21828 volume_server_handlers.go:75] read error: <nil> /5,1001e1b02c1b01
    

    It means that the file for [fid:5,1001e1b02c1b01] has be damaged?

    And how to fix this?

    How could be happened? any suggestions?

    opened by yourchanges 45
  • Hot, warm, cold storage [feature]

    Hot, warm, cold storage [feature]

    I'm currently running a setup with 3 dedicated masters/filers and 4 volume servers 10 TB HDD each. Our app is writing data to daily buckets. When workload is kind of write-only, everything performs quite well. But if random reading kicks in, performance seem to suffer a lot. I'm currently investigating which side is suffering, our app or the storage. But I want to clear this out for myself: is it possible to organize hot, warm and cold tiers inside one cluster?

    I mean, create new buckets on hot storage, for example NVMe SSD based volume servers, and later move it by a call to not so often accessed HDD volume servers (warm). I've read about cloud tier uploads, so it would work for cold phase I guess. But what about hot to warm transition?

    Or maybe I'm missing something and I have just misconfigured something so I can really speed up my cluster without any extra abstractions?

    opened by divanikus 40
  • brain split happened when network interrupts between dc

    brain split happened when network interrupts between dc

    weed vesion:1.15

    1. weed master: deployed in 3 DC , 3 nodes in dc1 + 2 in dc2 +in 2 dc3 ,
    2. volumer server : 3 nodes in dc1 + 3 in dc2 + 3 in dc3
    3. when the network between dc3 and dc2 interrupt , dc3 and dc1 also interrupt ,but network between dc1 and dc2 is ok , so dc3 is a network isolated island 。
    4. the original leader in dc1 , but when network issue happened , the second leader selected in dc3 and dc3's volumer server connect to the dc3 new leader. Forming two clusters , dc1 and dc2 is a cluster ,dc3 is a cluster
    5. when the network issue between dc3 and dc1、dc2 solved , also two leader in whole cluster util restart the dc3's leader.
    opened by accumulatepig 34
  • Filer hangs or restarts on deleting large buckets

    Filer hangs or restarts on deleting large buckets

    Describe the bug I'm running single master, single filer setup with s3 gateway. Filer's store is leveldb2.

    I store lots of small files (up to 1MB) in separate per day buckets. Files are stored in nested directories (not a one dir for all). It works pretty well, unless I'm trying to drop bucket, both through API or bucket.delete. Filer might just hang with other components spitting rpc error: code = Unavailable desc = transport is closing or simply do a restart. Deleting collections is breezily fast though. Am I missing something so I can painlessly delete a whole bucket in one operation? Or I should move to another filer store?

    System Setup

    • Debian 10
    • version 30GB 2.16 6912bf9 linux amd64
    opened by divanikus 30
  • [Emergency] Cluster failed after upgrading weedfs to version 2.62

    [Emergency] Cluster failed after upgrading weedfs to version 2.62

    Hi, I upgrade masters and volume servers from v 2.40 8000G to 2.62 8000G but cluster doesn't work Log on masters:

    Aug 10 15:06:27 weed-master-1 seaweedfs-master[1145]: I0810 15:06:27  1145 volume_layout.go:425] Volume 1180 becomes crowded
    Aug 10 15:06:27 weed-master-1 seaweedfs-master[1145]: I0810 15:06:27  1145 volume_layout.go:425] Volume 308 becomes crowded
    Aug 10 15:06:27 weed-master-1 seaweedfs-master[1145]: I0810 15:06:27  1145 volume_layout.go:425] Volume 1182 becomes crowded
    Aug 10 15:06:27 weed-master-1 seaweedfs-master[1145]: I0810 15:06:27  1145 volume_layout.go:425] Volume 309 becomes crowded
    Aug 10 15:06:27 weed-master-1 seaweedfs-master[1145]: I0810 15:06:27  1145 volume_layout.go:425] Volume 1179 becomes crowded
    Aug 10 15:06:27 weed-master-1 seaweedfs-master[1145]: I0810 15:06:27  1145 volume_layout.go:425] Volume 1181 becomes crowded
    Aug 10 15:06:27 weed-master-1 seaweedfs-master[1145]: I0810 15:06:27  1145 volume_layout.go:425] Volume 307 becomes crowded
    Aug 10 15:06:27 weed-master-1 seaweedfs-master[1145]: I0810 15:06:27  1145 volume_layout.go:425] Volume 1177 becomes crowded
    Aug 10 15:06:27 weed-master-1 seaweedfs-master[1145]: I0810 15:06:27  1145 volume_layout.go:425] Volume 220 becomes crowded
    Aug 10 15:06:27 weed-master-1 seaweedfs-master[1145]: I0810 15:06:27  1145 volume_layout.go:425] Volume 312 becomes crowded
    Aug 10 15:06:27 weed-master-1 seaweedfs-master[1145]: I0810 15:06:27  1145 volume_layout.go:425] Volume 310 becomes crowded
    Aug 10 15:06:27 weed-master-1 seaweedfs-master[1145]: I0810 15:06:27  1145 volume_layout.go:425] Volume 311 becomes crowded
    Aug 10 15:06:27 weed-master-1 seaweedfs-master[1145]: I0810 15:06:27  1145 volume_layout.go:425] Volume 219 becomes crowded
    Aug 10 15:06:27 weed-master-1 seaweedfs-master[1145]: I0810 15:06:27  1145 volume_layout.go:425] Volume 222 becomes crowded
    Aug 10 15:06:27 weed-master-1 seaweedfs-master[1145]: I0810 15:06:27  1145 volume_layout.go:425] Volume 1178 becomes crowded
    Aug 10 15:06:33 weed-master-1 snapd[1002]: stateengine.go:150: state ensure error: Get https://api.snapcraft.io/api/v1/snaps/sections: dial tcp: lookup api.snapcraft.io on 172.30.100.3:53: server misbehaving
    

    Other master:

    Aug 10 15:06:20 weed-master-2 seaweedfs-master[1154]: I0810 15:06:20  1154 common.go:50] response method:GET URL:/dir/lookup?volumeId=370 with httpStatus:404 and JSON:{"volumeId":"370","error":"volume id 370 not found"}
    Aug 10 15:06:20 weed-master-2 seaweedfs-master[1154]: I0810 15:06:20  1154 common.go:50] response method:GET URL:/dir/lookup?volumeId=1067 with httpStatus:404 and JSON:{"volumeId":"1067","error":"volume id 1067 not found"}
    Aug 10 15:06:20 weed-master-2 seaweedfs-master[1154]: I0810 15:06:20  1154 common.go:50] response method:GET URL:/dir/lookup?volumeId=1234 with httpStatus:404 and JSON:{"volumeId":"1234","error":"volume id 1234 not found"}
    Aug 10 15:06:20 weed-master-2 seaweedfs-master[1154]: I0810 15:06:20  1154 common.go:50] response method:GET URL:/dir/lookup?volumeId=1236 with httpStatus:404 and JSON:{"volumeId":"1236","error":"volume id 1236 not found"}
    Aug 10 15:06:20 weed-master-2 seaweedfs-master[1154]: I0810 15:06:20  1154 common.go:50] response method:GET URL:/dir/lookup?volumeId=609 with httpStatus:404 and JSON:{"volumeId":"609","error":"volume id 609 not found"}
    Aug 10 15:06:20 weed-master-2 seaweedfs-master[1154]: I0810 15:06:20  1154 common.go:50] response method:GET URL:/dir/lookup?volumeId=24 with httpStatus:404 and JSON:{"volumeId":"24","error":"volume id 24 not found"}
    Aug 10 15:06:20 weed-master-2 seaweedfs-master[1154]: I0810 15:06:20  1154 common.go:50] response method:GET URL:/dir/lookup?volumeId=83 with httpStatus:404 and JSON:{"volumeId":"83","error":"volume id 83 not found"}
    Aug 10 15:06:20 weed-master-2 seaweedfs-master[1154]: I0810 15:06:20  1154 common.go:50] response method:GET URL:/dir/lookup?volumeId=320 with httpStatus:404 and JSON:{"volumeId":"320","error":"volume id 320 not found"}
    Aug 10 15:06:20 weed-master-2 seaweedfs-master[1154]: I0810 15:06:20  1154 common.go:50] response method:GET URL:/dir/lookup?volumeId=1356 with httpStatus:404 and JSON:{"volumeId":"1356","error":"volume id 1356 not found"}
    Aug 10 15:06:20 weed-master-2 seaweedfs-master[1154]: I0810 15:06:20  1154 common.go:50] response method:GET URL:/dir/lookup?volumeId=1352 with httpStatus:404 and JSON:{"volumeId":"1352","error":"volume id 1352 not found"}
    
    

    Master03:

    Aug 10 15:06:18 weed-master-3 seaweedfs-master[1149]: I0810 15:06:18  1149 volume_layout.go:376] Volume 210 has 0 replica, less than required 2
    Aug 10 15:06:18 weed-master-3 seaweedfs-master[1149]: I0810 15:06:18  1149 topology_event_handling.go:79] Removing Volume 387 from the dead volume server weed-volume-012:8086
    Aug 10 15:06:18 weed-master-3 seaweedfs-master[1149]: I0810 15:06:18  1149 volume_layout.go:376] Volume 387 has 0 replica, less than required 2
    Aug 10 15:06:18 weed-master-3 seaweedfs-master[1149]: I0810 15:06:18  1149 topology_event_handling.go:79] Removing Volume 537 from the dead volume server weed-volume-012:8086
    Aug 10 15:06:18 weed-master-3 seaweedfs-master[1149]: I0810 15:06:18  1149 volume_layout.go:376] Volume 537 has 0 replica, less than required 2
    Aug 10 15:06:18 weed-master-3 seaweedfs-master[1149]: I0810 15:06:18  1149 topology_event_handling.go:79] Removing Volume 1362 from the dead volume server weed-volume-012:8086
    Aug 10 15:06:18 weed-master-3 seaweedfs-master[1149]: I0810 15:06:18  1149 volume_layout.go:376] Volume 1362 has 0 replica, less than required 2
    Aug 10 15:06:18 weed-master-3 seaweedfs-master[1149]: I0810 15:06:18  1149 topology_event_handling.go:79] Removing Volume 759 from the dead volume server weed-volume-012:8086
    Aug 10 15:06:18 weed-master-3 seaweedfs-master[1149]: I0810 15:06:18  1149 volume_layout.go:376] Volume 759 has 1 replica, less than required 2
    Aug 10 15:06:18 weed-master-3 seaweedfs-master[1149]: I0810 15:06:18  1149 topology_event_handling.go:79] Removing Volume 801 from the dead volume server weed-volume-012:8086
    
    
    opened by kaaaq 29
  • 迁移问题

    迁移问题

    有2台服务器进行迁移,迁移不成功需要指导,最终需要实现的目的就是把mysql里的数据迁移到redis里面。 参照https://github.com/chrislusf/seaweedfs/wiki/Async-Replication-to-another-Filer#replicate-existing-files

    配置如下: 主机 数据库 192.168.20.51 mysql 192.168.20.55 redis

    192.168.20.51的配置如下 启了一个master,三个volume,一个filer: nohup /home/seaweedfs/weed master -mdir=/home/seaweedfs/data/master -port=9333 -ip 192.168.20.51 -defaultReplication=010 & /home/seaweedfs/weed volume -dir=/home/seaweedfs/volume/volume1 -max=100 -mserver=192.168.20.51:9333 -port=9001 -ip=192.168.20.51 -dataCenter=dc1 -rack=rack1 & >> /home/seaweedfs/logs/vol1.log & /home/seaweedfs/weed volume -dir=/home/seaweedfs/volume/volume2 -max=100 -mserver=192.168.20.51:9333 -port=9002 -ip=192.168.20.51 -dataCenter=dc1 -rack=rack2 & >> /home/seaweedfs/logs/vol2.log & /home/seaweedfs/weed volume -dir=/home/seaweedfs/volume/volume3 -max=100 -mserver=192.168.20.51:9333 -port=9003 -ip=192.168.20.51 -dataCenter=dc1 -rack=rack3 & >> /home/seaweedfs/logs/vol3.log & /home/seaweedfs/weed filer -master=192.168.20.51:9333 -ip=192.168.20.51 & >> /home/seaweedfs/logs/filer.log &

    cd /etc/seaweedfs,配置如下 cat filer.toml [mysql] enabled = true hostname = "192.168.20.51" port = 3306 username = "root" password = "123456" database = "filer" # create or use an existing database connection_max_idle = 2 connection_max_open = 100

    cat notification.toml [notification.kafka] enabled = true hosts = [ "localhost:9092" ] topic = "seaweedfs_filer"

    cat replication.toml [source.filer] enabled = true grpcAddress = "192.168.20.51:18888" directory = "/"

    [sink.filer] enabled = true grpcAddress = "192.168.20.55:18888" directory = "/" replication = "" collection = "" ttlSec = 0

    192.168.20.55的配置如下 启了一个master,三个volume,一个filer: nohup /home/seaweedfs/weed master -mdir=/home/seaweedfs/data/master -port=9333 -ip 192.168.20.55 -defaultReplication=010 & /home/seaweedfs/weed volume -dir=/home/seaweedfs/volume/volume1 -max=100 -mserver=192.168.20.55:9333 -port=9001 -ip=192.168.20.55 -dataCenter=dc1 -rack=rack1 & >> /home/seaweedfs/logs/vol1.log & /home/seaweedfs/weed volume -dir=/home/seaweedfs/volume/volume2 -max=100 -mserver=192.168.20.55:9333 -port=9002 -ip=192.168.20.55 -dataCenter=dc1 -rack=rack2 & >> /home/seaweedfs/logs/vol2.log & /home/seaweedfs/weed volume -dir=/home/seaweedfs/volume/volume3 -max=100 -mserver=192.168.20.55:9333 -port=9003 -ip=192.168.20.55 -dataCenter=dc1 -rack=rack3 & >> /home/seaweedfs/logs/vol3.log & /home/seaweedfs/weed filer -master=192.168.20.55:9333 -ip=192.168.20.55 & >> /home/seaweedfs/logs/filer.log &

    cd /etc/seaweedfs cat filer.toml [redis] enabled = true address = "localhost:6379" password = "" db = 0

    然后在192.168.20.51上 1).先启动了kafka 2).启动weed filer.replicate 3).启动weed filer 都成功,最后执行了此命令echo 'fs.meta.notify' | weed shell后,都是如下错误

    image

    最后的表现为: image image

    目录都过去了,为什么目录下面的数据却复制不过去,而且voulme里面的数据也没有迁移过去?

    我重新上传数据,两边却很快就同步了 image image

    总结:两边配置以后,新上传的数据,可以同步,然后之前的老数据却同步不过去,求指导,为啥?

    opened by gitlihua 28
  • Webdav server stops responding after saving some files

    Webdav server stops responding after saving some files

    WebDav server stop responding after file with some extension was uploaded.

    My configuration: I have 3 nodes in cluster. Every node runs 2 docker-container:

    • official cockroachdb container for filer metadata (v.20.1.0)
    • my custom seaweedfs container with:
    • weed master -ip=b-img02 -mdir=/opt/seaweedfs/data/master -peers=b-img00:9333,b-img01:9333,b-img02:9333 -defaultReplication=200
    • weed volume -ip=b-img02 -mserver=b-img00:9333,b-img01:9333,b-img02:9333 -dataCenter=b-img02 -rack=rack -dir=/opt/seaweedfs/data/
    • weed filer -collection=filer -defaultReplicaPlacement=200
    • weed webdav Weed version: 30GB 1.79

    Screenshot_20200603_140300

    All works fine until I trying to upload file in filer.

    In examples I create test file, upload it into filer and download from webdav.

    Example#1 - all works fine

    echo ok > data.html
    
    curl -F [email protected] http://b-img00:8888/
    >> {"name":"data.html","size":3,"fid":"7,08ba03a642ff84","url":"http://b-img02:8080/7,08ba03a642ff84"}
    
    curl -vvv http://b-img00:7333/data.html
    *   Trying x.x.x.x...
    * TCP_NODELAY set
    * Connected to b-img00 (x.x.x.x) port 7333 (#0)
    > GET /data.html HTTP/1.1
    > Host: b-img00:7333
    > User-Agent: curl/7.58.0
    > Accept: */*
    >
    < HTTP/1.1 200 OK
    < Accept-Ranges: bytes
    < Content-Length: 3
    < Content-Type: text/html; charset=utf-8
    < Etag: "161503d94245de003"
    < Last-Modified: Wed, 03 Jun 2020 11:04:35 GMT
    < Date: Wed, 03 Jun 2020 11:04:57 GMT
    <
    ok
    * Connection #0 to host b-img00 left intact
    
    curl -X DELETE -vvv http://b-img00:7333/data.html
    *   Trying x.x.x.x...
    * TCP_NODELAY set
    * Connected to b-img00 (x.x.x.x) port 7333 (#0)
    > DELETE /data.html HTTP/1.1
    > Host: b-img00:7333
    > User-Agent: curl/7.58.0
    > Accept: */*
    >
    < HTTP/1.1 204 No Content
    < Date: Wed, 03 Jun 2020 11:08:15 GMT
    <
    * Connection #0 to host b-img00 left intact
    

    Ok. Example#2 - I change file extention from .html to .txt After file uploaded webdav stops to serv folder with this file (it can be downloaded via filer successfully). Another folders in webdav works fine.

    echo ok > data.txt
    
    curl -F [email protected] http://b-img00:8888/
    
    curl -vvv http://b-img00:7333/data.txt
    *   Trying xx.xx.xx.xx...
    * TCP_NODELAY set
    * Connected to b-img00 (xx.xx.xx.xx) port 7333 (#0)
    > GET /data.txt HTTP/1.1
    > Host: b-img00:7333
    > User-Agent: curl/7.58.0
    > Accept: */*
    >
    (( I await some time - but no response received ))
    ^C
    
    curl -vvv http://b-img00:8888/data.txt
    *   Trying xx.xx.xx.xx...
    * TCP_NODELAY set
    * Connected to b-img00 (xx.xx.xx.xx) port 8888 (#0)
    > GET /data.txt HTTP/1.1
    > Host: b-img00:8888
    > User-Agent: curl/7.58.0
    > Accept: */*
    >
    < HTTP/1.1 200 OK
    < Accept-Ranges: bytes
    < Content-Disposition: inline; filename="data.txt"
    < Content-Length: 3
    < Content-Type: text/plain
    < Etag: "908e596f"
    < Last-Modified: Wed, 03 Jun 2020 11:09:35 GMT
    < Date: Wed, 03 Jun 2020 11:10:06 GMT
    <
    ok
    * Connection #0 to host b-img00 left intact
    
    curl -X DELETE http://b-img00:8888/data.txt
    (ok)
    
    opened by shm-dmitry 27
  • Unable to add filer

    Unable to add filer

    Describe the bug When trying to add a new filer I1129 04:33:38 16088 meta_aggregator.go:190] subscribing remote IP1:8888 meta change: rpc error: code = Unknown desc = reading from persisted logs: reading /topics/.system/log/2021-11-22/09-56.8ff86979: volume 6 not found

    System Setup

    • List the command line to start "weed master", "weed volume", "weed filer", "weed s3", "weed mount". S1 /opt/weed -logdir=/var/log/weed filer -ip=IP1 -s3 -defaultReplicaPlacement=000 -dataCenter=DC1 S2 /opt/weed -logdir=/var/log/weed filer -ip=IP2 -s3 -defaultReplicaPlacement=000 -dataCenter=DC1
    • output of weed version version 8000GB 2.79 f3c789d662eeab2dcc07804cbb665752b3870d65 linux amd64
    • if using filer, show the content of filer.toml
    [leveldb3]
    # similar to leveldb2.
    # each bucket has its own meta store.
    enabled = true
    dir = "/srv/filer_leveldb3"                    # directory to store level db files
    

    Everything else is "false"

    In "volume.list" I also don't see any volume with this id and I do not see missing servers in volume.list. I even checked the presence/consistency of data in other buckets, just in case, and found no problems.

    So, maybe you have some idea what to do about it?

    opened by LuckyFrost 0
  • [IAM, S3] Could not create users via IAM gateway for S3 access

    [IAM, S3] Could not create users via IAM gateway for S3 access

    Bug description According to the wiki, it should be possible to start the IAM gateway with the server command with the flag -iam. However, the flag is not available, see the output below:

    /data # weed server -iam
    flag provided but not defined: -iam
    Example: weed server -dir=/tmp -volume.max=5 -ip=server_name
    Default Usage:
      -cpuprofile string
        	cpu profile output file
      -dataCenter string
        	current volume server's data center name
      -debug
        	serves runtime profiling data, e.g., http://localhost:6060/debug/pprof/goroutine?debug=2
      -debug.port int
        	http port for debugging (default 6060)
      -dir string
        	directories to store data files. dir[,dir]... (default "/tmp")
      -disableHttp
        	disable http requests, only gRPC operations are allowed.
      -filer
        	whether to start filer
      -filer.collection string
        	all data will be stored in this collection
      -filer.concurrentUploadLimitMB int
        	limit total concurrent upload size (default 64)
      -filer.defaultReplicaPlacement string
        	default replication type. If not specified, use master setting.
      -filer.dirListLimit int
        	limit sub dir listing size (default 1000)
      -filer.disableDirListing
        	turn off directory listing
      -filer.encryptVolumeData
        	encrypt data on volume servers
      -filer.maxMB int
        	split files larger than the limit (default 4)
      -filer.port int
        	filer server http listen port (default 8888)
      -filer.port.grpc int
        	filer server grpc listen port
      -filer.port.public int
        	filer server public http listen port
      -filer.saveToFilerLimit int
        	Small files smaller than this limit can be cached in filer store.
      -garbageThreshold float
        	threshold to vacuum and reclaim spaces (default 0.3)
      -idleTimeout int
        	connection idle seconds (default 30)
      -ip string
        	ip or server name, also used as identifier (default "172.18.0.4")
      -ip.bind string
        	ip address to bind to
      -master
        	whether to start master server (default true)
      -master.defaultReplication string
        	Default replication type if not specified.
      -master.dir string
        	data directory to store meta data, default to same as -dir specified
      -master.peers string
        	all master nodes in comma separated ip:masterPort list
      -master.port int
        	master server http listen port (default 9333)
      -master.port.grpc int
        	master server grpc listen port
      -master.volumePreallocate
        	Preallocate disk space for volumes.
      -master.volumeSizeLimitMB uint
        	Master stops directing writes to oversized volumes. (default 30000)
      -memprofile string
        	memory profile output file
      -metrics.address string
        	Prometheus gateway address
      -metrics.intervalSeconds int
        	Prometheus push interval in seconds (default 15)
      -metricsPort int
        	Prometheus metrics listen port
      -msgBroker
        	whether to start message broker
      -msgBroker.port int
        	broker gRPC listen port (default 17777)
      -options string
        	a file of command line options, each line in optionName=optionValue format
      -rack string
        	current volume server's rack name
      -resumeState
        	resume previous state on start master server
      -s3
        	whether to start S3 gateway
      -s3.allowEmptyFolder
        	allow empty folders (default true)
      -s3.cert.file string
        	path to the TLS certificate file
      -s3.config string
        	path to the config file
      -s3.domainName string
        	suffix of the host name in comma separated list, {bucket}.{domainName}
      -s3.key.file string
        	path to the TLS private key file
      -s3.port int
        	s3 server http listen port (default 8333)
      -volume
        	whether to start volume server (default true)
      -volume.compactionMBps int
        	limit compaction speed in mega bytes per second
      -volume.concurrentDownloadLimitMB int
        	limit total concurrent download size (default 64)
      -volume.concurrentUploadLimitMB int
        	limit total concurrent upload size (default 64)
      -volume.dir.idx string
        	directory to store .idx files
      -volume.disk string
        	[hdd|ssd|<tag>] hard drive or solid state drive or any tag
      -volume.fileSizeLimitMB int
        	limit file size to avoid out of memory (default 256)
      -volume.images.fix.orientation
        	Adjust jpg orientation when uploading.
      -volume.index string
        	Choose [memory|leveldb|leveldbMedium|leveldbLarge] mode for memory~performance balance. (default "memory")
      -volume.max string
        	maximum numbers of volumes, count[,count]... If set to zero, the limit will be auto configured. (default "8")
      -volume.minFreeSpace string
        	min free disk space (value<=100 as percentage like 1, other as human readable bytes, like 10GiB). Low disk space will mark all volumes as ReadOnly.
      -volume.minFreeSpacePercent string
        	minimum free disk space (default to 1%). Low disk space will mark all volumes as ReadOnly (deprecated, use minFreeSpace instead). (default "1")
      -volume.port int
        	volume server http listen port (default 8080)
      -volume.port.grpc int
        	volume server grpc listen port
      -volume.port.public int
        	volume server public port
      -volume.pprof
        	enable pprof http handlers. precludes --memprofile and --cpuprofile
      -volume.preStopSeconds int
        	number of seconds between stop send heartbeats and stop volume server (default 10)
      -volume.publicUrl string
        	publicly accessible address
      -volume.readMode string
        	[local|proxy|redirect] how to deal with non-local volume: 'not found|read in remote node|redirect volume location'. (default "proxy")
      -volume.tcp
        	<exprimental> enable tcp port
      -webdav
        	whether to start WebDAV gateway
      -webdav.cacheCapacityMB int
        	local cache capacity in MB (default 1000)
      -webdav.cacheDir string
        	local cache directory for file chunks (default "/tmp")
      -webdav.cert.file string
        	path to the TLS certificate file
      -webdav.collection string
        	collection to create the files
      -webdav.disk string
        	[hdd|ssd|<tag>] hard drive or solid state drive or any tag
      -webdav.key.file string
        	path to the TLS private key file
      -webdav.port int
        	webdav server http listen port (default 7333)
      -webdav.replication string
        	replication to create the files
      -whiteList string
        	comma separated Ip addresses having write permission. No limit if empty.
    Description:
      start both a volume server to provide storage spaces
      and a master server to provide volume=>location mapping service and sequence number of file ids
    
      This is provided as a convenient way to start both volume server and master server.
      The servers acts exactly the same as starting them separately.
      So other volume servers can connect to this master server also.
    
      Optionally, a filer server can be started.
      Also optionally, a S3 gateway can be started.
    

    In order to find a workaround, the iam service seems not to be able to receive the information about the configured identities which are configured via -s3.config. After some debugging, it looks as if the config cannot be loaded, also see docker-compose logs below. The line

    filer_1   | W1129 10:55:59     1 auth_credentials.go:79] fail to load config: read S3 config: filer: no entry is found in filer store
    

    looks as if there is any problem.

    However, since it is only a warning, I tried to use the AWS CLI to add a new user.

    $ aws --endpoint-url http://127.0.0.1:9001 iam create-access-key --user-name Bob
    
    An error occurred (Unknown) when calling the CreateAccessKey operation (reached max retries: 2): Unknown
    
    $ aws --endpoint-url http://127.0.0.1:9001 iam list-access-keys
    
    An error occurred (Unknown) when calling the ListAccessKeys operation (reached max retries: 2): Unknown
    

    System Setup

    • AWS CLI version: aws-cli/2.2.7
    • docker-compose.yml:
    version: "2"
    
    services:
      master:
        image: chrislusf/seaweedfs
        ports:
          - 9333:9333
          - 19333:19333
        command: "master -ip=master"
      volume:
        image: chrislusf/seaweedfs
        ports:
          - 8080:8080
          - 18080:18080
        command: "volume -mserver=master:9333 -port=8080 -ip=volume -preStopSeconds=1"
        depends_on:
          - master
      filer:
        image: chrislusf/seaweedfs
        ports:
          - 9000:8333
          - 9001:8111
          - 8888:8888
          - 18888:18888
        command: 'filer -master="master:9333" -s3 -s3.config=/etc/iam/identity.json -iam'
        volumes:
          - "./config.json:/etc/iam/identity.json"
        depends_on:
          - master
          - volume
    
    • config.json for s3(/iam) authentication:
    {
      "identities": [
        {
          "name": "s3adminwhatever",
          "credentials": [
            {
              "accessKey": "s3adminwhatever",
              "secretKey": "s3adminwhatever"
            }
          ],
          "actions": ["Admin", "Read", "List", "Tagging", "Write"]
        }
      ]
    }
    
    • output of weed version:
    /data # weed version
    version 30GB 2.80 7227cfdd linux amd6
    
    • if using filer, show the content of filer.toml: (default one)
    [leveldb2]
    enabled = true
    dir = "/data/filerldb2"
    
    • log output of docker-compose:
    master_1  | I1129 10:55:36     1 file_util.go:23] Folder /data Permission: -rwxr-xr-x
    master_1  | I1129 10:55:36     1 master.go:170] current: master:9333 peers:
    master_1  | I1129 10:55:36     1 master_server.go:120] Volume Size Limit is 1024 MB
    master_1  | I1129 10:55:36     1 master_server.go:227] adminScripts:
    master_1  |             lock
    master_1  |             ec.encode -fullPercent=95 -quietFor=1h
    master_1  |             ec.rebuild -force
    master_1  |             ec.balance -force
    master_1  |             volume.deleteEmpty -quietFor=24h -force
    master_1  |             volume.balance -force
    master_1  |             volume.fix.replication
    master_1  |             s3.clean.uploads -timeAgo=24h
    master_1  |             unlock
    master_1  |
    volume_1  | I1129 10:55:36     1 file_util.go:23] Folder /data Permission: -rwxr-xr-x
    volume_1  | I1129 10:55:36     1 volume_grpc_client_to_master.go:41] checkWithMaster master:9333: get master master:9333 configuration: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.18.0.2:19333: connect: connection refused"
    master_1  | I1129 10:55:36     1 master.go:125] Start Seaweed Master 30GB 2.80 7227cfdd at :9333
    master_1  | I1129 10:55:36     1 raft_server.go:71] Starting RaftServer with master:9333
    master_1  | I1129 10:55:36     1 raft_server.go:131] current cluster leader:
    volume_1  | I1129 10:55:38     1 volume_grpc_client_to_master.go:41] checkWithMaster master:9333: get master master:9333 configuration: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.18.0.2:19333: connect: connection refused"
    filer_1   | I1129 10:55:40     1 s3.go:162] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | I1129 10:55:40     1 iam.go:60] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    volume_1  | I1129 10:55:40     1 volume_grpc_client_to_master.go:41] checkWithMaster master:9333: get master master:9333 configuration: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.18.0.2:19333: connect: connection refused"
    filer_1   | I1129 10:55:41     1 iam.go:60] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | I1129 10:55:41     1 s3.go:162] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    volume_1  | I1129 10:55:42     1 volume_grpc_client_to_master.go:41] checkWithMaster master:9333: get master master:9333 configuration: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.18.0.2:19333: connect: connection refused"
    filer_1   | I1129 10:55:42     1 s3.go:162] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | I1129 10:55:42     1 iam.go:60] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | I1129 10:55:43     1 iam.go:60] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | I1129 10:55:43     1 s3.go:162] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    volume_1  | I1129 10:55:43     1 volume_grpc_client_to_master.go:41] checkWithMaster master:9333: get master master:9333 configuration: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.18.0.2:19333: connect: connection refused"
    filer_1   | I1129 10:55:44     1 s3.go:162] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | I1129 10:55:44     1 iam.go:60] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | I1129 10:55:45     1 iam.go:60] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | I1129 10:55:45     1 s3.go:162] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    volume_1  | I1129 10:55:45     1 volume_grpc_client_to_master.go:41] checkWithMaster master:9333: get master master:9333 configuration: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.18.0.2:19333: connect: connection refused"
    filer_1   | I1129 10:55:46     1 iam.go:60] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | I1129 10:55:46     1 s3.go:162] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | I1129 10:55:47     1 s3.go:162] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | I1129 10:55:47     1 iam.go:60] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    volume_1  | I1129 10:55:47     1 volume_grpc_client_to_master.go:41] checkWithMaster master:9333: get master master:9333 configuration: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.18.0.2:19333: connect: connection refused"
    filer_1   | I1129 10:55:48     1 iam.go:60] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | I1129 10:55:48     1 s3.go:162] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | I1129 10:55:49     1 iam.go:60] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | I1129 10:55:49     1 s3.go:162] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    volume_1  | I1129 10:55:49     1 volume_grpc_client_to_master.go:41] checkWithMaster master:9333: get master master:9333 configuration: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.18.0.2:19333: connect: connection refused"
    filer_1   | I1129 10:55:50     1 s3.go:162] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | I1129 10:55:50     1 iam.go:60] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    volume_1  | I1129 10:55:51     1 volume_grpc_client_to_master.go:41] checkWithMaster master:9333: get master master:9333 configuration: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.18.0.2:19333: connect: connection refused"
    filer_1   | I1129 10:55:51     1 iam.go:60] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | I1129 10:55:51     1 s3.go:162] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | I1129 10:55:52     1 iam.go:60] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | I1129 10:55:52     1 s3.go:162] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    volume_1  | I1129 10:55:52     1 volume_grpc_client_to_master.go:41] checkWithMaster master:9333: get master master:9333 configuration: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.18.0.2:19333: connect: connection refused"
    filer_1   | I1129 10:55:53     1 s3.go:162] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | I1129 10:55:53     1 iam.go:60] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    master_1  | I1129 10:55:54     1 master.go:148] Start Seaweed Master 30GB 2.80 7227cfdd grpc server at :19333
    filer_1   | I1129 10:55:54     1 iam.go:60] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | I1129 10:55:54     1 s3.go:162] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | I1129 10:55:55     1 s3.go:162] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | I1129 10:55:55     1 iam.go:60] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    master_1  | I1129 10:55:55     1 masterclient.go:79] No existing leader found!
    master_1  | I1129 10:55:55     1 raft_server.go:158] Initializing new cluster
    master_1  | I1129 10:55:55     1 master_server.go:163] leader change event:  => master:9333
    master_1  | I1129 10:55:55     1 master_server.go:165] [ master:9333 ] master:9333 becomes leader.
    filer_1   | I1129 10:55:56     1 iam.go:60] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | I1129 10:55:56     1 s3.go:162] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | I1129 10:55:57     1 s3.go:162] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | I1129 10:55:57     1 iam.go:60] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | W1129 10:55:58     1 filer_server.go:122] skipping default store dir in ./filerldb2
    master_1  | I1129 10:55:58     1 master_grpc_server.go:262] + client [email protected]:8888
    filer_1   | I1129 10:55:58     1 leveldb2_store.go:42] filer store leveldb2 dir: /data/filerldb2
    filer_1   | I1129 10:55:58     1 file_util.go:23] Folder /data/filerldb2 Permission: -rwxr-xr-x
    filer_1   | I1129 10:55:58     1 s3.go:162] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | I1129 10:55:58     1 iam.go:60] wait to connect to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | I1129 10:55:58     1 filer.go:121] create filer.store.id to -1463490671
    filer_1   | I1129 10:55:58     1 configuration.go:28] configured filer store to leveldb2
    filer_1   | I1129 10:55:58     1 filer.go:89] the cluster has 1 filers
    filer_1   | I1129 10:55:58     1 filer.go:222] Start Seaweed Filer 30GB 2.80 7227cfdd at 172.18.0.4:8888
    filer_1   | I1129 10:55:58     1 meta_aggregator.go:100] connecting to peer filer 172.18.0.4:8888: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.18.0.4:18888: connect: connection refused"
    master_1  | I1129 10:55:59     1 master_grpc_server.go:262] + client [email protected]:9333
    master_1  | I1129 10:55:59     1 master_grpc_server.go:262] + client [email protected]
    filer_1   | I1129 10:55:59     1 iam.go:56] IAM read filer configuration: masters:"master:9333" max_mb:4 dir_buckets:"/buckets" signature:-1463490671 metrics_interval_sec:15 version:"30GB 2.80 7227cfdd"
    filer_1   | I1129 10:55:59     1 iam.go:63] connected to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | I1129 10:55:59     1 s3.go:158] S3 read filer buckets dir: /buckets
    filer_1   | I1129 10:55:59     1 s3.go:165] connected to filer 172.18.0.4:8888 grpc address 172.18.0.4:18888
    filer_1   | W1129 10:55:59     1 auth_credentials.go:79] fail to load config: read S3 config: filer: no entry is found in filer store
    filer_1   | I1129 10:55:59     1 iam.go:76] NewIamApiServer created
    filer_1   | I1129 10:55:59     1 iam.go:89] Start Seaweed IAM API Server 30GB 2.80 7227cfdd at http port 8111
    filer_1   | I1129 10:55:59     1 s3.go:201] Start Seaweed S3 API Server 30GB 2.80 7227cfdd at http port 8333
    filer_1   | I1129 10:55:59     1 filer_grpc_server_sub_meta.go:242] + listener [email protected]:60256
    filer_1   | I1129 10:55:59     1 filer_grpc_server_sub_meta.go:31]  [email protected]:60256 starts to subscribe /etc/iam/identity.json from 2021-11-29 10:55:59.241463685 +0000 UTC
    filer_1   | I1129 10:55:59     1 filer_grpc_server_sub_meta.go:242] + listener filer:172.18.0.4:[email protected]:60256
    filer_1   | I1129 10:55:59     1 filer_grpc_server_sub_meta.go:89]  filer:172.18.0.4:[email protected]:60256 local subscribe / from 2021-11-29 10:54:58.26448181 +0000 UTC
    filer_1   | I1129 10:55:59     1 filer_grpc_server_sub_meta.go:101] read on disk filer:172.18.0.4:[email protected]:60256 local subscribe / from 2021-11-29 10:54:58.26448181 +0000 UTC
    filer_1   | I1129 10:55:59     1 filer_grpc_server_sub_meta.go:117] read in memory filer:172.18.0.4:[email protected]:60256 local subscribe / from 2021-11-29 10:54:58.26448181 +0000 UTC
    volume_1  | I1129 10:55:59     1 disk_location.go:180] Store started on dir: /data with 0 volumes max 0
    volume_1  | I1129 10:55:59     1 disk_location.go:183] Store started on dir: /data with 0 ec shards
    volume_1  | I1129 10:55:59     1 volume_grpc_client_to_master.go:50] Volume server start with seed master nodes: [master:9333]
    volume_1  | I1129 10:55:59     1 volume.go:360] Start Seaweed volume server 30GB 2.80 7227cfdd at :8080
    master_1  | I1129 10:55:59     1 node.go:222] topo adds child DefaultDataCenter
    master_1  | I1129 10:55:59     1 node.go:222] topo:DefaultDataCenter adds child DefaultRack
    master_1  | I1129 10:55:59     1 node.go:222] topo:DefaultDataCenter:DefaultRack adds child volume:8080
    master_1  | I1129 10:55:59     1 node.go:222] topo:DefaultDataCenter:DefaultRack:volume:8080 adds child
    master_1  | I1129 10:55:59     1 master_grpc_server.go:70] added volume server 0: volume:8080
    volume_1  | I1129 10:55:59     1 volume_grpc_client_to_master.go:107] Heartbeat to: master:9333
    

    Expected behavior Using the IAM gateway, i would expect, that the IAM can work on top of the identities configured in the S3, as it might be intended.

    opened by schwiftysmith 3
  • fix volume idx error

    fix volume idx error

    I have copy volume data to another server with same OS ubuntu 20.04. But start server get the error message: volumeDataIntegrityChecking failed verifyNeedleIntergrity /home/administrator/data/vol2/1.idx failed : index key 0x3439333966 does not match needle's Id 0x65633432383130303030303

    I use command to fix : weed fix -dir=/home/administrator/data/vol2 volumeId=1 ,still error with readSuperBlock volume 1 version 3 scan .dat File: cannot read needle header at offset -900977296: readat /home/administrator/data/vol2/1.dat negative offset

    How can fix it weed version: linux 2.64 big file

    opened by wanghche 2
  • failed to write to local disk

    failed to write to local disk

    Problem Upload 2GB video failed {"name":"ceshi.mp4","size":2314382721,"error":"failed to write to local disk: volume size limit 34359738368 exceeded! current size is 57394","eTag":"07f99010"}

    Seaweedfs Setup

    • nohup /data/xxx/swf/weed master -ip=$IP -port=9333 -volumeSizeLimitMB=30000 -mdir=/data/xxx/swf/data -peers=$IP:9333 > /data/xxx/swf/data/master.log &
    • nohup /data/xxx/swf/weed volume -fileSizeLimitMB=30000 -dataCenter dc1 -rack rack1 -dir /data/xxx/swf/volume -ip $IP -port 9222 -ip.bind $IP -max 20 -mserver $IP:9333 -publicUrl $IP:9222 > /data/xxx/swf/volume/volume.log &
    • disk space is ok
    opened by DuncanPlayer 1
  • about mount safety

    about mount safety

    Mount You can specify the directory to mount by using '-filer.path'

    However, this is very insecure, and if the root directory is mounted and the '/bucket' is mistakenly deleted, it could be a security risk

    Quarantine paths, any solutions?

    opened by zemul 1
  • Why other volume ids were invisble in layouts and could't be assigned by the master?

    Why other volume ids were invisble in layouts and could't be assigned by the master?

    Sponsors SeaweedFS via Patreon https://www.patreon.com/seaweedfs

    Describe the bug I have run seaweedfs docker and done some tests recently. Seaweed is wonderful and Everything is ok until I found master always returned fids on a fixed volume id(eg 26) rather then other volume ids(1/2/3/4/5/6/7).

    curl "http://172.88.0.31:9333/dir/assign"
    {"fid":"26,1d0c6f6018","url":"172.88.0.32:8080","publicUrl":"172.88.0.32:8080","count":1}z
    curl "http://172.88.0.31:9333/dir/assign"
    {"fid":"26,1e3848303c","url":"172.88.0.32:8080","publicUrl":"172.88.0.32:8080","count":1}
    curl "http://172.88.0.31:9333/dir/assign"
    {"fid":"26,1fdeaa3076","url":"172.88.0.32:8080","publicUrl":"172.88.0.32:8080","count":1}
    curl "http://172.88.0.31:9333/dir/assign"
    {"fid":"26,20b6f1898d","url":"172.88.0.32:8080","publicUrl":"172.88.0.32:8080","count":1}
    
    

    I found only volume id 26 was in layouts:

    curl "http://172.88.0.31:9333/dir/status?pretty=y"
    {
      "Topology": {
        "DataCenters": [
          {
            "Free": 3,
            "Id": "DefaultDataCenter",
            "Max": 20,
            "Racks": [
              {
                "DataNodes": [
                  {
                    "EcShards": 0,
                    "Free": 3,
                    "Max": 20,
                    "PublicUrl": "172.88.0.32:8080",
                    "Url": "172.88.0.32:8080",
                    "Volumes": 17
                  }
                ],
                "Free": 3,
                "Id": "DefaultRack",
                "Max": 20
              }
            ]
          }
        ],
        "Free": 3,
        "Max": 20,
        "layouts": [
          {
            "collection": "",
            "replication": "000",
            "ttl": "",
            "writables": [
              26
            ]
          },
          {
            "collection": "",
            "replication": "000",
            "ttl": "1m",
            "writables": [
              28,
              29,
              30,
              31,
              32,
              33
            ]
          }
        ]
      },
      "Version": "30GB 1.44"
    }
    
    

    But also there were some other volume ids(1/2/3/4/5/6/7) in my volume server:

    curl "http://172.88.0.32:8080/status?pretty=y"
    {
      "Version": "30GB 1.44",
      "Volumes": [
        {
          "Id": 1,
          "Size": 947123,
          "ReplicaPlacement": {
            "SameRackCount": 0,
            "DiffRackCount": 0,
            "DiffDataCenterCount": 0
          },
          "Ttl": {
            "Count": 0,
            "Unit": 0
          },
          "Collection": "",
          "Version": 3,
          "FileCount": 7,
          "DeleteCount": 0,
          "DeletedByteCount": 0,
          "ReadOnly": false,
          "CompactRevision": 0,
          "ModifiedAtSecond": 0
        },
        {
          "Id": 2,
          "Size": 8428,
          "ReplicaPlacement": {
            "SameRackCount": 0,
            "DiffRackCount": 0,
            "DiffDataCenterCount": 0
          },
          "Ttl": {
            "Count": 0,
            "Unit": 0
          },
          "Collection": "",
          "Version": 3,
          "FileCount": 1,
          "DeleteCount": 0,
          "DeletedByteCount": 0,
          "ReadOnly": false,
          "CompactRevision": 0,
          "ModifiedAtSecond": 0
        },
        {
          "Id": 3,
          "Size": 371,
          "ReplicaPlacement": {
            "SameRackCount": 0,
            "DiffRackCount": 0,
            "DiffDataCenterCount": 0
          },
          "Ttl": {
            "Count": 0,
            "Unit": 0
          },
          "Collection": "",
          "Version": 3,
          "FileCount": 1,
          "DeleteCount": 0,
          "DeletedByteCount": 0,
          "ReadOnly": false,
          "CompactRevision": 0,
          "ModifiedAtSecond": 0
        },
        {
          "Id": 4,
          "Size": 43,
          "ReplicaPlacement": {
            "SameRackCount": 0,
            "DiffRackCount": 0,
            "DiffDataCenterCount": 0
          },
          "Ttl": {
            "Count": 0,
            "Unit": 0
          },
          "Collection": "",
          "Version": 3,
          "FileCount": 1,
          "DeleteCount": 0,
          "DeletedByteCount": 0,
          "ReadOnly": false,
          "CompactRevision": 0,
          "ModifiedAtSecond": 0
        },
        {
          "Id": 5,
          "Size": 0,
          "ReplicaPlacement": {
            "SameRackCount": 0,
            "DiffRackCount": 0,
            "DiffDataCenterCount": 0
          },
          "Ttl": {
            "Count": 0,
            "Unit": 0
          },
          "Collection": "",
          "Version": 3,
          "FileCount": 0,
          "DeleteCount": 0,
          "DeletedByteCount": 0,
          "ReadOnly": false,
          "CompactRevision": 0,
          "ModifiedAtSecond": 0
        },
        {
          "Id": 6,
          "Size": 6100757,
          "ReplicaPlacement": {
            "SameRackCount": 0,
            "DiffRackCount": 0,
            "DiffDataCenterCount": 0
          },
          "Ttl": {
            "Count": 0,
            "Unit": 0
          },
          "Collection": "",
          "Version": 3,
          "FileCount": 3,
          "DeleteCount": 0,
          "DeletedByteCount": 0,
          "ReadOnly": false,
          "CompactRevision": 0,
          "ModifiedAtSecond": 0
        },
        {
          "Id": 7,
          "Size": 540,
          "ReplicaPlacement": {
            "SameRackCount": 0,
            "DiffRackCount": 0,
            "DiffDataCenterCount": 0
          },
          "Ttl": {
            "Count": 0,
            "Unit": 0
          },
          "Collection": "",
          "Version": 3,
          "FileCount": 1,
          "DeleteCount": 0,
          "DeletedByteCount": 0,
          "ReadOnly": false,
          "CompactRevision": 0,
          "ModifiedAtSecond": 0
        },
        .........
    }
    
    

    It seemed that volume ids(1/2/3/4/5/6/7) were not writable. I was confused about this.

    Why other volume ids(1/2/3/4/5/6/7) were invisble in layouts and can't be assigned by master?

    System Setup

    • docker images: chrislusf/seaweedfs.
    • Alpine Linux 3.10
    • version 30GB 1.44 linux amd64

    Expected behavior volumes(1/2/3/4/5/6/7) are visible in layouts and can be assigned by master.

    Screenshots image

    Additional context My volume server was started with default numbers of volumes at first.

    # command: 'volume -mserver="master:9333" -port=8080  -metricsPort=9325'
    

    After some tests I stoped docker, and start my volume server with -max 20.

    command: 'volume -max=20 -mserver="master:9333" -port=8080'
    
    opened by hopeday6688 2
  • Multipart uploads fail with

    Multipart uploads fail with "The specified multipart upload does not exist. [...]"

    Describe the bug We use pgbackrest postgres backup tool to send weekly backups to seaweed via S3 filer. This Sunday many backups had failed with the error: The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed. Attemps to re-run backups fail with the same error. We have been using pgbackrest and seaweed for almost a year now and not run into this problem before, and it did not coincide with a seaweed version upgrade.

    System Setup

    • List the command line to start master: /usr/local/sbin/weed master -ip.bind 0.0.0.0 -ip 10.84.101.100 -mdir /opt/seaweedfs/seaweed-backup001.int/master -defaultReplication 001 -garbageThreshold=0.05
    • List the command line to start volume: /usr/local/sbin/weed volume -port 8080 -metricsPort 9327 -publicUrl http://seaweed-backup001.int -max 72 -ip.bind 0.0.0.0 -ip 10.84.101.108 -dir /opt/seaweedfs/seaweed-backup001.int/volume -mserver 10.84.101.100:9333 -dataCenter sto4 -rack rack1 -idle Timeout=300
    • List the command line to start filer: /usr/local/sbin/weed filer -port 8889 -metricsPort 9328 -ip.bind 0.0.0.0 -ip 10.84.101.108 -s3 -master 10.84.101.100:9333 -dataCenter=sto4 -rack=rack1 -s3.config=s3_config.json -concurrentUploadLimitMB=2048 -maxMB=32 OS version: CentOS 8 output of weed version: version 30GB 2.74 182f43ae5fa2aed5a2526aaa23390e9b8d376e63 linux amd64
    • filer.toml
    [postgres]
    enabled = true
    
    hostname = "pg-backup001-db001.int"
    port = 5432
    username = "seaweedfiler"
    password = "<redacted>"
    database = "seaweedfiler"          # create or use an existing database
    sslmode = "require"
    connection_max_idle = 10
    connection_max_open = 10
    connection_max_lifetime_seconds = 300
    
    # if insert/upsert failing, you can disable upsert or update query syntax to match your RDBMS syntax:
    enableUpsert = true
    upsertQuery = """INSERT INTO "%[1]s" (dirhash,name,directory,meta) VALUES($1,$2,$3,$4) ON CONFLICT (dirhash,name) DO UPDATE SET meta = EXCLUDED.meta WHERE "%[1]s".meta 
    != EXCLUDED.meta"""
    

    Expected behavior Multipart uploads should work

    Additional context Seaweed filer outputs this error in log: E1116 08:55:46 13802 filer_multipart.go:73] completeMultipartUpload dev-pgbackrest b4526bed-0534-4024-a238-fff3a24f33dd error: <nil>, entries:0

    Please let me know if I can provide additional details.

    opened by MannerMan 11
  • Fix meta aggregation problems

    Fix meta aggregation problems

    • For mount and filer.meta.backup
    • Fixes unlimited reconnection loop on ReadPersistedLogBuffer fail, allowing to progress to use memory read. Can happen if some logs are missing, and always fails for the latest log while it is in memory
    • Added more waits before retries
    opened by Dando-Real-ITA 1
  • [filer] fs.filer.AggregateFromPeers is not required functionality

    [filer] fs.filer.AggregateFromPeers is not required functionality

    It seems this is not required functionality for SQL store ? https://github.com/chrislusf/seaweedfs/blob/c4e22b5a9a38979a922a400d0fb3a4d74dbe417c/weed/server/filer_server.go#L145

    i have Since there was no connection between DC

    I1115 11:08:44     1 meta_aggregator.go:100] connecting to peer filer 172.24.197.137:9090: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.24.197.137:19090: i/o timeout"
    

    System Setup

    2.78
    
    opened by kmlebedev 1
  • [mount] Fuse Mount seems not writeable through MacOS Finder

    [mount] Fuse Mount seems not writeable through MacOS Finder

    Bug description

    I use weed mount on my macOS Developer machine (exact environment description below). The mountpoint is created correctly (a.k.a no startup errors for weed mount)

    The Bug:
    I can't write to the FuseMount when opening it via the desktop icon. (see screenshots)

    What Works:
    Accessing the same mount directly at the mountpoint in the filesystem

    System Setup

    My git repo

    https://github.com/bjesuiter/macnas-seaweedfs/tree/bare-metal-macos/baremetal. All relevant work is in the baremetal folder (outside the baremetal folder is an experiment with docker-compose)

    My Weed Commands - TL;DR (for details, see additional context section)

    weed server -dir="./data" --filer=true
    weed mount -filer=localhost:8888 -dir=mnt/SeaweedFS -allowOthers -volumeServerAccess=direct -nonempty -replication=000 -filer.path=/
    

    macOS

    macOS BigSur 11.6.1

    macFUSE (OSXFUSE before, but was renamed back to macFuse after BigSur Release)

    4.2.1

    weed version

    version 8000GB 2.76 fc9e246592351085b31ce9ad0d6a4790fa54b4f1 darwin amd64

    filer.toml

    ####################################################
    # Customizable filer server options
    ####################################################
    [filer.options]
        # with http DELETE, by default the filer would check whether a folder is empty.
        # recursive_delete will delete all sub folders and files, similar to "rm -Rf"
        recursive_delete = false
    
        ####################################################
        # The following are filer store options
        ####################################################
    
    [leveldb2]
        # local on disk, mostly for simple single-machine setup, fairly scalable
        # faster than previous leveldb, recommended.
        enabled = true
        dir     = "./filerldb2" # directory to store level db files
    

    Expected behavior

    Accessing the fuse mount from the desktop (as virtual drive) should work exactly the same as accessing it from the mountpoint inside the filesystem.

    Screenshots

    My Fuse Mount on the Desktop:

    Bildschirmfoto 2021-11-13 um 15 51 59

    Error when dragging a File into it: Bildschirmfoto 2021-11-13 um 15 53 36

    Interestingly, I CAN drag'n drop a file into the exact mount location on the file system (it's the baremetal/mnt/SeaweedFS Folder):

    Bildschirmfoto 2021-11-13 um 15 55 45

    Additional context

    My Weed Commands - Details

    In my git repo setup, I use 'bonnie' cli as a script alias manager. The scripts I used are from this baremetal/bonnie.toml file.

    I used bonnie start & bonnie mount test commands.

    This is not relevant to this issue, however, I included it so that you know whats going on there. Note: The doppler run cli is irrelevant here, it does only fill in environment variables.

    help wanted 
    opened by bjesuiter 5
Releases(2.80)
Owner
Chris Lu
https://github.com/chrislusf/seaweedfs SeaweedFS the distributed file system and object store for billions of small files ...
Chris Lu
GeeseFS is a high-performance, POSIX-ish S3 (Yandex, Amazon) file system written in Go

GeeseFS is a high-performance, POSIX-ish S3 (Yandex, Amazon) file system written in Go Overview GeeseFS allows you to mount an S3 bucket as a file sys

Yandex.Cloud 169 Nov 29, 2021
Bigfile -- a file transfer system that supports http, rpc and ftp protocol https://bigfile.site

Bigfile ———— a file transfer system that supports http, rpc and ftp protocol 简体中文 ∙ English Bigfile is a file transfer system, supports http, ftp and

null 212 Dec 1, 2021
Embed arbitrary resources into a go executable at runtime, after the executable has been built.

ember Ember is a lightweight library and tool for embedding arbitrary resources into a go executable at runtime. The resources don't need to exist at

null 51 Nov 12, 2021
Cross-platform file system notifications for Go.

File system notifications for Go fsnotify utilizes golang.org/x/sys rather than syscall from the standard library. Ensure you have the latest version

fsnotify 6.2k Nov 30, 2021
FSManager - Tree view Simple util to displays the directory structure of a path or of the disk in a drive graphically.

FSManager - Tree view Simple util to displays the directory structure of a path or of the disk in a drive graphically. If you don't specify a drive or

Kostiantyn Denysov 1 Oct 9, 2021
High Performance, Kubernetes Native Object Storage

MinIO Quickstart Guide MinIO is a High Performance Object Storage released under Apache License v2.0. It is API compatible with Amazon S3 cloud storag

High Performance, Kubernetes Native Object Storage 30.5k Dec 5, 2021
A virtual file system for small to medium sized datasets (MB or GB, not TB or PB). Like Docker, but for data.

AetherFS assists in the production, distribution, and replication of embedded databases and in-memory datasets. You can think of it like Docker, but f

mya 6 Nov 22, 2021
A small cross-platform fileserver for CTFs and penetration tests.

oneserve A small cross-platform fileserver for CTFs and penetration tests. Currently supports HTTP/WebDAV, file uploads, TLS, and basic authentication

null 1 Nov 10, 2021
A cross-platform changelog file generator

changelogger A changelog file generator heavily inspired on towncrier. Why? I'm looking for an alternative to towncrier that doesn't require a Python

Diego Garcia 12 Oct 25, 2021
A blazingly-fast simple-to-use tool to find duplicate files on your computer, portable hard drives etc.

A fast and simple tool to find duplicate files (photos, videos, music, documents) on your computer, portable hard drives etc.

Manu Manjunath 190 Nov 16, 2021
An API for handling file meta data

dp-files-api An API for handling file meta data Getting started Run make debug Dependencies No further dependencies other than those defined in go.mod

ONS Digital 0 Nov 29, 2021
app-services-go-linter plugin analyze source tree of Go files and validates the availability of i18n strings in *.toml files

app-services-go-linter app-services-go-linter plugin analyze source tree of Go files and validates the availability of i18n strings in *.toml files. A

Red Hat Developer 2 Nov 29, 2021
Split text files into gzip files with x lines

hakgzsplit split lines of text into multiple gzip files

Luke Stephens (hakluke) 6 Aug 30, 2021
Easily create Go files from stub files

go-stubs Easily create .go files from stub files in your projects. Usage go get github.com/nwby/go-stubs Create a stub file: package stubs type {{.Mo

Sam Newby 3 Sep 26, 2021
Download an upload large files to Google Drive (API v3)

gdriver gdriver is a command-line tool, written in Go, used for uploading and downloading large personal files from Google Drive (API v3). The tool pr

Marcin Tojek 62 Nov 18, 2021
A Docker-powered stateless API for PDF files.

Gotenberg provides a developer-friendly API to interact with powerful tools like Chromium and LibreOffice to convert many documents (HTML, Markdown, Word, Excel, etc.) to PDF, transform them, merge them, and more!

Gotenberg 3.1k Dec 1, 2021
Abstract File Storage

afs - abstract file storage Please refer to CHANGELOG.md if you encounter breaking changes. Motivation Introduction Usage Matchers Content modifiers S

Viant, Inc 163 Nov 30, 2021
Dragonfly is an intelligent P2P based image and file distribution system.

Dragonfly Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in o

dragonflyoss 5.7k Nov 30, 2021
A FileSystem Abstraction System for Go

A FileSystem Abstraction System for Go Overview Afero is a filesystem framework providing a simple, uniform and universal API interacting with any fil

Steve Francia 4.1k Dec 5, 2021