#cat backup-mongodb.sh

#!/bin/bash
# need root 
export AWS_ACCESS_KEY_ID=xxx && \
        export AWS_SECRET_ACCESS_KEY=xxx && \
        /home/tx/workspace/go/bin/gosync -l "error" /mnt/mongodb s3://qianlima/mongodb

sh backup-mongodb.sh #first time ok

sh backup-mongodb.sh #again error

1410594494377402743 [Error] Received error 'The AWS Access Key Id you provided does not exist in our records.'

sh backup-mongodb.sh #again ok

sh backup-mongodb.sh #again error

1410594469147053830 [Error] Received error 'The AWS Access Key Id you provided does not exist in our records.'

beside log-level and concurrent, there should be flags to set aws key pairs so that gosync can be used in another program easily without setting the ENV variables.

To support Windows, convert backslashes in path to slashes.

Problem gosync supposed that directory separetor is slash. But backslash is used on Windows.

Solution Convert backslash (directory separetor) to slash by using filepath.ToSlash() function.

For example:

[email protected]:~/sudandola-builder/output$ gosync -l 'debug' . s3://samplebucket
1412960077220223640 [Info] Setting log level 'debug'.
1412960077220322948 [Info] Setting source to '.'.
1412960077220329709 [Info] Setting target to 's3://samplebucket'.
1412960077220337792 [Info] Setting concurrent transfers to '20'.
1412960077220351742 [Info] Syncing to S3.
1412960077895114765 [Debug] Loaded '3266' files from '.'.
1412960077895127120 [Info] Loading local files complete.
1412960077895136487 [Info] Looking up region for bucket 'amplebucket'.
1412960077895142600 [Debug] Looking for bucket 'amplebucket' in 'ap-southeast-1'.
1412960083887147934 [Error] Received error 'The specified bucket does not exist'

Multiple 's':

[email protected]:~/sudandola-builder/output$ gosync -l 'debug' . s3://sssamplebucket
1412960329501528504 [Info] Setting log level 'debug'.
1412960329501623024 [Info] Setting source to '.'.
1412960329501629625 [Info] Setting target to 's3://sssamplebucket'.
1412960329501639645 [Info] Setting concurrent transfers to '20'.
1412960329501655483 [Info] Syncing to S3.
1412960330173879163 [Debug] Loaded '3266' files from '.'.
1412960330173890884 [Info] Loading local files complete.
1412960330173900326 [Info] Looking up region for bucket 'amplebucket'.
1412960330173905764 [Debug] Looking for bucket 'amplebucket' in 'ap-southeast-2'.
1412960335638316002 [Error] Received error 'The specified bucket does not exist'

The prefixed downloads was trimming the prefix off key paths. This cause downloads formatted as s3://mybucket/mysubbucket to not download properly.

Fixed by returning full file path rather than relative.

Currently the region searching logic will skip over them, this could be fixed by adding code to correctly lookup a bucket region.

https://github.com/brettweavnet/gosync/blob/master/gosync/s3.go#L73-L78

Currently gosync fails if upload file size is more than 2GB. I think the cause is that ioutil.ReadFile is called in sync.go : line 173.

Reference URL: https://code.google.com/p/go/issues/detail?id=2743

Here is my results of execution.

$ gosync sync isos s3://s3sync-test/test1
Syncing isos with s3://s3sync-test/test1
panic: read isos/2Gdd.img: invalid argument

goroutine 1 [running]:
github.com/brettweavnet/gosync/gosync.func·001(0xc2000b2750, 0xd, 0xc2000af4b0, 0xc2000af5f0, 0x0, ...)
    /Users/ryuta/.go/src/github.com/brettweavnet/gosync/gosync/sync.go:175 +0x123
path/filepath.walk(0xc2000b2750, 0xd, 0xc2000af4b0, 0xc2000af5f0, 0x7607f8, ...)
    /usr/local/go/src/pkg/path/filepath/path.go:341 +0x70
path/filepath.walk(0x7fff5fbff8ba, 0x4, 0xc2000af4b0, 0xc2000af500, 0x7607f8, ...)
    /usr/local/go/src/pkg/path/filepath/path.go:359 +0x32a
path/filepath.Walk(0x7fff5fbff8ba, 0x4, 0x7607f8, 0x300000050, 0x1ffbe, ...)
    /usr/local/go/src/pkg/path/filepath/path.go:380 +0xb5
github.com/brettweavnet/gosync/gosync.loadLocalFiles(0x7fff5fbff8ba, 0x4, 0x720040)
    /Users/ryuta/.go/src/github.com/brettweavnet/gosync/gosync/sync.go:184 +0x8e
github.com/brettweavnet/gosync/gosync.(*SyncPair).syncDirToS3(0xc2000af410, 0x4, 0xc2000af400)
    /Users/ryuta/.go/src/github.com/brettweavnet/gosync/gosync/sync.go:59 +0x3d
github.com/brettweavnet/gosync/gosync.(*SyncPair).Sync(0xc2000af410, 0xc2000af410, 0x760ac8)
    /Users/ryuta/.go/src/github.com/brettweavnet/gosync/gosync/sync.go:30 +0x119
main.func·001(0xc2000c5520)
    /Users/ryuta/Repos/gosync/gosync.go:39 +0x38f
github.com/codegangsta/cli.Command.Run(0x29cf80, 0x4, 0x0, 0x0, 0x2d8390, ...)
    /Users/ryuta/.go/src/github.com/codegangsta/cli/command.go:25 +0x2a5
github.com/codegangsta/cli.(*App).Run(0xc2000c7150, 0xc200090000, 0x4, 0x4)
    /Users/ryuta/.go/src/github.com/codegangsta/cli/app.go:57 +0x5f7
main.main()
    /Users/ryuta/Repos/gosync/gosync.go:49 +0x15f

goroutine 2 [syscall]:
$

Regards