K
K
Ka4a2017-11-27 12:30:09
Amazon Web Services
Ka4a, 2017-11-27 12:30:09

Why can't upload 130Gb backup to S3?

Hello. Need help with the problem of uploading a large backup to S3 (aws). The problem is that it fails with an error Connection reset by peer. How to fix? Thank you
Full exhaust

10001 0 5242880
10001 57344 5242880
10005 0 5242880
10005 57344 5242880
10002 0 5242880
10002 57344 5242880
10002 114688 5242880
10002 172032 5242880
10003 0 5242880
10003 57344 5242880
10003 0 5242880
10003 57344 5242880
10006 0 5242880
10006 57344 5242880
10003 0 5242880
10003 57344 5242880
10006 0 5242880
10006 57344 5242880
10006 0 5242880
10006 57344 5242880
Traceback (most recent call last):
  File "/usr/local/bin/s3upload", line 9, in <module>
    load_entry_point('s3upload==0.2.4', 'console_scripts', 's3upload')()
  File "/usr/local/lib/python2.7/dist-packages/s3upload.py", line 161, in cli
    progress_cb=cb, replace=True, threads=options.threads)
  File "/usr/local/lib/python2.7/dist-packages/s3upload.py", line 111, in upload
    waiter()
  File "/usr/local/lib/python2.7/dist-packages/s3upload.py", line 98, in waiter
    check_errors()
  File "/usr/local/lib/python2.7/dist-packages/s3upload.py", line 94, in check_errors
    raise exc
thread.error: <DummyProcess(Thread-3, started daemon 139716663592704)> error(104, 'Connection reset by peer')
ERROR: Failed to pack application data!

The command which I perform archiving and loading. (ps - there is a 40GB archive, it uploads normally)
/bin/tar --exclude "imageupload/images" -cf - -C /var/www/blabla.com/www/ . | gzip| s3upload -K KEY -s secretkey -b archive -k test.tar.gz

Ubuntu 14 x64

Answer the question

In order to leave comments, you need to log in

1 answer(s)
R
Roman Mirilaczvili, 2017-11-30
@2ord

The connection may be interrupted due to short-term problems with the network. It is worth trying to download with smaller portions and glue the parts at the end - this is how some programs do.
The portion size can be 10-20 MB.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question