AWS experts chinna doubt va

[DesiPokiri]

I am trying to copy data from one s3 bucket in one aws account to another s3 bucket in another aws account .. since the bucket data is huge like around 750 Gb, I am getting below error in the middle of the transfer.

client_loop: send disconnect: connection  reset

how to avoid this error?

I googled that we should edit config file in .ssh to make the tcpkeepalive no.

will this actually work. I am getting the error after like 3 hours. So am not in a luxury to try trial and error methods, suggest if you faced this issue va

[Anta Assamey]

AWS full form tappite naaku em teliyadu ...

[Spartan]

two options

run a script in loop which will restart evry 3 hrs

leda enable replication but it will copy only new objects not the old ones

inko option undi where u can create a manifest file dump of objects and send it to dest bucket whcih will replicate 

step by step details evening post chesta

[dasari4kntr]

4 minutes ago, Spartan said:

two options

run a script in loop which will restart evry 3 hrs

leda enable replication but it will copy only new objects not the old ones

inko option undi where u can create a manifest file dump of objects and send it to dest bucket whcih will replicate 

step by step details evening post chesta

Aws datasync wont work…?

[DesiPokiri]

5 minutes ago, Spartan said:

two options

run a script in loop which will restart evry 3 hrs

leda enable replication but it will copy only new objects not the old ones

inko option undi where u can create a manifest file dump of objects and send it to dest bucket whcih will replicate 

step by step details evening post chesta

Madhyalo fail aithe only remaining data or files pampinchadaniki process undha, not grepping for whole data in the bucket 

[tennisluvrredux]

15 minutes ago, DesiPokiri said:

I am trying to copy data from one s3 bucket in one aws account to another s3 bucket in another aws account .. since the bucket data is huge like around 750 Gb, I am getting below error in the middle of the transfer.

client_loop: send disconnect: connection  reset

how to avoid this error?

I googled that we should edit config file in .ssh to make the tcpkeepalive no.

will this actually work. I am getting the error after like 3 hours. So am not in a luxury to try trial and error methods, suggest if you faced this issue va

https://brakertech.com/ssh-keeps-disconnecting/

[DesiPokiri]

3 minutes ago, dasari4kntr said:

Aws datasync wont work…?

Aws data sync won’t work in cross accounts andhuke vere process ki moved 

[tennisluvrredux]

1 minute ago, DesiPokiri said:

Madhyalo fail aithe only remaining data or files pampinchadaniki process undha, not grepping for whole data in the bucket 

 

18 minutes ago, DesiPokiri said:

I am trying to copy data from one s3 bucket in one aws account to another s3 bucket in another aws account .. since the bucket data is huge like around 750 Gb, I am getting below error in the middle of the transfer.

client_loop: send disconnect: connection  reset

how to avoid this error?

I googled that we should edit config file in .ssh to make the tcpkeepalive no.

will this actually work. I am getting the error after like 3 hours. So am not in a luxury to try trial and error methods, suggest if you faced this issue va

You can use Minio client aka mc, link https://github.com/minio/mc. It is open source and compatible with AWS S3.

Minio client has resume session command which will upload the object where it was left in last disconnect.

I have created a Youtube video showing same https://www.youtube.com/watch?v=Cp7FHbmBrEo

Hope it helps. Disclaimer : I work for Minio

 

Posted from a URL I found while looking for information pertaining to your error, might help you. Check it out. 

[DesiPokiri]

1 minute ago, tennisluvrredux said:

 

You can use Minio client aka mc, link https://github.com/minio/mc. It is open source and compatible with AWS S3.

Minio client has resume session command which will upload the object where it was left in last disconnect.

I have created a Youtube video showing same https://www.youtube.com/watch?v=Cp7FHbmBrEo

Hope it helps. Disclaimer : I work for Minio

 

Posted from a URL I found while looking for information pertaining to your error, might help you. Check it out. 

Thanks baa will try this 

[Spartan]

10 minutes ago, DesiPokiri said:

Madhyalo fail aithe only remaining data or files pampinchadaniki process undha, not grepping for whole data in the bucket 

replication enable chesi pedite fail avvadu 

old objects kuda second method works

recent ga almost 750gb replicate chesa

[Spartan]

11 minutes ago, dasari4kntr said:

Aws datasync wont work…?

no

[DesiPokiri]

5 minutes ago, Spartan said:

replication enable chesi pedite fail avvadu 

old objects kuda second method works

recent ga almost 750gb replicate chesa

Aa cli commands ikkada pettu vaa, and how did you enable replication? In cli or in bucket policy ?

[Spartan]

13 minutes ago, DesiPokiri said:

Aa cli commands ikkada pettu vaa, and how did you enable replication? In cli or in bucket policy ?

u need to setup bucketpolicy and permissions.

follow this below config for setting up replication.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication-how-setup.htmlhttps://aws.amazon.com/blogs/storage/replicating-existing-objects-between-s3-buckets/

 

[DesiPokiri]

Looks like this issue is solved, for future reference if anyone encountered it.

fix is add “tcp_keepalive = false” in the config File and that will do the magic..

Thanks @Spartan @tennisluvrredux baas

[Spartan]

52 minutes ago, DesiPokiri said:

Looks like this issue is solved, for future reference if anyone encountered it.

fix is add “tcp_keepalive = false” in the config File and that will do the magic..

Thanks @Spartan @tennisluvrredux baas

just fyi Shell script is old fashioned way, for reliable copy use AWS replication