Home Discussion Colfax Cluster Seeking better way to copy files of bigger size from local computer to colfax.

This topic contains 2 replies, has 2 voices, and was last updated by  vikas 7 months ago.

Viewing 3 posts - 1 through 3 (of 3 total)
  • Author
    Posts
  • #8722

    vikas
    Participant

    Hello Sir,
    For copying directory, I have used the following command,

    scp -r -v <path/to/directory/at my computer> colfax:/home/uXXXXXX

    I getting the following error.

    cat: Connection reset by peer.
    packet_write_wait: Connection to UNKNOWN port 0: Broken pipe
    lost connection

    Could you please suggest a better way of copying a directory of size 100 GB?

    Thank you in advance.

    Regards.
    Vikas

    #8724

    Andrey
    Keymaster

    Hello Vikas,

    If you copy large files to the login node, the copy process may get terminated due to the wall CPU time limit on the login node. To work around it, you have to options:

    1) Package everything into a single archive and upload via the Jupyter Notebook interface. Connection instructions: https://access.colfaxresearch.com/?p=connect#sec-jup

    2) Alternatively, you can copy files into a compute node. This way, you don’t have to package files into an archive, but you need some prior preparation. First, edit ~/.ssh/config on your local machine and add the following lines to it:

    Host s???-n???
    User uXXXXX
    IdentityFile ~/.ssh/colfax-access-key-XXXXX
    IdentitiesOnly yes
    ProxyCommand ssh -T colfax nc %h %p

    Replace XXXXX with your user ID. Then, get a job on the cluster by running from the login node:
    qsub -I
    when the job starts, you will see the hostname of the node on which it runs in the format “sYYY-nZZZ”. For example, it may be “s001-n123”. Now, go back to your local computer and copy files like this:
    scp -r -v <path/to/directory/at my computer> sYYY-nZZZ:/home/uXXXXX
    Replace XXXXX with your user ID and replace YYY and ZZZ with the ID of the node on which your job is running.

    #8726

    vikas
    Participant

    Hello Andrey

    Thank you for your reply.
    I tried the above suggested method.
    Still, I am not able to send all the files. I am getting the following message.

    fourier_val_data.bin 48% 3146MB 6.8MB/s 08:14 ETAN
    cat: Connection reset by peer.
    packet_write_wait: Connection to UNKNOWN port 0: Broken pipe
    packet_write_wait: Connection to UNKNOWN port 0: Broken pipe
    lost connection

    Am I doing something wrong?
    Could you please help me to fix this issue?

    Thank you,
    Vikas

Viewing 3 posts - 1 through 3 (of 3 total)

You must be logged in to reply to this topic.