Forum Replies Created

Viewing 15 posts - 1 through 15 (of 309 total)
  • Author
    Posts

  • Andrey
    Keymaster

    Hello Vikas,

    If you copy large files to the login node, the copy process may get terminated due to the wall CPU time limit on the login node. To work around it, you have to options:

    1) Package everything into a single archive and upload via the Jupyter Notebook interface. Connection instructions: https://access.colfaxresearch.com/?p=connect#sec-jup

    2) Alternatively, you can copy files into a compute node. This way, you don’t have to package files into an archive, but you need some prior preparation. First, edit ~/.ssh/config on your local machine and add the following lines to it:

    Host s???-n???
    User uXXXXX
    IdentityFile ~/.ssh/colfax-access-key-XXXXX
    IdentitiesOnly yes
    ProxyCommand ssh -T colfax nc %h %p

    Replace XXXXX with your user ID. Then, get a job on the cluster by running from the login node:
    qsub -I
    when the job starts, you will see the hostname of the node on which it runs in the format “sYYY-nZZZ”. For example, it may be “s001-n123”. Now, go back to your local computer and copy files like this:
    scp -r -v <path/to/directory/at my computer> sYYY-nZZZ:/home/uXXXXX
    Replace XXXXX with your user ID and replace YYY and ZZZ with the ID of the node on which your job is running.

    in reply to: Module about Perl #8720

    Andrey
    Keymaster

    You are welcome!

    in reply to: Module about Perl #8718

    Andrey
    Keymaster

    The last command works in my account:

    u7234@login-1:~$ perl -MXML::Parser -e 1
    u7234@login-1:~$ echo $?
    0

    I checked in your home folder, and it seems that your ~/.bashrc does not have the environment variables for local Perl modules. My ~/.bashrc does have these lines, and they were added there when I ran “cpan”. This is what they look like:

    u7234@login-1:~$ tail -5 ~/.bashrc
    PATH="/home/u7234/perl5/bin${PATH:+:${PATH}}"; export PATH;
    PERL5LIB="/home/u7234/perl5/lib/perl5${PERL5LIB:+:${PERL5LIB}}"; export PERL5LIB;
    PERL_LOCAL_LIB_ROOT="/home/u7234/perl5${PERL_LOCAL_LIB_ROOT:+:${PERL_LOCAL_LIB_ROOT}}"; export PERL_LOCAL_LIB_ROOT;
    PERL_MB_OPT="--install_base \"/home/u7234/perl5\""; export PERL_MB_OPT;
    PERL_MM_OPT="INSTALL_BASE=/home/u7234/perl5"; export PERL_MM_OPT;

    To fix this issue in your account, I suggest that you remove the CPAN install and re-run it like this:

    u7234@s001-n023:~$ rm -rf .cpan 
    u7234@s001-n023:~$ cpan
    Loading internal null logger. Install Log::Log4perl for logging messages
    
    CPAN.pm requires configuration, but most of it can be done automatically.
    If you answer 'no' below, you will enter an interactive dialog for each
    configuration option instead.
    
    Would you like to configure as much as possible automatically? [yes] 
    
    Use of uninitialized value $what in concatenation (.) or string at /usr/share/perl/5.26/App/Cpan.pm line 669, <STDIN> line 1.
     <install_help>
    
    Warning: You do not have write permission for Perl library directories.
    
    To install modules, you need to configure a local Perl library directory or
    escalate your privileges.  CPAN can help you by bootstrapping the local::lib
    module or by configuring itself to use 'sudo' (if available).  You may also
    resolve this problem manually if you need to customize your setup.
    
    What approach do you want?  (Choose 'local::lib', 'sudo' or 'manual')
     [local::lib] 
    
    Autoconfiguration complete.
    
    Attempting to bootstrap local::lib...
    
    Writing /home/u7234/.cpan/CPAN/MyConfig.pm for bootstrap...
    commit: wrote '/home/u7234/.cpan/CPAN/MyConfig.pm'
    Fetching with HTTP::Tiny:
    http://www.cpan.org/authors/01mailrc.txt.gz
    Reading '/home/u7234/.cpan/sources/authors/01mailrc.txt.gz'
    ............................................................................DONE
    Fetching with HTTP::Tiny:
    http://www.cpan.org/modules/02packages.details.txt.gz
    Reading '/home/u7234/.cpan/sources/modules/02packages.details.txt.gz'
      Database was generated on Thu, 11 Apr 2019 19:54:23 GMT
      HTTP::Date not available
    .............
      New CPAN.pm version (v2.26) available.
      [Currently running version is v2.18]
      You might want to try
        install CPAN
        reload cpan
      to both upgrade CPAN.pm and run the new version without leaving
      the current session.
    
    ...............................................................DONE
    Fetching with HTTP::Tiny:
    http://www.cpan.org/modules/03modlist.data.gz
    Reading '/home/u7234/.cpan/sources/modules/03modlist.data.gz'
    DONE
    Writing /home/u7234/.cpan/Metadata
    Fetching with HTTP::Tiny:
    http://www.cpan.org/authors/id/H/HA/HAARG/local-lib-2.000024.tar.gz
    Fetching with HTTP::Tiny:
    http://www.cpan.org/authors/id/H/HA/HAARG/CHECKSUMS
    Checksum for /home/u7234/.cpan/sources/authors/id/H/HA/HAARG/local-lib-2.000024.tar.gz ok
    'YAML' not installed, will not store persistent state
    Configuring H/HA/HAARG/local-lib-2.000024.tar.gz with Makefile.PL
    Attempting to create directory /home/u7234/perl5
    
    Checking if your kit is complete...
    Looks good
    Generating a Unix-style Makefile
    Writing Makefile for local::lib
    Writing MYMETA.yml and MYMETA.json
      HAARG/local-lib-2.000024.tar.gz
      /usr/bin/perl Makefile.PL --bootstrap -- OK
    Running make for H/HA/HAARG/local-lib-2.000024.tar.gz
    cp lib/POD2/PT_BR/local/lib.pod blib/lib/POD2/PT_BR/local/lib.pod
    cp lib/lib/core/only.pm blib/lib/lib/core/only.pm
    cp lib/POD2/DE/local/lib.pod blib/lib/POD2/DE/local/lib.pod
    cp lib/local/lib.pm blib/lib/local/lib.pm
    Manifying 4 pod documents
      HAARG/local-lib-2.000024.tar.gz
      /usr/bin/make -- OK
    Running make test
    PERL_DL_NONLAZY=1 "/usr/bin/perl" "-I/home/u7234/perl5/lib/perl5" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
    t/bad_variables.t ...... ok   
    t/carp-mismatch.t ...... ok   
    t/classmethod.t ........ ok   
    t/de-dup.t ............. ok   
    t/lib-core-only.t ...... ok   
    t/pipeline.t ........... ok   
    t/shell.t .............. ok     
    t/stackable.t .......... ok     
    t/subroutine-in-inc.t .. ok   
    t/taint-mode.t ......... ok   
    All tests successful.
    Files=10, Tests=148,  4 wallclock secs ( 0.07 usr  0.04 sys +  1.20 cusr  1.25 csys =  2.56 CPU)
    Result: PASS
      HAARG/local-lib-2.000024.tar.gz
      /usr/bin/make test -- OK
    Running make install
    Manifying 4 pod documents
    Installing /home/u7234/perl5/lib/perl5/local/lib.pm
    Installing /home/u7234/perl5/lib/perl5/POD2/PT_BR/local/lib.pod
    Installing /home/u7234/perl5/lib/perl5/POD2/DE/local/lib.pod
    Installing /home/u7234/perl5/lib/perl5/lib/core/only.pm
    Installing /home/u7234/perl5/man/man3/POD2::DE::local::lib.3pm
    Installing /home/u7234/perl5/man/man3/POD2::PT_BR::local::lib.3pm
    Installing /home/u7234/perl5/man/man3/lib::core::only.3pm
    Installing /home/u7234/perl5/man/man3/local::lib.3pm
    Appending installation info to /home/u7234/perl5/lib/perl5/x86_64-linux-gnu-thread-multi/perllocal.pod
      HAARG/local-lib-2.000024.tar.gz
      /usr/bin/make install  -- OK
    
    local::lib is installed. You must now add the following environment variables
    to your shell configuration files (or registry, if you are on Windows) and
    then restart your command line shell and CPAN before installing modules:
    
    PATH="/home/u7234/perl5/bin${PATH:+:${PATH}}"; export PATH;
    PERL5LIB="/home/u7234/perl5/lib/perl5${PERL5LIB:+:${PERL5LIB}}"; export PERL5LIB;
    PERL_LOCAL_LIB_ROOT="/home/u7234/perl5${PERL_LOCAL_LIB_ROOT:+:${PERL_LOCAL_LIB_ROOT}}"; export PERL_LOCAL_LIB_ROOT;
    PERL_MB_OPT="--install_base \"/home/u7234/perl5\""; export PERL_MB_OPT;
    PERL_MM_OPT="INSTALL_BASE=/home/u7234/perl5"; export PERL_MM_OPT;
    
    Would you like me to append that to /home/u7234/.bashrc now? [yes] 
    
    commit: wrote '/home/u7234/.cpan/CPAN/MyConfig.pm'
    
    You can re-run configuration any time with 'o conf init' in the CPAN shell
    Terminal does not support AddHistory.
    
    cpan shell -- CPAN exploration and modules installation (v2.18)
    Enter 'h' for help.
    
    cpan[1]> 

    (press Ctrl+D to exit).

    There may be a nicer way to do this but I don’t user Perl enough to troubleshoot.

    in reply to: Module about Perl #8716

    Andrey
    Keymaster

    Here is how you can install the required Perl module locally:

    1) Get a terminal on a compute node (e.g., execute “qsub -I”)
    2) Run “cpan” and agree to all defaults
    3) Log out of the terminal session
    4) Log back in to the compute node to reread your updated ~/.bashrc
    5) Run “cpan install XML::Parser”. It takes a while.

    Now your application should work.

    in reply to: Module about Perl #8714

    Andrey
    Keymaster

    I am sorry, I have a hard time understanding. Where did the pyrouge package come from? I installed pyrouge in my account, but could not find a Perl tool inside.

    in reply to: Cannot use mpirun anymore (ssh password) #8712

    Andrey
    Keymaster

    You are welcome!

    in reply to: Module about Perl #8711

    Andrey
    Keymaster

    Hello Yiming,

    could you please provide a minimal reproducer code or command for the issue?

    Andrey

    in reply to: Cannot use mpirun anymore (ssh password) #8706

    Andrey
    Keymaster

    It seems that you have re-generated the RSA key on the cloud since your account was created. As a result, the nodes do not recognize you when you log in from another node. To fix authorization, you need to add the new public key to your authorized_keys file on the cloud. This way, when you SSH from one node to another, your own key will be authorized. Please run the following:

    cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

    It should work after that.

    in reply to: Cannot use mpirun anymore (ssh password) #8704

    Andrey
    Keymaster

    Could you please provide a minimal reproducer script/command?

    in reply to: Different node is showing after doing SSH colfax #8691

    Andrey
    Keymaster

    This is the expected result after our move to Ubuntu. The hostnames of compute nodes have also changed (previously c009-n0.., now s001-n0..). Don’t make too much of it, the hardware that you are running on and your data is the same.

    in reply to: I can't transfer files #8677

    Andrey
    Keymaster

    If the error is still occurring, please provide the output of the same command with the ‘-v’ argument, e.g.,
    scp -v /Users/molivag/Documents/Cursera/hello-world/main.cc colfax:/home/u24835/

    in reply to: I can connect but something goes wierd #8676

    Andrey
    Keymaster

    It probably has to do with the locale settings on the computer from which you are logging in. You can probably ignore it, unless it interferes with compilation or terminal I/O. If you find any interference, you can try to set the environment variable LC_ALL=en_US before logging in to the cluster. For example, if you are logging in from a Bash shell, you would do:

    export LC_ALL=en_US
    ssh colfax
    in reply to: Unable to create a conda environment on node #8673

    Andrey
    Keymaster

    Hello, please provide more detail about the issue that you are seeing: what you are trying to do, what you expect to see, and what actually happens. If you can, please attach screenshots or terminal output.

    in reply to: Couldn't ssh into the cluster #8666

    Andrey
    Keymaster

    It should be fixed now, please try again.

    in reply to: Matlab Installation #8660

    Andrey
    Keymaster

    Unfortunately, we are not providing MATLAB.

Viewing 15 posts - 1 through 15 (of 309 total)