Server configuration virtual environment, jupyter, common commands

Catalog

Server configuration virtual environment, jupyter

Because the server is not used by one person and needs different environments according to the needs, it is convenient to install the virtual environment on the server. This article mainly records the methods and problems in the virtual environment installation process. (according to the summary of selfless sharing by online tycoons)

Virtual environment creation

The following commands are used to create a general virtual environment:

CONDA create - n your env? Name python = x.x (your env? Name refers to the name of the virtual environment you created)

Activate virtual environment:

Linux:  source activate your_env_name

Windows: activate your_env_name

Exit virtual environment:

Linux:  source deactivate your_env_name

Windows:deactivate your_env_name #You can also use 'activate root' to switch back to the root environment

To delete a virtual environment and a package:

conda remove -n your_env_name --all

conda remove --name your_env_name package_name  #Delete a package

Image source add

After activation, install the package according to your own needs. Generally speaking, you need to add the image source (to speed up the download speed) when using the conda installation package. The main recommended image source is Tsinghua's image source. Of course, there are other image sources.

conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/
conda config --set show_channel_urls yes
#Attach a pytorch Library
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud//pytorch/

There are two ways to install a package using pip.
The first specifies the mirror source:

#Either way
pip install Library name -i https://pypi.tuna.tsinghua.edu.cn/simple 
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple Library name

The second is to permanently add a mirror source:
Linux: try to modify ~ /. pip/pip.conf (if you don't need to create it yourself): enter VI ~ /. pip/pip.conf in the user directory, input the following contents into the file (note whether the input is complete or not, if not, manually supplement it), and then press Esc key to input: wq save to exit vim, if you don't report an error (E212: Can't open file for writing), report the error origin Because there is no such file

#Create a new. PIP folder, and then a new file pip.conf
$ cd ~ 
$ mkdir .pip
$ cd .pip
$ vi pip.conf

#modify
[global]
index-url = https://pypi.tuna.tsinghua.edu.cn/simple

Windows: create a PIP directory directly in the user directory, such as C:\Users\xx\pip, and create a new file pip.ini

[global]
index-url = https://pypi.tuna.tsinghua.edu.cn/simple

When specifying the alicloud image source, if there is an error reported, The repository located at mirrors.aliyun.com is not a trusted "* *".

#One time solution
pip install scrapy -i http://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com 

#Permanent solution, add under pip.conf
[global]
trusted-host=mirrors.aliyun.com

There is a problem with pip install. You can install the package with the following command:

python -m pip install Library name 

#Server pip installation error, just add one -- user after install 
python -m pip install --user --upgrade pip==9.0.3

The server is configured with jupyter and pycharm remote connection

After the server is configured, you want to debug the local code, so you need to connect to the service remotely, and use the resources and environment of the server.

Install jupyter

Install the jupyter notebook (this process is followed by the activation of the environment)

conda install jupyter notebook

Configure jupyter notebook remote access

The default jupyter notebook does not require a configuration file. Therefore, you need to generate the file with the following command.
The main configuration is as follows:

  1. Set remote access password
  2. Set accessible ip, global access
  3. Disable server-side startup browser

Build profile

jupyter notebook --generate-config

If the root user executes the above command, a problem will occur

Running as root it not recommended. Use --allow-root to bypass.

The prompt is obvious. The root user needs to add the -- allow root option when executing.

jupyter notebook --generate-config --allow-config

After successful execution, the following message appears:

Writing default config to: /root/.jupyter/jupyter_notebook_config.py

Open ipython

ipython

Generate the key. After passwd(), you will be asked to enter the password twice (the password is used for the client to log in to jupyter, and for the configuration file after the key)

In [1]: from notebook.auth import passwd
In [2]: passwd()
Enter password: 
Verify password: 
Out[2]: 'sha1:8361f5f08937:081cdf40730cb5548e2c213ddd36813a5313192f'

Modify the configuration file: (note that the configuration file is python code, so you need to remove the "ා" in front of the modified settings.)

c.NotebookApp.ip='*'
c.NotebookApp.password = u'sha:ce...The key you just copied'
c.NotebookApp.open_browser = False
c.NotebookApp.port =8888 #You can specify a port by yourself and use it when accessing

Add conda virtual environment to the Jupyter Notebook Kernel

conda install nb_conda  #Just install the plug-in

Configure pycharm remote access


Click the option "Tools" - > "Start SSH session..." , select the corresponding server, that is, you can open the Terminal of the server under pycharm.

Pit for pytorch installation

When using the conda command to install pytorch, a very hard error will appear:
ImportError: dlopen: cannot load any more object with static TLS
After searching Baidu, we didn't solve the problem in stack overflow. Finally, my younger martial brother found a solution in a discussion area in git (learn from my younger martial brother). The bug is that after the server conda installs pytorch, import torch is normal in cmd, but an error will be reported in the configured jupyter. The method given is to install pytorch with pip.

conda uninstall pytorch #Uninstall pytorch installed by conda
pip install torch torchvision

Upgrade of underlying Library

If the system version is too low, pytorch is not supported and an error is reported:
ImportError: /lib64/libstdc++.so.6: version 'CXXABI_1.3.9' not found(required by anaconda3/lib/python3.6/site-packages/scipy/sparse/_sparsetools.cpython-36m-x86_64-linux-gnu.so)

To solve the glibc method of Linux underlying Library:

To view the currently available versions:

strings /lib64/libc.so.6 |grep GLIBC_

Upgrade to 2.17 with rpm (download rpm if not):

$ wget http://copr-be.cloud.fedoraproject.org/results/mosquito/myrepo-el6/epel-6-x86_64/glibc-2.17-55.fc20/glibc-2.17-55.el6.x86_64.rpm
$ wget http://copr-be.cloud.fedoraproject.org/results/mosquito/myrepo-el6/epel-6-x86_64/glibc-2.17-55.fc20/glibc-common-2.17-55.el6.x86_64.rpm
$ wget http://copr-be.cloud.fedoraproject.org/results/mosquito/myrepo-el6/epel-6-x86_64/glibc-2.17-55.fc20/glibc-devel-2.17-55.el6.x86_64.rpm
$ wget http://copr-be.cloud.fedoraproject.org/results/mosquito/myrepo-el6/epel-6-x86_64/glibc-2.17-55.fc20/glibc-headers-2.17-55.el6.x86_64.rpm

Installation:

sudo rpm -Uvh glibc-2.17-55.el6.x86_64.rpm \
glibc-common-2.17-55.el6.x86_64.rpm \
glibc-devel-2.17-55.el6.x86_64.rpm \
glibc-headers-2.17-55.el6.x86_64.rpm

To solve the C/C + + underlying library CXXABI:

To view the currently available versions:

strings /usr/lib64/libstdc++.so.6 | grep 'CXXABI'

Enter the Lib folder under the python installation path. I found the latest version of libstdc++.so.6 in ~ / anaconda3/lib:

ls |grep libstdc++.so.6

I'm 6.0.26

libstdc++.so.6
libstdc++.so.6.0.26

Copy the latest version to the directory / usr/lib64 /, delete the original libstdc++.so.6, and name the latest version libstdc++.so.6, then connect:

# Copy to
cp libstdc++.so.6.0.26 /usr/lib64/
# Entry directory
cd /usr/lib64
# Delete original library
rf -rf /usr/lib64/libstdc++.so.6
# Rebuild Library
ln -s libstdc++.so.6.0.26 libstdc++.so.6

GPU view

#To view the usage of server GPU, NVIDIA's command:
nvidia-smi

#cuda available:
torch.cuda.is_available()

#Number of GPUs returned:
torch.cuda.device_count()

#Return gpu Name:
torch.cuda.get_device_name(0)

#Returns the current device index:
torch.cuda.current_device()

Screen common commands

Since you use remote connection to open Jupyter on the server, once you close the connected terminal, Jupyter will also close. At this time, you need to use the Screen tool to let the terminal run in the background, so that Jupyter can always log in.

#View the current screen:
screen -ls

#To create a screen is equivalent to opening a new terminal:
screen -S {screen name}

#Return the current screen background to the terminal before opening:
CTRL+A+D  #Shortcut keys

#Back to screen:
screen -r {screen name}

#Close the screen in the background:
screen -X -S {name you want to kill} quit
Published 1 original article, praised 0 and visited 3
Private letter follow

Tags: pip jupyter glibc RPM

Posted on Mon, 13 Jan 2020 06:21:31 -0500 by mamavi