Python learning record

1, LInux

Version: Ubuntu, CentOS, Redhat

Virtual machine software: VMware Workstation installation package – virtual machine - operating system Ubuntu

  • To install an Ubuntu system:

    Software tool rufus-3.15.exe:

    Ubuntu system:

    Installation steps (dual system):

    Dual system startup settings: sudo update grub

    To install pychar for Ubuntu:

    Details: 1. Unzip. 2.bin * * $. /**

    To create a pycharm shortcut:

  • Configure environment Rec:

    Graphics card: AMD Radeon R5 M230 video memory: 2G
    (base) pyl@PYL:~$ which conda
    (base) pyl@PYL:~$ which python
    (base) pyl@PYL:~$ ipython
    Python 3.7.4 (default, Aug 13 2019, 20:35:49)

    conda environmental management:

    Delete: conda remove --name tf2 python=3.7
    Viewing environment: conda info -e

    First create a virtual environment named torchCpu19:

    conda create -n torchCpu19 python=3.7

# To activate this environment, use

$ conda activate torchCpu19

To deactivate an active environment, use

$ conda deactivate

Pytorch installed on the official website = = 1.9.1
###### Pytorch installed on the official website = = 1.9.1

conda install pytorch torchvision torchaudio cpuonly -c pytorch 

ENV include:
tensorflow 1.15.0	#tf2
tf==2.1.0		#tf1
pytorch==1.9.1		#torchCpu

* tf1.15 In its compat.v2 Module contains 2.0 API Complete implementation of. It can be used enable_v2_behavior()Function simulation 2.0 behavior

* Ubuntu Next solution Pycharm Unable to input Chinese normally

* ### ==**#Three methods of use and their meanings**==

* #### # Note Content 

* #### Coding mode

  #_*_ coding:utf-8_*_    Indicates the specified file encoding format

* ####  Interpreter path

  #!/ Path of python interpreter: an interpreter used by the operating system to directly execute file selection


## 1. Directory structure

There is only one root directory /   No drive letter

/bin :Directory of executable binaries

/etc : Directory of system configuration files

/home :User home directory

### 1.1 directory instructions

**ls**  View the information in the current directory

pwd  View current directory

tree   View multi tier directory information		tree Folder name


#### 1.1.1 directory switching instruction

cd catalogue	   Switch to the specified directory

==cd ~==			Switch to the current user's**home directory**

cd ..			Switch to the previous directory

cd .			 Switch to**current directory**

==cd -==				Switch to the last directory

#### 1.1.2 absolute path and relative path

1. Absolute path switch to desktop(twice TaB Key display prompt: what is in the current directory)

 cd /home/blackey/Desktop

2. Relative path

 cd ../Desktop

#### 1.1.3 addition and deletion of documents and directories

touch file name			#increase        

* 1 Create with absolute path  touch  /.111.txt

* 2 Create multiple: touch  1.txt  2.txt  3.txt

mkdir Directory name				#Create folder - p create dependent folder

* mkdir  A/B  **-p**   # A and B can still be created when a does not exist

rm file name		#Delete file by default   

* Delete directory name:**rm Directory name -r**    #Delete non empty directory
* Before deletion**inquiry**: rm 1.txt **-i**
* Delete, ignore if file does not exist: rm 1.txt **-f**

rmdir Directory name				#Delete empty directory = = RM - D

#### 1.1.4 copying and moving files and directories

cp     copy			# cp  1.txt  AA    

* Copy directory and its contents: **cp  AA CC -r**
* Displays the copied path Description: cp  AA  BB  **-v**
* **Retain the original permissions of the file**Folders can be copied without loss:**-a**     (Mainly for cp to**Other users**)        

mv    Move, rename		# mv AA CC

* Display path:-v

#### 1.1.5 redirect command

The information displayed on the terminal is saved in the file 

">"   amount to w Operation: if ls  >  ./Info.txt

**">>"  Append mode write information: ls  >> ./Info.txt**

Append and write custom content: echo  "Custom content"  >>  file1

#### 1.1.6 viewing file content

cat    View (multiple) small files

* The Conduit  |    Used for temporary viewing without saving

as ls  /bin  |  cat			# The pipe is equivalent to a container for temporary storage of terminals

gedit    Open file

more  	Split screen view of large files

### 1.2 terminal command format

**command**  [-options] [parameter]

* scp Command requires first[-opt]Re parameter

* 1.Short options:-r

2.Long options:--help

View command help:

* --help  or man

ls Command options use:

-l    : Display as a list

-h   : Display in size units, default bytes

-a	: Show or hide files and directories

ls -alh   ==  ls  -a  -h  -l

**ll == ls -la**

e.g.   **d**rwxrwxr  3  blackey blackey 4.0k  10 month  AA

​          **-**rwxrwxr    **-** Represents a normal file     **d** Represents a directory

explain:**file type**User rights number of hard links user name user group file size modification time file name

### 1.3 soft link and hard link

* **Soft link (shortcut)  	ln -s   source file(Absolute path)  Shortcut name**

use:  ln -s    /home/blackey/desktop/ a.txt    ../a_s.txt

File type: l     .......    a_s.txt  ->  a.txt

* Hard link (alias of source file)      In  source file    alias

use:  ln     a.txt    a-h.txt    #Delete the source file, and it can still be accessed through the alias

### 1.4 text search & find

* **search(In text)**: 	**grep  'Search content'   File address  -n**

​								-i   Ignore case; -n  Display the matching line number;  -v Search content inversion 

* Combined with regular expression:      ^ : Starts with the specified string;   $ : Ends with the specified string;    .   : Matches a non newline character

​								grep   '^d'    Note.txt   -n        : with d Line search at the beginning

​								grep  's$'    Note.txt  -n

​                                       grep  'd.s'    Note.txt  -n        :use  .  Replace characters, such as find des

Query file name: e.g ls ./bin | grep 'file name'				#Quotation marks can be omitted 

* **lookup**: **find  ./   -name  '11.txt'**

* Wildcard:    *    Represents 0 or more arbitrary characters;     ?   Represents any 1 character

  ​			For example: find  .  -name    1?.txt             # Use wildcards, not quotes


### 1.5 compression & decompression

Supported formats:  .gz     .bz2  ;       .zip(Large compressed capacity)

use:    tar(Compression and decompression)  : ==tar -zcvf  test.tar.gz   *.txt==   (compress)      Decompression: tar -zxvf   test.tar.gz 

​		(Only decompression to the specified directory is supported: tar -zxvf  test.gar.gz  -C File1)

​			tar   -c   Create package file;  -x   Unpack

​					 -v Show files in package    -f   Specify the file name and put it last

​					-z    Compression or decompression;    -C       Extract to the specified directory

​		# Decompression of. Bz2 file: tar - = = J = = xvf test.bz2

​			zip     		zip  *.txt

​			unzip  -d		:  (Extract to the specified directory)

### 1.6 * * permission * * -- File & User

* Modify file permissions: chmod   u-r  1.txt

​											u :user    g:grop   o:othet  a:all user     

   + ​                                    + :Add permissions     -: Revoke permissions       = : Set permissions 

   + Permissions include:                 r, w, ==**x(Executable)**==, -  ((no permission)


* chmod   u+x            

* The interpreter needs to be declared before execution:

which Python    >>  /home/pyl/Anaconda3/bin/python3


* ==**Executive document:./**==

####  <u> * * Chmod 777 file.txt (numeric method, 777 is the highest permission, all read-write execution) * * < / u > 

On the contrary, it is 000: ---

r  :4    w: 2    x : 1   - :0       **rwx: 7**

### 1.7 administrator authority (whoami)

* sudo -s   Enter administrator       root@PYL:~/Desktop#

exit                                   pyl@PYL:~/Desktop$

passwd  Change Password

Shutdown and restart: shutdown -h now          reboot

* **1.==Create user group==**

sudo  groupadd    User group name    

To modify a user's primary group: sudo usermod  -g  User group name user group name 1

Delete user group:  sudo gropedel  User group name

* **2.**==**useradd   Create user**==

sudo useradd -m  -g  pyler     Create user and home directory      -g Set primary group

sudo passwd  pyler    Specify user set password

su  -  pyler    Switch users

* **Modify user information**: usermod       -G  Set up an additional group        -g   Modify user group
* **sudo usermod  -G  sudo pyler**      # The second sudo is the additional group name

* userdel     delete user

sudo userdel -r  pyler

### 1.8 SSH Remote Login and operation

apt list | grep openssh-server     # (server side) check whether the software is installed

client(cmd):  ssh  -Vss

Remote copy:    scp  1.txt       # Local remote copy to server

​						scp   ./			#Copy to local

[Copying folders requires -r ]:   scp -r  

### 1.9 Vim editor (command editing)

* **Command mode**(default)   →(i) Edit mode       esc                →(:) Last line mode     esc

* Edit mode( i )

* Last line mode   ( : )  

* preservation&sign out:    wq or x      Force exit without saving: q!

* Other instructions:

  yy    Copy cursor line                      dd   Delete the line where the cursor is located

  **u    revoke                     					ctrl+r   Anti revocation                  p   paste**

  G   Go back to the last line          				gg  Go back to the first line

​         Search: in last line mode:/Search content        next: n

​		Modification: :26,29s/Hello/Hi                #Limit the replacement of Hello from lines 26-29 with Hi

#### 1.10 software installation and uninstallation (Ubuntu)

* Offline installation:(package.deb)

sudo dpkg   -i   package.deb   

* Online installation: apt-get

​       sudo apt-get install  Installation package 

View all installation files:   **apt  list**

* Uninstall software

sudo dpkg  -r  Installation package name    (through the uninstallation of offline installation package software)

sudo apt-get remove Package name



# 2, Numpy and Pandas

## 1.1 Numpy

* N Dimension group-ndarray

ndarray.shape		Tuple of array dimension

ndarray.size			Number of elements

ndarray.itemsize	Length of an element(byte)

* Create array

a = np.array([ [1,2],[2,3] ] , [ [3,4],[5,6] ])		# 3D array, shape (height 2, row 2, column 3)

* Specify array type(Do not specify default int64,Decimal default float64)

as np.object_		:Python object

a = np.array( [ [1,2],[2,3] ] , [ [3,4],[5,6] ], **detype=np.float32**)

a = np.array(['tensor', 'torch', 'python'], **detype=np.string_**)

* Generate array:

1. 0,1 array

f0 = np.ones([4,8]);	np.zeros()

 np.one_like(f0)		#Generate by f0 dimension

2. f= np.array([ [1,2],[2,3] ] , [ [3,4],[5,6] ] );	# Deep copy

3. np.asarray(f)                                            #Shallow copy

4. Generate specified range:

   np.linspace(0,100,**10**)     # Generate 10 num numbers from 0-100 equal intervals

   np.logspace(0,100,**2**)     # Generated at 2 intervals of 0-100 	 step

* Generate random array

np.random(2,3)     # 2 rows and 3 columns random array

**uniform distribution--**

#Variance of uniform distribution: the larger the value, the more concentrated the data, and the thinner the height

​	np.random.rand(d0,d1,...dn)      #Returns a set of evenly distributed numbers in [0,1)

​	np.random.randint()

​	np.random.uniform(low=0, high=1,size=(3,5))      #Randomly sample from a uniform distribution [low, high] to generate an array of 3 rows and 5 columns

**Normal distribution**--api

* np.random.normal()

* Array index, slicing

[*, #]

* Shape modification:

1. object.reshape       # Generate new variable

2. object.resize           # Change to original value

3. object.T                  #  Row column interchange


* Type modification


* Array de duplication


#### 1.2 ndarray operation

1. Logical operation

 stock = np.random.normal(0, 1, (8, 10) )

 qiepian = stock[0:5, 0:5]       #Values for 5 rows and 5 columns

 qiepian[qiepian >1] =2         # The value greater than 1 is assigned as 2 $$if greater than or less than 1, it can be directly assigned if it meets the requirements

2. General judgment function

 np.all()							# Return True only when all requirements are met

 np.any()					#Returns True if any requirement is met


3. Ternary operator    

 np.where(val >0, 1 ,0)    # If Val > 0, it is assigned 1; otherwise, it is assigned 0

 np.where(np.logical_and(val >0.5, val<1),1,0)   # If > 0.5 and < 1 are satisfied at the same time, the value is 1
 np.where(np.logical_or())     # or
  1. statistical indicators

    val.max(axis=1) # find the maximum value by line

    val.argmax(axis=1) # find the subscript of the maximum value of the row

    midian() # median

    mean() std() # standard deviation

    var() # variance

1.3 matrix operation

arr = np.array([1,3,5,7,9])
arr += 1			# [2,4,6,8,10]
  • Array and array operation -- it needs to meet the broadcast mechanism
    • Dimension equality
    • shape, the corresponding position is 1


The dimensions of a (3) and B (3) are equal and can be calculated

​ A( , 2, 1)

B(8, 1, 3), the relative position dimension is 1, which can be calculated,

​ res(8, 2, 3)

arr1 = np.array([[1],[2]])          # shape(2,1)
arr2 = np.array([[1,2,3],[5,6,1]])  # shape(2,3)
print(arr1+arr2)        # Can operate, ans[[2,3,4],[7,8,3]], shape(2,3)
  • Matrix multiplication

    np.matmul(A, B), B) # difference: dot support matrix * number

2.1 Pandas

III. commodity object detection items

  • Transfer learning: the model task is similar, and the trained model in the old field is applied to the new model
    • Method: fine tuning
      • Trained model: pre trained model

Pre training model of TF: inception v1-4 inception RESNET V2 RESNET V1-2

* process

3.1 reading image data (imagedatagenerator)

import tensorflow as tf
from tensorflow import keras
from tensorflow.python.keras.preprocessing.image import ImageDataGenerator
  • train_gen = ImageDataGenerator()

    The batch tensor value of the production picture provides data enhancement

    • rescale = 1.0 / 255, # standardized
    • zca_whitening = False # PCA dimensionality reduction for images to reduce redundancy
    • width_shift_range # default horizontal translation
  • flow(x_train, y_train, batch_sz) # e.g. reading of minister data set

  • Local dataset read method:

    flow_from_directory(directory=path, target_size=(h, w), batch_size, class_mode='binary', shuffle=True)

    class_mode = 'binary' indicates that the format of the target value is 1D label

    Category means 2D one hot encoded labels

    The directory format of path is fixed: data/train/dogs / content; cats / content

3.1.1 notop model

  • Does it include the last three full connection layers dedicated to fine tune

      from tensorflow.python.keras.applications.vgg16 import VGG16
        # In__ init__ Add in
        # VGG pre training weight in imagenet competition, using resnet training
        self.base_model = VGG16(weights='imagenet',include_top=False)

    05 Section IV 04.21

++++++++++++++++++Unfinished to be continued+++++++++++++++++

4, Git and Github

Install git: sudo apt get install Git

Workspace – staging area – warehouse area -~- server (GitHub)

Add, delete and change in the workspace: git add file or directory; git rm file; git checkout – file

Workspace to staging area: git add

From staging area to warehouse area: git commit -m "remarks"

Work area directly to warehouse area: git commit -am "remarks"

4.1 Git single person local warehouse

  • Create warehouse

    git init

    ctrl +h show hidden folder.git

    $$create user information:

    git config PYL

    git config

    $$ touch

    git status; git add . ; git commit -m "login to warehouse completed"; git log

    pyl@PYL:/media/pyl/HDD_disk/Gitclub$ git status
     In branch master
       Not yet submitted
       Untracked files:
      (use &quot;git add &lt;file&gt;...&quot; (to include what to submit)
       Commit is empty, but there are files that have not been tracked (using &quot;git add&quot; Establish tracking)
    ## Track all files and submit to staging area
    pyl@PYL:/media/pyl/HDD_disk/Gitclub$ git add .   
    In branch master
     Not yet submitted
     Changes to submit:
      (use "git rm --cached <file>..." (to cancel staging)
    	New file:
    ## Submit to warehouse area
    pyl@PYL:/media/pyl/HDD_disk/Gitclub$ git commit -m  "Login warehouse completed"
    [master (Root submission) 79 d8e03] Login warehouse completed
     1 file changed, 0 insertions(+), 0 deletions(-)
     create mode 100644
    ## see information
    pyl@PYL:/media/pyl/HDD_disk/Gitclub$ git log
    commit 79d8e031b887170f40e4d70ff21c6c48ac67a0aa (HEAD -> master)
    Author: PYL <>
    Date:   Sun Oct 17 16:47:27 2021 +0800
        Login warehouse completed

    Fallback version: git reset --hard HEAD ^ # a ^ means fallback one version forward, or HEAD~4, fallback four versions

    View all versions: git reflog # is currently git log

    79d8e03 (HEAD -> master) HEAD@{0}: reset: moving to HEAD^
    ae34583 HEAD@{1}: commit: Optimization update
    79d8e03 (HEAD -> master) HEAD@{2}: commit (initial): Login warehouse completed

    Fallback to the specified version: git reset --hard ae34583

    Fallback from staging area to workspace: git checkout

    From server to local: git clone

Tags: Python Pycharm Ubuntu

Posted on Mon, 18 Oct 2021 03:11:16 -0400 by sloede