1, LInux
Version: Ubuntu, CentOS, Redhat
Virtual machine software: VMware Workstation installation package – virtual machine - operating system Ubuntu
-
To install an Ubuntu system:
Software tool rufus-3.15.exe: http://rufus.ie/zh/
Ubuntu system: https://ubuntu.com/
Installation steps (dual system): https://www.bilibili.com/video/BV11k4y1k7Li
Dual system startup settings: sudo update grub
https://blog.csdn.net/jyqxerxes/article/details/80355343
To install pychar for Ubuntu: https://www.bilibili.com/video/BV16N411R7GC
Details: 1. Unzip. 2.bin * * $. / pycharm.sh**
To create a pycharm shortcut: https://blog.csdn.net/qq_20515461/article/details/90745100
-
Configure environment Rec:
Graphics card: AMD Radeon R5 M230 video memory: 2G
(base) pyl@PYL:~$ which conda
/home/pyl/Anaconda3/bin/conda
(base) pyl@PYL:~$ which python
/home/pyl/Anaconda3/bin/python
(base) pyl@PYL:~$ ipython
Python 3.7.4 (default, Aug 13 2019, 20:35:49)conda environmental management:
Delete: conda remove --name tf2 python=3.7
Viewing environment: conda info -eFirst create a virtual environment named torchCpu19:
conda create -n torchCpu19 python=3.7
# To activate this environment, use
$ conda activate torchCpu19
To deactivate an active environment, use
$ conda deactivate
Pytorch installed on the official website = = 1.9.1
###### Pytorch installed on the official website = = 1.9.1
conda install pytorch torchvision torchaudio cpuonly -c pytorch
ENV include:
tensorflow 1.15.0 #tf2
tf==2.1.0 #tf1
pytorch==1.9.1 #torchCpu
* tf1.15 In its compat.v2 Module contains 2.0 API Complete implementation of. It can be used enable_v2_behavior()Function simulation 2.0 behavior
* Ubuntu Next solution Pycharm Unable to input Chinese normally
https://blog.csdn.net/m0_54032194/article/details/119081008
* ### ==**#Three methods of use and their meanings**==
* #### # Note Content
* #### Coding mode
```python
#_*_ coding:utf-8_*_ Indicates the specified file encoding format
```
* #### Interpreter path
```python
#!/home/pyl/Anaconda3/bin/python3
#!/ Path of python interpreter: an interpreter used by the operating system to directly execute file selection
```
## 1. Directory structure
There is only one root directory / No drive letter
/bin :Directory of executable binaries
/etc : Directory of system configuration files
/home :User home directory
### 1.1 directory instructions
**ls** View the information in the current directory
pwd View current directory
tree View multi tier directory information tree Folder name
clear
#### 1.1.1 directory switching instruction
cd catalogue Switch to the specified directory
==cd ~== Switch to the current user's**home directory**
cd .. Switch to the previous directory
cd . Switch to**current directory**
==cd -== Switch to the last directory
#### 1.1.2 absolute path and relative path
1. Absolute path switch to desktop(twice TaB Key display prompt: what is in the current directory)
cd /home/blackey/Desktop
2. Relative path
cd ../Desktop
#### 1.1.3 addition and deletion of documents and directories
touch file name #increase
* 1 Create with absolute path touch /.111.txt
* 2 Create multiple: touch 1.txt 2.txt 3.txt
mkdir Directory name #Create folder - p create dependent folder
* mkdir A/B **-p** # A and B can still be created when a does not exist
rm file name #Delete file by default
* Delete directory name:**rm Directory name -r** #Delete non empty directory
* Before deletion**inquiry**: rm 1.txt **-i**
* Delete, ignore if file does not exist: rm 1.txt **-f**
rmdir Directory name #Delete empty directory = = RM - D
#### 1.1.4 copying and moving files and directories
cp copy # cp 1.txt AA
* Copy directory and its contents: **cp AA CC -r**
* Displays the copied path Description: cp AA BB **-v**
* **Retain the original permissions of the file**Folders can be copied without loss:**-a** (Mainly for cp to**Other users**)
mv Move, rename # mv AA CC
* Display path:-v
#### 1.1.5 redirect command
The information displayed on the terminal is saved in the file
">" amount to w Operation: if ls > ./Info.txt
**">>" Append mode write information: ls >> ./Info.txt**
Append and write custom content: echo "Custom content" >> file1
#### 1.1.6 viewing file content
cat View (multiple) small files
* The Conduit | Used for temporary viewing without saving
as ls /bin | cat # The pipe is equivalent to a container for temporary storage of terminals
gedit Open file
more Split screen view of large files
### 1.2 terminal command format
**command** [-options] [parameter]
* scp Command requires first[-opt]Re parameter
* 1.Short options:-r
2.Long options:--help
View command help:
* --help or man
ls Command options use:
-l : Display as a list
-h : Display in size units, default bytes
-a : Show or hide files and directories
ls -alh == ls -a -h -l
**ll == ls -la**
e.g. **d**rwxrwxr 3 blackey blackey 4.0k 10 month AA
**-**rwxrwxr **-** Represents a normal file **d** Represents a directory
explain:**file type**User rights number of hard links user name user group file size modification time file name
### 1.3 soft link and hard link
* **Soft link (shortcut) ln -s source file(Absolute path) Shortcut name**
use: ln -s /home/blackey/desktop/ a.txt ../a_s.txt
File type: l ....... a_s.txt -> a.txt
* Hard link (alias of source file) In source file alias
use: ln a.txt a-h.txt #Delete the source file, and it can still be accessed through the alias
### 1.4 text search & find
* **search(In text)**: **grep 'Search content' File address -n**
-i Ignore case; -n Display the matching line number; -v Search content inversion
* Combined with regular expression: ^ : Starts with the specified string; $ : Ends with the specified string; . : Matches a non newline character
grep '^d' Note.txt -n : with d Line search at the beginning
grep 's$' Note.txt -n
grep 'd.s' Note.txt -n :use . Replace characters, such as find des
Query file name: e.g ls ./bin | grep 'file name' #Quotation marks can be omitted
* **lookup**: **find ./ -name '11.txt'**
* Wildcard: * Represents 0 or more arbitrary characters; ？ Represents any 1 character
For example: find . -name 1?.txt # Use wildcards, not quotes
### 1.5 compression & decompression
Supported formats: .gz .bz2 ; .zip(Large compressed capacity)
use: tar(Compression and decompression) : ==tar -zcvf test.tar.gz *.txt== (compress) Decompression: tar -zxvf test.tar.gz
(Only decompression to the specified directory is supported: tar -zxvf test.gar.gz -C File1)
tar -c Create package file; -x Unpack
-v Show files in package -f Specify the file name and put it last
-z Compression or decompression; -C Extract to the specified directory
# Decompression of. Bz2 file: tar - = = J = = xvf test.bz2
zip zip testz.zip *.txt
unzip -d : (Extract to the specified directory)
### 1.6 * * permission * * -- File & User
* Modify file permissions: chmod u-r 1.txt
u :user g:grop o:othet a:all user
+ + :Add permissions -: Revoke permissions = : Set permissions
+ Permissions include: r, w, ==**x(Executable)**==， - ((no permission)
* chmod u+x fx.py
* The interpreter needs to be declared before execution:
which Python >> /home/pyl/Anaconda3/bin/python3
==**#!/home/pyl/Anaconda3/bin/python3**==
* ==**Executive document:./fx.py**==
#### <u> * * Chmod 777 file.txt (numeric method, 777 is the highest permission, all read-write execution) * * < / u >
On the contrary, it is 000: ---
r :4 w: 2 x : 1 - :0 **rwx: 7**
### 1.7 administrator authority (whoami)
* sudo -s Enter administrator root@PYL:~/Desktop#
exit pyl@PYL:~/Desktop$
passwd Change Password
Shutdown and restart: shutdown -h now reboot
* **1.==Create user group==**
sudo groupadd User group name
To modify a user's primary group: sudo usermod -g User group name user group name 1
Delete user group: sudo gropedel User group name
* **2.**==**useradd Create user**==
sudo useradd -m -g pyler Create user and home directory -g Set primary group
sudo passwd pyler Specify user set password
su - pyler Switch users
* **Modify user information**: usermod -G Set up an additional group -g Modify user group
* **sudo usermod -G sudo pyler** # The second sudo is the additional group name
* userdel delete user
sudo userdel -r pyler
### 1.8 SSH Remote Login and operation
apt list | grep openssh-server # (server side) check whether the software is installed
client(cmd): ssh -Vss
Remote copy: scp 1.txt pyl@111.xxx.xxx.xxx:/home/pyl/Desktop # Local remote copy to server
scp pyl@111.xxx.xxx.xxx:/home/pyl/Desktop/1.txt ./ #Copy to local
[Copying folders requires -r ]: scp -r
### 1.9 Vim editor (command editing)
* **Command mode**(default) →(i) Edit mode esc →(:) Last line mode esc
* Edit mode( i )
* Last line mode ( : )
* preservation&sign out: wq or x Force exit without saving: q!
* Other instructions:
yy Copy cursor line dd Delete the line where the cursor is located
**u revoke ctrl+r Anti revocation p paste**
G Go back to the last line gg Go back to the first line
Search: in last line mode:/Search content next: n
Modification: :26，29s/Hello/Hi #Limit the replacement of Hello from lines 26-29 with Hi
#### 1.10 software installation and uninstallation (Ubuntu)
* Offline installation:(package.deb)
sudo dpkg -i package.deb
* Online installation: apt-get
sudo apt-get install Installation package
View all installation files: **apt list**
* Uninstall software
sudo dpkg -r Installation package name (through the uninstallation of offline installation package software)
sudo apt-get remove Package name
------------------
--------------------
# 2, Numpy and Pandas
## 1.1 Numpy
* N Dimension group-ndarray
ndarray.shape Tuple of array dimension
ndarray.size Number of elements
ndarray.itemsize Length of an element(byte)
* Create array
a = np.array([ [1,2],[2,3] ] , [ [3,4],[5,6] ]) # 3D array, shape (height 2, row 2, column 3)
* Specify array type(Do not specify default int64，Decimal default float64)
as np.object_ :Python object
a = np.array( [ [1,2],[2,3] ] , [ [3,4],[5,6] ], **detype=np.float32**)
a = np.array(['tensor', 'torch', 'python'], **detype=np.string_**)
* Generate array:
1. 0,1 array
f0 = np.ones([4,8]); np.zeros()
np.one_like(f0) #Generate by f0 dimension
2. f= np.array([ [1,2],[2,3] ] , [ [3,4],[5,6] ] ); # Deep copy
3. np.asarray(f) #Shallow copy
4. Generate specified range:
np.linspace(0,100,**10**) # Generate 10 num numbers from 0-100 equal intervals
np.logspace(0,100,**2**) # Generated at 2 intervals of 0-100 step
* Generate random array
np.random(2,3) # 2 rows and 3 columns random array
**uniform distribution--**
#Variance of uniform distribution: the larger the value, the more concentrated the data, and the thinner the height
np.random.rand(d0,d1,...dn) #Returns a set of evenly distributed numbers in [0,1)
np.random.randint()
np.random.uniform(low=0, high=1,size=(3,5)) #Randomly sample from a uniform distribution [low, high] to generate an array of 3 rows and 5 columns
**Normal distribution**--api
* np.random.normal()
* Array index, slicing
[*, #]
* Shape modification:
1. object.reshape # Generate new variable
2. object.resize # Change to original value
3. object.T # Row column interchange
* Type modification
object.astype()
* Array de duplication
np.unique()
#### 1.2 ndarray operation
1. Logical operation
stock = np.random.normal(0, 1, (8, 10) )
qiepian = stock[0:5, 0:5] #Values for 5 rows and 5 columns
qiepian[qiepian >1] =2 # The value greater than 1 is assigned as 2 $$if greater than or less than 1, it can be directly assigned if it meets the requirements
2. General judgment function
np.all() # Return True only when all requirements are met
np.any() #Returns True if any requirement is met
3. Ternary operator
np.where(val >0, 1 ,0) # If Val > 0, it is assigned 1; otherwise, it is assigned 0
```Python
np.where(np.logical_and(val >0.5, val<1),1,0) # If > 0.5 and < 1 are satisfied at the same time, the value is 1
np.where(np.logical_or()) # or
-
statistical indicators
val.max(axis=1) # find the maximum value by line
val.argmax(axis=1) # find the subscript of the maximum value of the row
midian() # median
mean() std() # standard deviation
var() # variance
1.3 matrix operation
arr = np.array([1,3,5,7,9]) arr += 1 # [2,4,6,8,10]
- Array and array operation -- it needs to meet the broadcast mechanism
- Dimension equality
- shape, the corresponding position is 1
If:
The dimensions of a (3) and B (3) are equal and can be calculated
A( , 2, 1)
B(8, 1, 3), the relative position dimension is 1, which can be calculated,
res(8, 2, 3)
arr1 = np.array([[1],[2]]) # shape(2,1) arr2 = np.array([[1,2,3],[5,6,1]]) # shape(2,3) print(arr1+arr2) # Can operate, ans[[2,3,4],[7,8,3]], shape(2,3)
-
Matrix multiplication
np.matmul(A, B)
np.dot(A, B) # difference: dot support matrix * number
2.1 Pandas
III. commodity object detection items
- Transfer learning: the model task is similar, and the trained model in the old field is applied to the new model
- Method: fine tuning
- Trained model: pre trained model
- Method: fine tuning
Pre training model of TF: inception v1-4 inception RESNET V2 RESNET V1-2
* process
3.1 reading image data (imagedatagenerator)
import tensorflow as tf from tensorflow import keras from tensorflow.python.keras.preprocessing.image import ImageDataGenerator
-
train_gen = ImageDataGenerator()
The batch tensor value of the production picture provides data enhancement
- rescale = 1.0 / 255, # standardized
- zca_whitening = False # PCA dimensionality reduction for images to reduce redundancy
- width_shift_range # default horizontal translation
-
flow(x_train, y_train, batch_sz) # e.g. reading of minister data set
-
Local dataset read method:
flow_from_directory(directory=path, target_size=(h, w), batch_size, class_mode='binary', shuffle=True)
class_mode = 'binary' indicates that the format of the target value is 1D label
Category means 2D one hot encoded labels
The directory format of path is fixed: data/train/dogs / content; cats / content
3.1.1 notop model
-
Does it include the last three full connection layers dedicated to fine tune
from tensorflow.python.keras.applications.vgg16 import VGG16 # In__ init__ Add in # VGG pre training weight in imagenet competition, using resnet training self.base_model = VGG16(weights='imagenet',include_top=False)05 Section IV 04.21
++++++++++++++++++Unfinished to be continued+++++++++++++++++
4, Git and Github
Install git: sudo apt get install Git
Workspace – staging area – warehouse area -~- server (GitHub)
Add, delete and change in the workspace: git add file or directory; git rm file; git checkout – file
Workspace to staging area: git add login.py
From staging area to warehouse area: git commit -m "remarks"
Work area directly to warehouse area: git commit -am "remarks"
4.1 Git single person local warehouse
-
Create warehouse
git init
ctrl +h show hidden folder.git
$$create user information:
git config user.name PYL
git config user.email zaixiadouing@qq.com
$$ touch login.py
git status; git add . ; git commit -m "login to warehouse completed"; git log
pyl@PYL:/media/pyl/HDD_disk/Gitclub$ git status In branch master Not yet submitted Untracked files: (use "git add <file>..." (to include what to submit) login.py Commit is empty, but there are files that have not been tracked (using "git add" Establish tracking) ## Track all files and submit to staging area pyl@PYL:/media/pyl/HDD_disk/Gitclub$ git add . In branch master Not yet submitted Changes to submit: (use "git rm --cached <file>..." (to cancel staging) New file: login.py ## Submit to warehouse area pyl@PYL:/media/pyl/HDD_disk/Gitclub$ git commit -m "Login warehouse completed" [master (Root submission) 79 d8e03] Login warehouse completed 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 login.py ## see information pyl@PYL:/media/pyl/HDD_disk/Gitclub$ git log commit 79d8e031b887170f40e4d70ff21c6c48ac67a0aa (HEAD -> master) Author: PYL <zaixiadouing@qq.com> Date: Sun Oct 17 16:47:27 2021 +0800 Login warehouse completedFallback version: git reset --hard HEAD ^ # a ^ means fallback one version forward, or HEAD~4, fallback four versions
View all versions: git reflog # is currently git log
79d8e03 (HEAD -> master) HEAD@{0}: reset: moving to HEAD^ ae34583 HEAD@{1}: commit: Optimization update 79d8e03 (HEAD -> master) HEAD@{2}: commit (initial): Login warehouse completedFallback to the specified version: git reset --hard ae34583
Fallback from staging area to workspace: git checkout login.py
From server to local: git clone https://github.com/PYLuck/upy3.git