Squid cache agent

catalogue

1, Squid proxy server

1. Working mechanism of agency

2. Type of agent

3. Benefits of using agents

2, Compile and install Squid

1. Compile and install Squid

2. Modify squid profile

3. Squid operation control

4. Create Squid service script

3, Traditional agent

1. squid server

2. web1 configuration

3. Client configuration (add agent)

4. Test results

4, Transparent proxy

1. Squid server configuration

2. Squid modifies firewall and ip forwarding

3,web1   Server modification configuration

4. Access web1 server

5, ACL access control

1. Define access control lists

Method 1

Method 2

6, Squid log analysis

7, Reverse proxy

1. Working mechanism

summary

1, Squid proxy server

Squid mainly provides cache acceleration and application layer filtering control functions

1. Working mechanism of agency

Instead of the client requesting data from the website, you can hide the user's real IP address
The obtained Web page data (static Web elements) is saved in the cache and sent to the client to respond quickly the next time the same data is requested

2. Type of agent

Traditional proxy: it is applicable to the Internet. You need to specify the address and port of the proxy server on the client

Transparent proxy: the client does not need to specify the address and port of the proxy server, but redirects Web access to the proxy server through default routing and firewall policies

Reverse proxy: if the requested resource is cached in the Squid reverse proxy server, the requested resource is returned directly to the client; Otherwise, the reverse proxy server will request resources from the background WEB server, and then return the requested response to the client. At the same time, it will also cache (statically) the response locally for use by the next requester

3. Benefits of using agents

Improve Web access speed
Hide the real IP address of the client

2, Compile and install Squid

1. Compile and install Squid

systemctl stop firewalld
systemctl disable firewalld
setenforce 0

yum -y install gcc gcc-c++ make
cd /opt
tar zxvf squid-3.5.28.tar.gz 
cd /opt/squid-3.5.28
./configure --prefix=/usr/local/squid \     #Specify the installation directory path
--sysconfdir=/etc \                         #Specify profile path
--enable-arp-acl \                          #MAC address control to prevent clients from using IP spoofing
--enable-linux-netfilter \                  #Using kernel filtering
--enable-linux-tproxy \                     #Support transparent mode
--enable-async-io=100 \                     #Asynchronous IO to improve storage performance
--enable-err-language="Simplify_ Chinese" \ #Display language of error message
--enable-underscore \                       #Allow underscores in URL s
--disable-poll \                            #Turn off the default use poll mode
--enable-gnuregex                           #Using GNU regular expressions


./configure --prefix=/usr/local/squid \
--sysconfdir=/etc \
--enable-arp-acl \
--enable-linux-netfilter \
--enable-linux-tproxy \
--enable-async-io=100 \
--enable-err-language="Simplify_Chinese" \
--enable-underscore \
--disable-poll \
--enable-epoll \
--enable-gnurege

make && make install
ln -s /usr/local/squid/sbin/* /usr/local/sbin
useradd -M -s /sbin/nologin squid
chown -R squid:squid /usr/local/squid/var/       #This directory is used to store cache files

2. Modify squid profile

vim /etc/squid.conf
......
-----56 that 's ok--insert------
http_access allow all    #Put on HTTP_ Before access deny all, any client is allowed to use the proxy service to control the top-down matching of rules
http_access deny all
http_port 3128           #Used to specify the address and port that the proxy service listens to (the default port number is 3128)
-----61 that 's ok--insert------
cache_effective_user squid     #Add, specify the program user, which is used to set the account of initialization and runtime cache. Otherwise, the startup will not succeed
cache_effective_group squid    #Add, specify account basic group
coredump_dir /usr/local/squid/var/cache/squid   #Specify cache file directory

3. Squid operation control

#Check whether the configuration file syntax is correct
squid -k parse
squid –k rec                      	##Reload profile

#Start Squid. When Squid service is started for the first time, the cache directory will be initialized automatically
squid -zX   #-The z option is used to initialize the cache directory
squid      #Start squid service

netstat -anpt | grep 3128

4. Create Squid service script

vim /etc/init.d/squid
#!/bin/bash
#chkconfig: 2345 90 25
#2345 is the default self startup level. If yes - means that no self startup is available at any level; 90 is the start priority, 25 is the stop priority, and the priority range is 0-100. The higher the number, the lower the priority.

PID="/usr/local/squid/var/run/squid.pid"
CONF="/etc/squid.conf"
CMD="/usr/local/squid/sbin/squid"

case "$1" in
   start)
     netstat -natp | grep squid &> /dev/null
     if [ $? -eq 0 ]
     then
       echo "squid is running"
       else
       echo "Starting squid..."
       $CMD
     fi
   ;;
   stop)
     $CMD -k kill &> /dev/null
     rm -rf $PID &> /dev/null
   ;;
   status)
     [ -f $PID ] &> /dev/null
        if [ $? -eq 0 ]
          then
            netstat -natp | grep squid
          else
            echo "squid is not running"
        fi
   ;;
   restart)
      $0 stop &> /dev/null
      echo "Closing squid..."
      $0 start &> /dev/null
      echo "Starting squid..."
   ;;
   reload)
      $CMD -k reconfigure
   ;;
   check)
      $CMD -k parse
   ;;
   *)
      echo "Usage: $0{start|stop|status|reload|check|restart}"
   ;;
esac


chmod +x /etc/init.d/squid
chkconfig --add squid
chkconfig --level 35 squid on
​
vim /etc/init.d/squid
#!/bin/bash
#chkconfig: 2345 90 25
#2345 is the default self startup level. If yes - means that no self startup is available at any level; 90 is the start priority, 25 is the stop priority, and the priority range is 0-100. The higher the number, the lower the priority.

PID="/usr/local/squid/var/run/squid.pid"
CONF="/etc/squid.conf"
CMD="/usr/local/squid/sbin/squid"

case "$1" in
   start)
     netstat -natp | grep squid &> /dev/null
     if [ $? -eq 0 ]
     then
       echo "squid is running"
       else
       echo "Starting squid..."
       $CMD
     fi
   ;;
   stop)
     $CMD -k kill &> /dev/null
     rm -rf $PID &> /dev/null
   ;;
   status)
     [ -f $PID ] &> /dev/null
        if [ $? -eq 0 ]
          then
            netstat -natp | grep squid
          else
            echo "squid is not running"
        fi
   ;;
   restart)
      $0 stop &> /dev/null
      echo "Closing squid..."
      $0 start &> /dev/null
      echo "Starting squid..."
   ;;
   reload)
      $CMD -k reconfigure
   ;;
   check)
      $CMD -k parse
   ;;
   *)
      echo "Usage: $0{start|stop|status|reload|check|restart}"
   ;;
esac


chmod +x /etc/init.d/squid
chkconfig --add squid
chkconfig --level 35 squid on

​​
vim /etc/init.d/squid
#!/bin/bash
#chkconfig: 2345 90 25
#2345 is the default self startup level. If yes - means that no self startup is available at any level; 90 is the start priority, 25 is the stop priority, and the priority range is 0-100. The higher the number, the lower the priority.

PID="/usr/local/squid/var/run/squid.pid"
CONF="/etc/squid.conf"
CMD="/usr/local/squid/sbin/squid"

case "$1" in
   start)
     netstat -natp | grep squid &> /dev/null
     if [ $? -eq 0 ]
     then
       echo "squid is running"
       else
       echo "Starting squid..."
       $CMD
     fi
   ;;
   stop)
     $CMD -k kill &> /dev/null
     rm -rf $PID &> /dev/null
   ;;
   status)
     [ -f $PID ] &> /dev/null
        if [ $? -eq 0 ]
          then
            netstat -natp | grep squid
          else
            echo "squid is not running"
        fi
   ;;
   restart)
      $0 stop &> /dev/null
      echo "Closing squid..."
      $0 start &> /dev/null
      echo "Starting squid..."
   ;;
   reload)
      $CMD -k reconfigure
   ;;
   check)
      $CMD -k parse
   ;;
   *)
      echo "Usage: $0{start|stop|status|reload|check|restart}"
   ;;
esac


chmod +x /etc/init.d/squid
chkconfig --add squid
chkconfig squid on
​service squid restart   ##The test is normal

3, Traditional agent

1. squid server

vim /etc/squid.conf 
......
http_access allow all
http_access deny all
http_port 3128
cache_effective_user squid
cache_effective_group squid

#63 line insert
cache_mem 64 MB				
#Specify the memory space used by the cache function to maintain frequently accessed WEB objects. The capacity is preferably a multiple of 4, in MB. It is recommended to set it to 1 / 4 of the physical memory
reply_body_max_size 10 MB			
#The maximum file size that users are allowed to download, in bytes. When downloading a Web object of more than the specified size, a prompt of "request or access too large" will appear on the error page of the browser. The default setting is 0, which means no restriction
maximum_object_size 4096 KB			
#The maximum object size allowed to be saved to the cache space, in KB. Files exceeding the size limit will not be cached, but will be forwarded directly to the user

service squid restart
systemctl restart squid

#Modify firewall rules
iptables -F
iptables -I INPUT -p tcp --dport 3128 -j ACCEPT

2. web1 configuration

systemctl stop firewalld.service
setenforce 0
yum -y install httpd
systemctl start httpd
netstat -natp | grep 80

3. Client configuration (add agent)

4. Test results

#Dynamically view the access log on web1 and observe the visiting IP
tail -f /var/log/httpd/access_log

Client access 192.168.255.150

4, Transparent proxy

1. Squid server configuration

cd /etc/sysconfig/network-scripts/
cp ifcfg-ens33 ifcfg-ens37
vim ifcfg-ens37
systemctl restart network
yum install iptables* -y


#In line 60, modify and add the IP address that provides intranet services, and support the transparent proxy option transparent
vim /etc/squid.conf
......
http_access allow all
http_access deny all

http_port 192.168.174.1:3128 transparent
systemctl restart squid

2. Squid modifies firewall and ip forwarding

echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.conf
sysctl -p

iptables -F
iptables -t nat -F
iptables -t nat -I PREROUTING -i ens37 -s 192.168.174.0/24 -p tcp --dport 80 -j REDIRECT --to 3128	
iptables -t nat -I PREROUTING -i ens37 -s 192.168.174.0/24 -p tcp --dport 443 -j REDIRECT --to 3128	
iptables -I INPUT -p tcp --dport 3128 -j ACCEPT

3,web1   Server modification configuration

to configure web1 Gateway points to 192.168.255.180
iptables -F

4. Access web1 server

web1 Dynamically view the access log and observe the visit IP
tail -f /var/log/httpd/access_log

5, ACL access control

In the configuration file squid.conf, ACL access control is realized through the following two steps:
① Use acl configuration items to define the conditions that need to be controlled
② Via http_ The access configuration item controls "allow" or "deny" access to the defined list

1. Define access control lists

acl List name list type list content 
List name: custom name, equivalent to acl Give a name (a bit similar to shell Script variable name)
List type: required squid Predefined values, corresponding to different categories of control conditions
 List content: refers to the specific object to be controlled. The corresponding contents of different types of lists are also different. There can be multiple values (separated by spaces and the relationship of "or")

Method 1

vim /etc/squid.conf
.......
acl localhost src 192.168.174.142/24            #The source address is 192.168.174.142
acl MYLAN src 192.168.255.0/24    				#Client network segment
acl destinationhost dst 192. 168.255.150/24     #The destination address is 192.168.255.150 
acl MC20 maxconn 20                             #Maximum concurrent connections 20
acl PORT port 21                                #Target port 21
acl DMBLOCK dstdomain .qq.com                   #Target domain, matching all sites in the domain
acl BURL url_regex -i ^rtsp:// ^emule://        	# URL s starting with RTSP: / /. Module: / /, - i means case is ignored
acl PURL urlpath_regex -i \.mp3$ \.mp4$ \.rmvb$   #URL path ending in. mp3,. mp4,. rmvb
acl WORKTIME time MTWHF 08:30-17:30             #The time is 8:30 ~ 17:30 from Monday to Friday, "MTWHF" is the English initials of each week


http_access deny host
service squid reload

Method 2

#Start object list management
mkdir /etc/squid
vim /etc/squid/dest.list
192.168.255.180      #Squid server IP
192.168.255.0/24     #Any required network segment

vim /etc/squid.conf
......
acl destinationhost dst "/etc/squid/dest.list"    #Call the contents of the list in the specified file
http_access deny (or allow) destinationhost      #Note that if it is a rejection list, it needs to be placed in http_access allow all

systemctl restart squid

6, Squid log analysis

sarg (Squid Analysis Report Generator) is a squid log analysis tool, which uses HTML format to list in detail the site information, time occupation information, ranking, connection times, traffic, etc. of each user accessing the Internet
 

#Install image processing package
yum install -y gd gd-devel pcre-devel 

mkdir /usr/local/sarg

#Upload the zxvf sarg-2.3.7.tar.gz compressed package to the / opt directory
tar zxvf sarg-2.3.7.tar.gz -C /opt/

cd /opt/sarg-2.3.7
./configure --prefix=/usr/local/sarg \
--sysconfdir=/etc/sarg \       #The configuration file directory is / usr/loca/etc by default
--enable-extraprotection       #Additional safety protection

./configure --prefix=/usr/local/sarg --sysconfdir=/etc/sarg --enable-extraprotection

make && make install

vim /etc/sarg/sarg.conf
--7 that 's ok--note off
access_log /usr/local/squid/var/logs/access.log  #Specify access log file
--25 that 's ok--note off
title "Squid User Access Reports"     #Page title
--120 that 's ok--Uncomment, modify
output_dir /var/www/html/sarg      #Report output directory
--178 that 's ok--note off
user_ip no           #Display with user name
--184 that 's ok--Uncomment, modify
topuser_sort_field connect reverse     #In top sorting, the specified connection times are arranged in descending order, and the ascending order is normal
--190 that 's ok--Uncomment, modify
user_sort_field connect reverse      #For user access records, the number of connections is sorted in descending order
--206 that 's ok--Uncomment, modify
exclude_hosts /usr/local/sarg/noreport    #Specifies files that are not included in the sorted site list
--257 that 's ok--note off
overwrite_report no         #Overwrite logs with the same name and date
--289 that 's ok--Uncomment, modify
mail_utility mailq.postfix       #Send mail report command
--434 that 's ok--Uncomment, modify
charset UTF-8          #Specifies the character set UTF-8
--518 that 's ok--note off
weekdays 0-6          #Week cycle of top ranking
--525 that 's ok--note off
hours 0-23           #Time period of top ranking
--633 that 's ok--note off
www_document_root /var/www/html      #Specify page root

#Add is not included in the site file, and the added domain name will not be displayed in the sorting
touch /usr/local/sarg/noreport

ln -s /usr/local/sarg/bin/sarg /usr/local/bin/

sarg --help   #get help 

#function
sarg     #Start a record

#verification
yum install httpd -y
systemctl start httpd

iptables -I INPUT -p tcp --dport 80 -j ACCEPT 

stay squid Use browser access on the server http://192.168.255.180/sarg, check the Sarg report page.

#Add scheduled tasks to perform daily report generation
vim /usr/local/sarg/report.sh
#/bin/bash
#Get current date
TODAY=$(date +%d/%m/%Y)
#Get one week ago today
YESTERDAY=$(date -d "1 day ago" +%d/%m/%Y)
/usr/local/sarg/bin/sarg -l /usr/local/squid/var/logs/access.log -o /var/www/html/sarg -z -d $YESTERDAY-$TODAY &> /dev/null
exit 0

chmod +x /usr/local/sarg/report.sh 

crontab -e
0 0 * * * /usr/local/sarg/report.sh


Manual mode
#Direct access squid http://192.168.255.180/sarg Periodic scheduling task execution generates reports every day, and crontab-o outputs them to the report directory
sarg -l /usr/local/squid/var/logs/access.log -o /var/www/html/sarg -z -d $(date -d "1 day ago" +%d/%m/%Y)-$(date +%d/%m/%Y)
##Execute again to generate a new report, and the page can see the log report record of one more day
http://192.168.255.180/sarg/index.html

sarg -l /usr/local/squid/var/logs/access.log -o /var/www/html/sarg -z -d $(date -d "1 day ago" +%d/%m/%Y)-$(date +%d/%m/%Y)

7, Reverse proxy

If the requested resource is cached in the Squid reverse proxy server, the requested resource is directly returned to the client; Otherwise, the reverse proxy server will request resources from the background Web server, and then return the requested response to the client. At the same time, it will also cache the response locally for use by the next requester

1. Working mechanism

Cache web page objects to reduce duplicate requests
Internet requests are polled or weighted to intranet web servers
Proxy user requests to avoid users directly accessing the Web server and improve security

Turn on firewall and turn off local HTTPD
systemctl start firewalld
systemctl stop httpd

iptables -F
iptables -t nat -F

iptables -I INPUT -p tcp --dport 80 -j ACCEPT 

vim /etc/squid.conf
------ 60 that 's ok--Modify, insert-------
http_port 192.168.255.180:80 accel vhost vport
cache_peer 192.168.255.150 parent 80 0 no-query originserver round-robin max_conn=30 weight=1 name=web1
cache_peer 192.168.255.200 parent 80 0 no-query originserver round-robin max_conn=30 weight=1 name=web2
cache_peer_domain web1 web2 www.ly.com
#Indicates a request for www.ly.com. squid sends a request to port 80 of 192.168.255.150 and 192.168.255.200

-------------------------------------------------------------------------------------
http_port 80 accel vhost vport
#Squid has changed from a cache to a Web server reverse proxy acceleration mode. At this time, squid listens to requests on port 80 and binds to the request port (vhost vport) of webserver. When a request arrives at squid, squid does not need to forward the request. Instead, it directly needs to take data from the cache or directly request data from the bound port.
accel :Reverse proxy acceleration mode
vhost:Support domain name or host name to represent proxy node
vport :support IP And port to represent the proxy node

parent :Represents the parent node, the parent node, the parent node, the parent node, the parent node, and the parent node
80:Agent internal web Port 80 of the server
0 :Not used icp,It means just one squid The server
no-query :Get data directly without query
originserver :Specify source server
round-robin :appoint squid The request is distributed to one of the parent nodes by polling
max_conn :Specify the maximum number of connections
weight :Specify weights
name :Set alias
----------------------------------------------------------------------------------------

systemctl stop httpd
service squid reload

#Backend web2 node server settings
yum install -y httpd
systemctl start httpd

#Node 1(web1):
echo "this is test01" >> /var/www/html/index.html
#Node 2(web2):
echo "this is test02" >> /var/www/html/index.html

#Domain name mapping configuration for client
/etc/hosts file
192.168.255.180 www.ly.com

#Proxy configuration for client
 Open browser, tools-->Internet option-->connect-->LAN settings-->Turn on the proxy server(address: Squid The server IP Address, port: 80)

Browser access http://www.ly.com

summary

squid   Positioning is cache acceleration. The cache is obtained from the back-end web server. The acceleration is for client access

squid has three modes:
1. Traditional proxy: the client needs to point to the squid proxy server, and the client can perceive the existence of the squid proxy server
2. Transparent proxy (commonly used): the client does not need to be configured, but can be accessed directly. The server completes the transparent proxy with the help of firewall rules and static routing
3. Reverse proxy: as a reverse proxy function similar to the Nginx server, it does not need a home page. It completes the reverse proxy based on IP: port and weight

For Squid's own management / functions
ACL: mainly used for http_ Permission and denial management of access (access based on http protocol)
sarg: log analysis function. Access can be specified in days_ The contents of the log are output to a web page (with the help of httpd) for display.
 

Tags: Operation & Maintenance server Cache

Posted on Sat, 06 Nov 2021 16:34:33 -0400 by Aptana