- Judgment method for whether a page is dynamically loaded (Ajax) data ⭐⭐
Dynamically loading data means that you can't directly get the page data by directly requesting the web address. We can locate the network through the packet capture tool on the web page to request the web page, and check whether a data on the web page is in the data page loaded by the network request page
- Detailed steps
How does the dynamically loaded data come from?
What is the relationship between selenium module and crawler?
- Convenient access to dynamically loaded data in the website
- Implementation of boundary simulation login
What is selenium module?
- A module based on browser automation
selenium module usage process:
- Installation environment: pip install selenium ⭐
- Download a browser driver: download the browser driver for the browser you use: Google driver
For the google browser currently in use, you can find the corresponding driver according to the version number. - Instantiate a browser object
- Write operation code based on browser automation:
- Initiate request: get(url)
- Label positioning: find series method
- Tag interaction: send_keys(‘xxx’)
- Execute js program: execte_script(‘jsCode’)
- Close browser: quit()
- Use selenium to crawl the data of the State Food and drug administration ⭐
from selenium import webdriver from lxml import etree from time import sleep # Instantiate a browser object (the driver passed in to the browser) bro = webdriver.Chrome(executable_path='E:\Google\chromedriver') # Let the browser initiate a request corresponding to the specified url bro.get('http://scxk.nmpa.gov.cn:81/xk/') # Get the source code data of the current page of the browser page_text = bro.page_source # Resolve enterprise name tree = etree.HTML(page_text) li_list = tree.xpath('//ul[@id="gzlist"]/li') for li in li_list: name = li.xpath('./dl/@title')[0] print(name) sleep(5) # Close browser bro.quit()
- Simple automation using selenium
from selenium import webdriver from time import sleep bro = webdriver.Chrome(executable_path=r'E:\Google\chromedriver') bro.get('https://www.taobao.com/') # Find search box: label positioning search_input = bro.find_element_by_id('q') # Returns the positioned label # Tag interaction search_input.send_keys('Iphone') # Execute a set of js programs: scroll the height of a screen bro.execute_script('window.scrollTo(0,document.body.scrollHeight)') sleep(2) # Click the search button class = 'BTN search TB BG' btn = bro.find_element_by_css_selector('.btn-search') # btn = bro.find_element_by_class_name('btn-search') btn = bro.find_element_by_class_name('tb-bg') btn.click() bro.get('https://www.baidu.com') sleep(2) # Back off bro.back() sleep(2) # forward bro.forward() sleep(5) bro.quit()2, selenium handles iframe
- If the positioned tag exists in the iframe tag, you must use switch_to.frame(id)
- Action chain (drag): from selenium.webdriver import ActionChains # action chain
-
- Instantiate action chain object: action = ActionChains(bro)
-
- action.click_and_hold(div): long press and click
-
- move_by_offset(x,y): the pixel value you want to offset
-
- action.move_by_offset(17,yoffset=0).perform(): make the action chain execute immediately
-
- action.release() releases the action chain object
- iframe
Iframe is our commonly used iframe tag: < iframe >. Iframe tag is a form of framework, which is also commonly used. Iframe is generally used to include other pages. For example, we can load the contents of other websites or other pages of our site on our own website page.
The biggest function of iframe tag is to make the page beautiful.
There are many uses of iframe tags. The main difference lies in the different forms of defining iframe tags, such as defining the length, width and height of iframe.
- Use selenium to achieve the following operations
- View and output dragged labels
from selenium import webdriver from time import sleep bro = webdriver.Chrome(executable_path=r'E:\Google\chromedriver') url = 'http://www.runoob.com/try/try.php?filename=jqueryui-api-droppable' bro.get(url) div = bro.find_element_by_id('draggable') print(div)
If the located tag exists in the iframe tag, it must be located by the following operations
# Fill in the id and name of the included iframme bro.switch_to.frame('iframeResult') # Toggles the scope of browser labels
Run again without error, and print the results:
- Complete action
from selenium import webdriver from time import sleep from selenium.webdriver import ActionChains # Action chain bro = webdriver.Chrome(executable_path=r'E:\Google\chromedriver') url = 'http://www.runoob.com/try/try.php?filename=jqueryui-api-droppable' bro.get(url) # If the located tag exists in the iframe tag, it must be located by the following operations bro.switch_to.frame('iframeResult') # Toggles the scope of browser labels div = bro.find_element_by_id('draggable') # print(div) # Action chain: trigger a series of operations action = ActionChains(bro) # Click and hold action.click_and_hold(div) for i in range(5): # move_by_offset(x,y) action.move_by_offset(17,yoffset=0).perform() # perform() executes the action chain immediately sleep(0.3) # Action chain release action.release() sleep(5) bro.quit()3, Simulated Login of selenium
- demand
Simulate login: https://qzone.qq.com/
- Simulate open URL
from selenium import webdriver from time import sleep bro = webdriver.Chrome(executable_path=r'E:\Google\chromedriver') bro.get('https://qzone.qq.com/')
- Jump iframe
from selenium import webdriver from time import sleep bro = webdriver.Chrome(executable_path=r'E:\Google\chromedriver') bro.get('https://qzone.qq.com/') bro.switch_to.frame('login_frame') a_tag = bro.find_element_by_id('switcher_plogin') a_tag.click() # Find the text for entering the account and password userName_tag = bro.find_element_by_id('u') password_tag = bro.find_element_by_id('p') sleep(1) userName_tag.send_keys('2328409226') sleep(1) password_tag.send_keys('123456789') sleep(1) # Login button btn = bro.find_element_by_id('login_button') btn.click() sleep(3) bro.quit()
4.1 no visual interface
How to make our Google browser do not carry out visual operation, that is, there is no visual interface (headless browser)
- Add code ⭐
from selenium.webdriver.chrome.options import Options # Create a parameter object to control chrome to open in no interface mode chrome_options = Options() chrome_options.add_argument('--headless') chrome_options.add_argument('--disable-gpu') bro = webdriver.Chrome(executable_path=r'E:\Google\chromedriver',chrome_options=chrome_options)
- verification
from selenium import webdriver from time import sleep from selenium.webdriver.chrome.options import Options # Create a parameter object to control chrome to open in no interface mode chrome_options = Options() chrome_options.add_argument('--headless') chrome_options.add_argument('--disable-gpu') bro = webdriver.Chrome(executable_path=r'E:\Google\chromedriver',chrome_options=chrome_options) # No visual interface (headless browser) # phantomJs is also a headless browser, but it has stopped updating and maintaining bro.get('https://www.baidu.com/') print(bro.page_source) bro.quit()
4.2 avoidance detection
- Add code ⭐
from selenium.webdriver import ChromeOptions # Implementation of circumvention detection option = ChromeOptions() option.add_experimental_option('excludeSwitches', ['enable-automation'])
- verification
from selenium import webdriver from time import sleep # Realize no visual interface from selenium.webdriver.chrome.options import Options # Implementation of circumvention detection from selenium.webdriver import ChromeOptions # Create a parameter object to control chrome to open in no interface mode chrome_options = Options() chrome_options.add_argument('--headless') chrome_options.add_argument('--disable-gpu') # Implementation of circumvention detection option = ChromeOptions() option.add_experimental_option('excludeSwitches', ['enable-automation']) # How to make selenium avoid the risk of being detected bro = webdriver.Chrome(executable_path=r'E:\Google\chromedriver',chrome_options=chrome_options,options=option) # No visual interface (headless browser) # phantomJs is also a headless browser, but it has stopped updating and maintaining bro.get('https://www.baidu.com/') print(bro.page_source) bro.quit()5, One two three zero six simulated Login
- Coding process
- Find account login
- Find the label corresponding to the account and password
- Find the label of the slider
- Sliding with action chain
Thank you for the solutions provided by the following Blogs:
5.1 step explanation
- No sliding
Since there is no time pause after clicking the button, set the time interval
- Slider sliding failed
bro.get('https://kyfw.12306.cn/otn/resources/login.html') #Prevent login identified as selenium by 12306 ⭐⭐ script = 'Object.defineProperty(navigator,"webdriver",);' bro.execute_script(script)
- Slider movement position
move_by_offset(xoffset, yoffset) -- move the mouse from the current position to a certain coordinate
# Action chain: trigger a series of operations action = ActionChains(bro) # Click and hold action.click_and_hold(div) for i in range(5): # move_by_offset(x,y) action.move_by_offset(17,yoffset=0).perform() # One time offset 17 pixels sleep(0.3) # Action chain release action.release()
move_to_element(to_element) -- move the mouse to an element
move_to_element_with_offset(to_element, xoffset, yoffset) -- how far is it from an element (upper left coordinate)
drag_and_drop_by_offset(source, xoffset, yoffset) -- drag to a coordinate and release
#Sliding verification code span = bro.find_element_by_xpath('//*XPath of [@ id="nc_1_n1z"]) # slider: / / * [@ id="nc_1_n1z"] bigspan = bro.find_element_by_class_name('nc-lang-cnt') print('big_span:',bigspan.size)# Length and width corresponding to the label # To div_tag for sliding operation action = ActionChains(bro) # Click and hold the specified label action.click_and_hold(span).perform() action.drag_and_drop_by_offset(span, 350, 0).perform() # Drag to a coordinate and release # action.drag_and_drop_by_offset(span, 400, 0) # Action chain release action.release()
Coordinate system of computer display screen: X increases from left to right and Y increases from top to bottom. That is, the coordinates of the top left visible pixel are (0, 0)
Questions about dragging coordinate points:
As for whether it is (350, 0) or (400, 0) or other coordinates (x,y), just drag the slider to the bottom
perform() -- execute the action chain immediately
5.2 source code
from selenium import webdriver from time import sleep from selenium.webdriver import ActionChains # Action chain from selenium.webdriver import ChromeOptions # Implementation of circumvention detection option = ChromeOptions() option.add_experimental_option('excludeSwitches', ['enable-automation']) bro = webdriver.Chrome(executable_path=r'E:\Google\chromedriver') bro.get('https://kyfw.12306.cn/otn/resources/login.html') #Prevent login identified as selenium by 12306 script = 'Object.defineProperty(navigator,"webdriver",);' bro.execute_script(script) # Find the account login button tag = bro.find_element_by_xpath('//*[@id="toolbar_Div"]/div[2]/div[2]/ul/li[2]/a') tag.click() sleep(1) # Enter account password userName_tag = bro.find_element_by_id('J-userName') password_tag = bro.find_element_by_id('J-password') sleep(1) userName_tag.send_keys('17349868689') sleep(1) password_tag.send_keys('ZHENHAO0903') sleep(1) # Login button btn = bro.find_element_by_id('J-login') btn.click() sleep(3) #Sliding verification code span = bro.find_element_by_xpath('//*XPath of [@ id="nc_1_n1z"]) # slider: / / * [@ id="nc_1_n1z"] bigspan = bro.find_element_by_class_name('nc-lang-cnt') print('big_span:',bigspan.size)# Length and width corresponding to the label # Slide div_tag action = ActionChains(bro) # Click and hold the specified label action.click_and_hold(span).perform() action.drag_and_drop_by_offset(span, 350, 0).perform() # Drag to a coordinate and release # action.drag_and_drop_by_offset(span, 400, 0) # Action chain release action.release() sleep(5) bro.quit()summary
- Slide click using action chain
Given the coordinates 234,11 | 21,3, click the data in the coordinates
- Convert coordinate data to a list
result = '253,83|253,153' all_list = [] # Stores the coordinates of the point to be clicked if '|' in result: list_1 = result.split('|') cout_1 = len(list_1) for i in range(count_1): xy_list = [] x = int(list_1[i].split(',')[0]) y = int(list_1[i].split(',')[1]) xy_list.append(x) xy_list.append(y) all_list.append(xy_list) else: xy_list = [] x = int(list_1[i].split(',')[0]) y = int(list_1[i].split(',')[1]) xy_list.append(x) xy_list.append(y) all_list.append(xy_list) print(all_list)
- Traverse the list and use the action chain operation
# Traverse the list and click the position specified by X and Y corresponding to each list element using the action chain for l in all_list: x = l[0] y = l[1] ActionChains(bro).move_to_element-with_offset(img,x,y).perform() # perform() executes immediately time.sleep(0.5) # img is the label that switches the reference to the current picture