python advanced crawler notes

Written in front selenium is a friendly crawler tool for novices, but I don't think it is suitable for novices. It is recommended that you look at selenium after you understand the reptiles of the requests system and have some common sense of reptiles. In fact, the crawler of requests system is enough ...

Posted on Wed, 05 Feb 2020 07:00:08 -0500 by davard

Remember a failed crawl

Receiving one day's inspiring false news led me to go to the public information website to find information about designated drugstores. Although the results were relatively unsuccessful, the process was very happy, so I can write down another article about water.The following is the original text: The page search function is limited, I made a ...

Posted on Mon, 03 Feb 2020 22:08:42 -0500 by ale1981

Using selenium library to crawl data from JD's interface

Environmental needs python running environment Selenium Library (pip install selenium) Pyquery Library (pip install pyquery) pymongo (pip install pymongo) there is a mongo database locally. If there is no mongo database, you don't need to install it Questions raised We open Jingdong's web page, e ...

Posted on Sat, 01 Feb 2020 07:25:12 -0500 by crwtrue

Python 3 + selenium + driver operation test

Preface If you want to use Selenium's violent operation of a disk, write a record Selenium Selenium is a tool for testing Web applications. Selenium tests run directly in the browser, just as real users do. Supported browsers include IE (7, 8, 9, 10, 11), Mozilla Firefox, Safari, Google Chrome, Opera ...

Posted on Wed, 22 Jan 2020 04:23:46 -0500 by php_coder_dvo

On unit test

Unit tests or the best project documentation. When I was learning to use Java for testing a long time ago, I got the help of a mysterious big man and talked about unit testing together. The basic conclusion is that unit testing is probably useless. As we all know, an obvious feature of automated testing compared with manual testing is that it ...

Posted on Wed, 22 Jan 2020 00:05:36 -0500 by marcusb

python anti crawler series I (text obfuscation)

python anti crawler series I (text obfuscation) Statement: for technical communication only, please do not use it for illegal purposes. Any loss caused by other illegal purposes is irrelevant to this blog Catalog python anti crawler series I (text obfuscation) 1. Picture camouflage anti reptile S ...

Posted on Tue, 21 Jan 2020 02:16:35 -0500 by tamir_malas

The wxpython interface of python3 simulates the login and crawls the academic system score

Preface Today, I write the code based on the previous code of climbing the academic record of the educational administration system with simulated Login, and make a visual operation interface with wxpython. The tools used are still selenium library, beutifulsoup4 library, Wx of the design interface, a ...

Posted on Mon, 20 Jan 2020 10:19:56 -0500 by bobocheez

Actually tested two GitHub open source ticket grabbing plug-ins, and all the pits have been stepped on for you

If you don't have any confidence in your speed and all kinds of "acceleration packs" on the market, you might as well try the programmer's method to grab tickets? What's more, [12306 official announced to block a large number of paid ticket grabbing software], which means that even if you pay the membership fee for these software, you ...

Posted on Sat, 18 Jan 2020 13:33:13 -0500 by newbeee

The first play of the imitative book retrieval system Python

Personal blog https://blog.fmujie.cn/ Production reason: Simply put, the library teacher has a project that needs to tell a company something and needs a retrieval system, but the data source has to crawl from the Internet. Because the account is very expensive, it's not worth buying another accou ...

Posted on Fri, 17 Jan 2020 10:29:41 -0500 by depsipher

python crawler -- [Baidu knows] auto answer

The first python crawler project I did. I just started to learn it. If there are any mistakes, point out that it's OK Baidu knows how to answer questions automatically function Visit Baidu Knows , we will see a lot of new questions. In fact, many of the questions have been explained or ready-made ...

Posted on Thu, 16 Jan 2020 10:52:03 -0500 by vombomin