GPT-Web-Crawler is a web crawler based on python and puppeteer. It can crawl web pages and extract content (including WebPages' title,url,keywords,description,all text content,all images and screenshot) from web pages. It is very easy to use and can be used to crawl web pages and extract content from web pages in a few lines of code. It is very suitable for people who are not familiar with web crawling and want to use web crawling to extract content from web pages.
The output of the spider can be a json file, which can be easily converted to a csv file, imported into a database or building an AI agent.
Step1. Install the package.
pip install gpt-web-crawlerStep2. Copy config_template.py and rename it to config.py. Then, edit the config.py file to config the openai api key and other settings, if you need use ProSpider to help you extract content from web pages. If you don't need to use ai help you extract content from web pages, you can keep the config.py file unchanged.
Step3. Run the following code to start a spider.
from gpt_web_crawler import run_spider,NoobSpider
run_spider(NoobSpider,
max_page_count= 10 ,
start_urls="https://www.jiecang.cn/",
output_file = "test_pakages.json",
extract_rules= r'.*\.html' )In the above code, the NoobSpider is used. There are four spiders in the package, which are NoobSpider, CatSpider, ProSpider and LionSpider. They are different in the content they can extract from the web page. The following table shows the differences between them.
| Spider Type | Description | Return Content |
|---|---|---|
| NoobSpider | Basic web page scraping | - titile - url - keywords - description - body :all text content of web page |
| CatSpider | Web page scraping with screenshots | - titile - url - keywords - description - body :all text content of web page - screenshot_path |
| ProSpider | Web page scraping with AI-extracted content | - titile - url - keywords - description - body :all text content of web page - ai_extract_content : gpt's extraction of body text |
| LionSpider | Web page scraping with all images extracted | - titile - url - keywords - description - body :all text content of web page - directory : the directory of all pics on web page |
Cat spider is a spider that can take screenshots of web pages. It is based on the Noob spider and uses puppeteer to simulate browser operations to take screenshots of the entire web page and save it as an image. So when you use the Cat spider, you need to install puppeteer first.You can refer to this answer to install npm, and then use the following command to install puppeteer:
npm install puppeteer- 支持无需配置config.py
- 爬虫更多自定义内容