Self-learning Web Scraping with AutoScraper in Python

AutoScraper is a Smart, Automatic, Fast and Lightweight Web Scraper for Python.

Developed by Alireza Mika, it can be downloaded at https://github.com/alirezamika/autoscraper

Concept and problem solved

Despite the availability of tools such as Beautiful Soup Web Scraping is difficult.

A library such Beautiful Soup helps you to:

• query a Web page
• parse the result of the query into a structured data structure: a tree
• query the resulting tree with an idiomatic way

But a Web Scraper doesn't write the query for you.

The purpose of a web page is to be consumed by humans not machines:

• the format of the page can change over time, and so the query could stop to work
• writing a "web scraping query" is time-consuming

What if a library could learn from an example and then can write the scrap query for you : it's "the reason d'être of AutoScraper".

How to use AutoScraper

Problem

I want to create a Web scraper for the Web site Quora to all the questions about a subject.

Phase 1 : the learning phase

• We train our library on a specific page: "https://www.quora.com/search?q=deep%20learning&time=year"
• We give to the library an example about what we seek on the page for example "When will deep learning finally die out?"

Install AutoScraper from Github

Copy
1pip install git+https://github.com/alirezamika/autoscraper.git

Create the file demo_train.py

Copy
1from autoscraper import AutoScraper
2
3# Parameters
4url = "https://www.quora.com/search?q=deep%20learning&time=year"
5model_name = "model_quora"
6
7wanted_list = ["When will deep learning finally die out?"]
8
9# We instanciate the AutoScraper
10scraper = AutoScraper()
11
12# We train the Scraper
13# Here we can also pass html content via the html parameter instead of the url (html=html_content)
14result = scraper.build(url, wanted_list)
15
16# We display the results if any
17if(result):
18  print("🚀 Great a query has been inferred !! Great gob.")
19  print(result)
20
21# If no result we leave with an error code
22if(result == None):
23  print("Sorry no query can be inferred ... 😿")
24  exit(-1)
25
26# We save the model for future use
27print(f"💿 > Save the model {model_name}")
28scraper.save(model_name)

We execute the file demo_train.py

Copy
1python3 demo_train.py

Result of the execution of demo_train.py

Copy
1🚀 Great a query has been inferred !! Great gob.
2['When will deep learning finally die out?', 'What newly developed machine learning models could surpass deep learning?', 'What is the future of machine learning/deep learning startups?', 'How promising is deep learning?', 'How can a regression problem be solved with deep learning?', 'What is the brutal truth about deep learning?', 'Why is there still no theory underlying deep learning?', 'What are the frameworks for deep learning modelling?', 'What is deep learning in terms of programming?']
3💿 > Save the model model_quora

Phase 2 : the usage phase

A model has been saved in the preceding step that contains all the rules of scraping.

Now, we can apply our model on a page that shares the same structure with the page we have used during the training phase.

We create a new file called demo.py

Copy
1from autoscraper import AutoScraper
2
3# AutoScraper must be installed with
4#  pip install git+https://github.com/alirezamika/autoscraper.git
5
6question = "france"
7time = "year"
8url = f"https://www.quora.com/search?q={question}&time={time}"
9model_name = "model_quora"
10
11scraper = AutoScraper()
12scraper.load(f"./{model_name}")
13# Get all the results in the page similar to our model
14results = scraper.get_result_similar(url)
15
16# if no results
17if results:
18  for r in results:
19    print(r)
20else:
21  print("No result found")

We test if the scraper works

Copy
1python3 demo.py

Result of the execution of demo.py

Copy
1Is France really as useless at war as portrayed in America?
2France fined Google 166M. Can Google just say no and not pay it? What are they going to do, ban Google in France?
3Is there freedom of expression in France?
4How is France dealing with Covid-19?
5What country is the oldest ally to France?
6Is France really 'littered' with abandoned chateaux?
7Why is France considered the most advanced country of Europe?
8Will Germany and France leave the European Union following Brexit?
9American expats to France, is France what it is cracked up to be?

Conclusion

The scraper must be trained again if the structure of the page changes.

The real advantage of the is approach is to be very reactive when a new format is available and to propose a new model quickly to continue the data extraction.

The library is very new. It's not perfect, but a big thanks to Alireza Mika for this great approach.