70
Demystifying Web Scraping: How to Extract Data from Complex Websites
In the previous part, of this tutorial, we did some base work like setting up the spider, collecting products, finding price API, and some other useful lessons. I will advise you to check that out as well.
In part 2 of Scraping Complex Websites, we will be seeing how to extract the extra products that are loaded as we scroll down.
Some things we will do involve
- Finding the right request from the network tab
- copying and formatting the curl request
- Mimicking the browser request in scrapy
- Making a post request in scrapy
- Merging the code
This video part has a lot of technical details that i decided to make in form of a video. feel free to check the source code on GitHub
