How to run Scrapy project in Jupyter?
On a Mac, I have Jupyter installed and when I type jupyter notebook from the root folder of my Scrapy project, it opens the notebook. I can browse all of the project files at this point.
How do I execute the project from the notebook?
If I click the Running tab, under Terminals, I see:
There are no terminals running.
There are two main ways to achieve that:
1. Under the Files tab open a new terminal: New > Terminal
Then simply run you spider: scrapy crawl [options] <spider>
2. Create a new notebook and use CrawlerProcess or CrawlerRunner classes to run in a cell:
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
process = CrawlerProcess(get_project_settings())
process.crawl('your-spider')
process.start() # the script will block here until the crawling is finished
Scrapy docs - Run Scrapy from a script
链接地址: http://www.djcxy.com/p/94204.html