snippetbashTip

scrapy — Web-crawling framework. More information: <https://docs.scrapy.org/en/latest/topics/commands.html#us

Submitted by: @import:tldr-pages·Mar 2, 2026·

Viewed 0 times

commandscrapyframeworkcliinformationwebmorecrawling

Problem

How to use the scrapy command: Web-crawling framework. More information: <https://docs.scrapy.org/en/latest/topics/commands.html#using-the-scrapy-tool>.

Solution

scrapy — Web-crawling framework. More information: <https://docs.scrapy.org/en/latest/topics/commands.html#using-the-scrapy-tool>.

Create a project:

scrapy startproject {{project_name}}

Create a spider (in project directory):

scrapy genspider {{spider_name}} {{website_domain}}

Edit spider (in project directory):

scrapy edit {{spider_name}}

Run spider (in project directory):

scrapy crawl {{spider_name}}

Fetch a webpage as Scrapy sees it and print the source to stdout:

scrapy fetch {{url}}

Open a webpage in the default browser as Scrapy sees it (disable JavaScript for extra fidelity):

scrapy view {{url}}

Open Scrapy shell for URL, which allows interaction with the page source in a Python shell (or IPython if available):

scrapy shell {{url}}

Code Snippets

Create a project

scrapy startproject {{project_name}}

Create a spider (in project directory)

scrapy genspider {{spider_name}} {{website_domain}}

Edit spider (in project directory)

scrapy edit {{spider_name}}

Run spider (in project directory)

scrapy crawl {{spider_name}}

Fetch a webpage as Scrapy sees it and print the source to `stdout`

scrapy fetch {{url}}

Context

tldr-pages: common/scrapy

Revisions (0)

No revisions yet.