SecT0uch
/
ImmoScrap
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Disable cookies, added download delay

This commit is contained in:
Jordan ERNST 2020-08-16 01:23:04 +02:00
parent 0655c01cc9
commit 214c351efc
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
ImmoScrap/settings.py
View File

@ -25,13 +25,13 @@ ROBOTSTXT_OBEY = False
# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
# DOWNLOAD_DELAY = 3
DOWNLOAD_DELAY = 2 # Beetween 1 and 3 seconds : RANDOMIZE_DOWNLOAD_DELAY enabled by default
# The download delay setting will honor only one of:
# CONCURRENT_REQUESTS_PER_DOMAIN = 16
# CONCURRENT_REQUESTS_PER_IP = 16
# Disable cookies (enabled by default)
# COOKIES_ENABLED = False
COOKIES_ENABLED = False # Some sites may use cookies to spot bot behaviour
# Disable Telnet Console (enabled by default)
# TELNETCONSOLE_ENABLED = False