twisted.internet.error.ReactorAlreadyInstalledError: reactor already installed

We Are Going To Discuss About twisted.internet.error.ReactorAlreadyInstalledError: reactor already installed. So lets Start this Python Article.

twisted.internet.error.ReactorAlreadyInstalledError: reactor already installed

  1. How to solve twisted.internet.error.ReactorAlreadyInstalledError: reactor already installed

    This solution avoids use of CrawlerProcess as stated in the docs.
    https://docs.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-script
    There’s another Scrapy utility that provides more control over the crawling process: scrapy.crawler.CrawlerRunner. This class is a thin wrapper that encapsulates some simple helpers to run multiple crawlers, but it won’t start or interfere with existing reactors in any way.
    It’s recommended you use CrawlerRunner instead of CrawlerProcess if your application is already using Twisted and you want to run Scrapy in the same reactor.
    from twisted.internet import reactor from scrapy.crawler import CrawlerRunner from scrapy.utils.project import get_project_settings from scrapy.utils.log import configure_logging from football.spiders.laliga import LaligaSpider # Enable logging for CrawlerRunner configure_logging() runner = CrawlerRunner(settings=get_project_settings()) for i in range(1, 29): runner.crawl(LaligaSpider, **{'week': i}) deferred = runner.join() deferred.addBoth(lambda _: reactor.stop()) reactor.run() # the script will block here until all crawling jobs are finished

  2. twisted.internet.error.ReactorAlreadyInstalledError: reactor already installed

    This solution avoids use of CrawlerProcess as stated in the docs.
    https://docs.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-script
    There’s another Scrapy utility that provides more control over the crawling process: scrapy.crawler.CrawlerRunner. This class is a thin wrapper that encapsulates some simple helpers to run multiple crawlers, but it won’t start or interfere with existing reactors in any way.
    It’s recommended you use CrawlerRunner instead of CrawlerProcess if your application is already using Twisted and you want to run Scrapy in the same reactor.
    from twisted.internet import reactor from scrapy.crawler import CrawlerRunner from scrapy.utils.project import get_project_settings from scrapy.utils.log import configure_logging from football.spiders.laliga import LaligaSpider # Enable logging for CrawlerRunner configure_logging() runner = CrawlerRunner(settings=get_project_settings()) for i in range(1, 29): runner.crawl(LaligaSpider, **{'week': i}) deferred = runner.join() deferred.addBoth(lambda _: reactor.stop()) reactor.run() # the script will block here until all crawling jobs are finished

Solution 1

This solution avoids use of CrawlerProcess as stated in the docs.
https://docs.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-script

There’s another Scrapy utility that provides more control over the crawling process: scrapy.crawler.CrawlerRunner. This class is a thin wrapper that encapsulates some simple helpers to run multiple crawlers, but it won’t start or interfere with existing reactors in any way.

It’s recommended you use CrawlerRunner instead of CrawlerProcess if your application is already using Twisted and you want to run Scrapy in the same reactor.

from twisted.internet import reactor
from scrapy.crawler import CrawlerRunner
from scrapy.utils.project import get_project_settings
from scrapy.utils.log import configure_logging

from football.spiders.laliga import LaligaSpider

# Enable logging for CrawlerRunner
configure_logging()

runner = CrawlerRunner(settings=get_project_settings())
for i in range(1, 29):
    runner.crawl(LaligaSpider, **{'week': i})

deferred = runner.join()
deferred.addBoth(lambda _: reactor.stop())

reactor.run()  # the script will block here until all crawling jobs are finished
Advertisements

Original Author Md. Mujtaba Raza Of This Content

Solution 2

For me this worked, I put it before the CrawlerProcess
import sys if "twisted.internet.reactor" in sys.modules: del sys.modules["twisted.internet.reactor"]

Advertisements

Original Author Mersad Of This Content

Solution 3

Delete the reactor installed before process.start()

For me this worked, I put it before the CrawlerProcess import sys if >”twisted.internet.reactor” in sys.modules: del >sys.modules[“twisted.internet.reactor”]

Same here, just mine worked when I put it before the process.start()

import sys

process = CrawlerProcess(get_project_settings())
process.crawl("articles")

if "twisted.internet.reactor" in sys.modules:
    del sys.modules["twisted.internet.reactor"]

process.start()
Advertisements

Original Author Atehe Of This Content

Solution 4

I’ve just run into this issue as well. It appears that the docs at https://docs.scrapy.org/en/latest/topics/practices.html are incorrect in stating that CrawlerProcess can be used to run multiple crawlers built with spiders, since each new crawler attempts to load a new reactor instance if you give it a spider. I was able to get my code to work by using CrawlerRunner instead, as also detailed on the same page.

import scrapy
from twisted.internet import reactor
from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging
from scrapy.utils.project import get_project_settings

class MySpider1(scrapy.Spider):
    # Your first spider definition
    ...

class MySpider2(scrapy.Spider):
    # Your second spider definition
    ...

configure_logging()
settings = get_project_settings() # settings not required if running
runner = CrawlerRunner(settings)  # from script, defaults provided
runner.crawl(MySpider1) # your loop would go here
runner.crawl(MySpider2)
d = runner.join()
d.addBoth(lambda _: reactor.stop())
reactor.run() # the script will block here until all crawling jobs are finished
Advertisements

Original Author z3r0fox Of This Content

Conclusion

Advertisements

So This is all About This Tutorial. Hope This Tutorial Helped You. Thank You.

Also Read,

ittutorial team

I am an Information Technology Engineer. I have Completed my MCA And I have 4 Year Plus Experience, I am a web developer with knowledge of multiple back-end platforms Like PHP, Node.js, Python and frontend JavaScript frameworks Like Angular, React, and Vue.

Leave a Comment