Skip to main content

RequestLoader

An abstract class defining the interface for classes that provide access to a read-only stream of requests.

Request loaders are used to manage and provide access to a storage of crawling requests.

Key responsibilities:

  • Fetching the next request to be processed.
  • Marking requests as successfully handled after processing.
  • Managing state information such as the total and handled request counts.

Hierarchy

Index

Methods

Methods

:

:

:

:

:

:

to_tandem

  • Combine the loader with a request manager to support adding and reclaiming requests.


    Parameters

    • optionalrequest_manager: RequestManager | None = None

      Request manager to combine the loader with. If None is given, the default request queue is used.

    Returns RequestManagerTandem

Page Options