API

many_requests

many_requests.many_requests module

class many_requests.many_requests_.ManyRequests(n_workers: int = 15, n_connections: int = 10, retries: int = 10, retry_sleep: float = 3, ok_codes: Iterable[int] = 200, ok_response_func: Callable = None, json: bool = False)

__init__(n_workers: int = 15, n_connections: int = 10, retries: int = 10, retry_sleep: float = 3, ok_codes: Iterable[int] = 200, ok_response_func: Callable = None, json: bool = False)

Dead easy interface for executing many HTTP requests asynchronously.

Parameters

• n_workers – Max number of workers to use. Too many workers will use a lot of memory and increase startup time, too few can lead to slower execution.

• n_connections – Max number of open connections to have open at once. The number of connections is also limited by the OS. For example, by default MacOS has a limit of 256 and Ubuntu has ~66k. These limits can be changed with OS configuration.

• retries – Number of retries to attempt if a request fails

• retry_sleep – How long to wait in seconds before retrying a request

• ok_codes – A sequence of HTTP status codes to accept as ok. If any, all responses will be assumed to be ok

• ok_response_func – A function to apply to the response to determine if ok. Should return True/False.

• json – Parse response body as json and return instead of full Responce object

Examples

Execute 10 GET requests to https://example.org

>>> responses = ManyRequests(n_workers=5, n_connections=5)(
>>>                 method='GET', url=[f'https://example.org' for i in range(10)])

__call__(method: Union[str, List[str]], url: Union[str, List[str]], params: Union[None, str, Dict, List[str], List[Dict]] = None, data: Union[None, str, Dict, List[str], List[Dict]] = None, json: Union[None, Dict, List[Dict]] = None, headers: Union[None, Dict, List[Dict]] = None, cookies: Union[None, Dict, List[Dict]] = None, auth: Union[None, asks.auth.AuthBase, List[asks.auth.AuthBase]] = None) → List[Union[asks.response_objects.Response, many_requests.common.BadResponse]]

Process asynchronously many requests, handling bad responses. Return the responses in the same order. If no ok response was obtained after retires a BadResponse will be included in the corresponding position of the output. A BadResponse will contain the last response and error reason.

Arguments mimic asks.request, which in turn mimics requests.request.

Each argument can be a single item or a list of N items, where N is the number of requests. When the argument is a single item, that attribute is duplicated N times for every request.

Parameters

• method – HTTP method type GET, OPTIONS, HEAD, POST, PUT, PATCH, or DELETE.

• url – URL of the Request

• params – Data to send in the query string of the Request. Dictionary or string.

• data – Data to send in the body of the Request. Dictionary or string.

• json – A JSON serializable dictionary to send as JSON in the body of the Request.

• headers – Dictionary of HTTP Headers to send with the Request

• cookies – Dictionary object to send with the Request

• auth – Enable Basic/Digest/Custom HTTP Auth. Should be a child of asks.AuthBase

Returns

A list of responses in the same order of the requests. Will include a BadResponse in the position of a request where no good response was obtained.

Return type

responses

many_requests.easy_async module

class many_requests.easy_async.EasyAsync(n_workers: int = 15)

__init__(n_workers: int = 15)

Dead simple parallel execution of async coroutines. n_workers are dispatched which asynchronously process each task given.

Parameters

n_workers – Number of workers to use

Examples

Each task sleeps for i seconds asyncronosly:

>>> EasyAsync(n_workers = 4)(delayed(trio.sleep)(i) for i in range(10))

Each task calculates isclose with a different a and b paramter. abs_tol is set the same for all.

>>> from math import isclose
>>> async def isclose_(a, b, abs_tol): return isclose(a=a, b=b, abs_tol=abs_tol)
>>> EasyAsync(n_workers = 4)(
>>>     delayed(isclose_)(**kwargs)
>>>     for kwargs in zip_kw(a=range(10), b=range(10)[::-1], abs_tol=4))

__call__(tasks: Iterable[many_requests.easy_async.delayed], length: Optional[int] = None)

Execute given coroutine tasks using workers. The order of output will match the order of input.

Parameters

• tasks – A sequence of coroutines to execute

• length – The number of tasks. If not specified it will try obtain the length automatically. Used for progress bar

Returns

A list of outputs from each task. The order of items is determined by the input.

many_requests.easy_async.delayed(func: Callable[[…], Coroutine])

Decorator used to capture an async function with arguments and delay its execution.

Examples

>>> import asks
>>> func_, args_, kwargs_ = delayed(asks.request)('GET', url='https://example.org')
>>> assert func_ == asks.request
>>> assert args_ == ('GET',)
>>> assert kwargs_ == {'url': 'https://example.org'}

many_requests.easy_async.zip_kw(**kwargs)

Return an iterator of N dictionaries, where the Nth item of each iterator specified in kwargs is paired together. Like zip but returns a dict instead. Also keyword arguments which are strings or not iterators are automatically repeated N times. zip_kw stops when the shortest iterator has been exhausted.