Use solvethree in a Scrapy downloader middleware

• Use Scrapy 2.19+, Python 3.10+ and the official solvethree Python SDK. The SDK is synchronous; it has no native async client.
• Create an active site profile for your workspace in the console. Sending a hostname and action doesn't create one. See site profile setup.
• Set TARGET_URL to the exact form submission URL, SOLVETHREE_PROXY to the proxy used for this application request, and SOLVETHREE_API_KEY to your workspace key. Keep the key server-side.
• Enable AsyncioSelectorReactor and register the middleware at priority 740, as shown below. This places it before HttpProxyMiddleware at 750.

# middlewares.py; Scrapy 2.19+, Python 3.10+, official solvethree SDK.
import asyncio
import os
from urllib.parse import urlsplit
from scrapy.http import FormRequest
from solvethree import Solvethree

class CaptchaMiddleware:
    async def process_request(self, request):
        if not request.meta.get("solvethree_form") or request.meta.get("solvethree_applied"):
            return None
        target = os.environ["TARGET_URL"]
        if request.url != target:
            raise ValueError("Only the configured form target may use this middleware")
        proxy = os.environ["SOLVETHREE_PROXY"]
        def solve():
            return Solvethree(os.environ["SOLVETHREE_API_KEY"]).solve({
                "site": urlsplit(target).hostname, "action": "login", "proxy": proxy,
                "requestId": request.meta["solvethree_request_id"],
            }, timeout=180)
        result = await asyncio.to_thread(solve)
        solution = result["solution"]
        fields = dict(request.meta["solvethree_fields"])
        fields["g-recaptcha-response"] = solution["token"]
        meta = {**request.meta, "proxy": proxy, "solvethree_applied": True,
                "dont_retry": True, "dont_redirect": True}
        return FormRequest(target, formdata=fields, callback=request.callback,
            errback=request.errback, headers={"User-Agent": solution["userAgent"],
                "sec-ch-ua": solution["secChUa"]}, meta=meta, dont_filter=True)

# settings.py:
# TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
# DOWNLOADER_MIDDLEWARES = {"myproject.middlewares.CaptchaMiddleware": 740}
# The spider first loads its form/CSRF data with the same proxy and cookiejar.
# Then yield a Request(TARGET_URL, callback=your_result_handler, meta={
#     "solvethree_form": True, "solvethree_request_id": one_saved_logical_job_id,
#     "solvethree_fields": {"csrf": token_from_the_form, "email": "your-test-value"},
#     "cookiejar": your_cookiejar_id,
# })
# Inspect the final response in your_result_handler; a ready task is not that response.

The middleware's process_request method takes request only, without the deprecated spider argument. Unmarked requests pass through unchanged. A marked request must match TARGET_URL exactly; a mismatch raises an error rather than sending a token to another endpoint.

asyncio.to_thread keeps the synchronous SDK's network calls and polling off the event-loop thread. Merely declaring process_request async wouldn't do that. The SDK creates a task and waits for a ready result or raises; the middleware then builds the form submission.

1. 1.

   Set the correct action and token field

   The example uses the login action and puts solution.token in g-recaptcha-response. Change those to match the target's integration. The task's site is derived from TARGET_URL's hostname; reCAPTCHA tasks don't take type or url fields. Omitting recaptchaType selects enterprise-v3.

2. 2.

   Supply existing form fields

   The spider must populate solvethree_fields with the endpoint's required fields, including its current CSRF value. The middleware copies those fields and adds the token; it doesn't load the form or extract hidden inputs.

3. 3.

   Keep the session consistent

   Load the initial form through the same proxy and browser identity required by the application. Pass its cookiejar ID in the marked request's metadata. The returned FormRequest retains that cookiejar and uses solution.userAgent and solution.secChUa, but the example doesn't prepare or repair the earlier session.

4. 4.

   Handle the application response

   Supply a callback that checks whether the form submission succeeded and an errback for request failures. A ready task means a solution was generated, not that the website accepted the subsequent request.

This example deliberately uses one proxy for solving and submission. Don't replace it with an unrelated proxy after solving. Also inspect any custom downloader middleware that rewrites User-Agent or client hints: it must not overwrite the identity returned with the solution.

Save solvethree_request_id before yielding the marked request, and keep it for that logical solve. A requestId accepts 1–64 ASCII letters, numbers, underscores or hyphens. Reusing it with identical task inputs recovers the existing task; changing those inputs while retaining the ID conflicts.

The returned FormRequest sets solvethree_applied so it won't trigger another solve. It also sets dont_retry and dont_redirect, preventing automatic retries and redirects for that submission. Those flags don't prevent your own callback or job runner from starting another request, so make that decision explicitly.

The solve call times out.

Determine whether a task was already created. A connection timeout doesn't establish that creation failed.

Try this: Retrieve the known task or recover it using the saved requestId and identical task inputs before creating a new logical solve.

Solving raises instead of returning a result.

Distinguish an HTTP request error from a terminal task failure. For a failed task, inspect errorCode and error.

Try this: Handle the failure rather than repeatedly yielding the same marked request. Correct invalid inputs before trying again; failed tasks must not continue polling.

The website rejects the submitted form.

Inspect the outgoing token field, CSRF fields, cookies, proxy and browser identity first.

Try this: Correct the application request before paying for another solve. The field must contain solution.token, not the task ID or a stale token.