Integration guides

Connecting a proxy to Scrapy

Scrapy is a Python tool for building crawlers that collect many pages. Put proxy in the request meta and every request goes through ProxySup.

Create accountSee other guides
  • Connect with one gateway address
  • Set country and session in the username
  • Use your access token as the password

What you need to connect

Gateway
gate.proxysup.com:7777
Username
user-country-CODE
Password
Your account access token
Last updated
  • 1-line install
  • 4 steps to follow
  • Up to 30 min sticky sessions
  • HTTP, HTTPS, SOCKS5

Follow along

Scrapy: steps to connect the proxy

Follow the steps from the top. Put your access token where it says YOUR_TOKEN.

  1. Step 1

    Install Scrapy

    Terminal
    pip install scrapy

  2. Step 2

    Basic connection

    Code
    import scrapy

class PriceSpider(scrapy.Spider):
    name = "price"
    start_urls = ["https://ipinfo.io/json"]
    custom_settings = {"CONCURRENT_REQUESTS_PER_DOMAIN": 4}

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url, meta={
                "proxy": "http://user-country-de:[email protected]:7777"
            })

    def parse(self, response):
        yield {"body": response.text}

  3. Step 3

    Use a sticky session

    Part to change
    meta={"proxy": "http://user-country-de-session-a1b2:[email protected]:7777"}
# Use the same session name when you browse several pages in a row after logging in

  4. Step 4

    Check the connection

    Example response
    { "ip": "203.0.113.24",
  "country": "US",
  "org": "Home internet provider" }

Username

Username rules

Join values in the username to set the country, city, session and product.

ValueWhat it does
country-CODESets the country to connect from
city-CITYSets the city too. Only in countries with city targeting
session-NAMETurns on a sticky session. The same name gets the same IP
type-mobile or type-dcSwitches to mobile or datacenter proxies. Without it, residential is used

How the username fits together

userStartcountry-usCountrysession-a1b2Sticky session

Result user-country-us-session-a1b2

Troubleshooting

Scrapy: fixing connection errors

  • I get a 407 error

    It means the username or token is wrong. Check that there are no spaces before or after the token and that the username starts with user-.

  • The connection times out

    Raise the timeout to about 30 seconds and lower the number of concurrent requests to the same site.

  • The same IP is not kept

    Check that the username has a session value. Sticky sessions last up to 30 minutes.

  • The city I want is not used

    Check on the locations page that the country has city targeting, and write the city name in lowercase English letters with underscores.

More

Other integration guides

  • Using a proxy with Python requests

    requests is one of the most used libraries for sending web requests in Python. Put the ProxySup gateway address in proxies and it connects right away.

    View guide

  • Connecting a proxy to Playwright

    Playwright is a tool for driving a browser with code. Put the gateway, username and token in the proxy option when you launch the browser, and every page request goes through ProxySup.

    View guide

Connect Scrapy right away

You get an access token as soon as you sign up. Top up your balance to use the code above as is.

Create accountSee pricing