Use cases

Proxies for AI training data collection

AI training data collection means gathering large numbers of public web documents and organizing them into datasets for language or search models. With ProxySup proxies you can spread requests across many IPs and countries.

Create accountSee integration guides
  • Collect only public documents allowed by robots.txt and the terms of use
  • Leave pages that contain personal information out of the collection

Recommended setup: AI training data collection

Product
Residential proxies, datacenter proxies for lightly protected sites
Session
Rotating sessions
Country
Countries that use the languages you collect
Speed
Set a limit on concurrent requests for each site
  • Residential proxies
  • Rotating sessions
  • 6 recommended countries
  • 4 steps

Why

Why AI training data collection needs a proxy

Collection at scale sends many requests in a short time, which one IP cannot keep up for long. Spreading requests across many home IPs keeps them from piling up in one place, and lets you collect documents in different languages evenly by country.

Repeated requests from one IP

  1. Request 1198.51.100.7
  2. Request 2198.51.100.7
  3. Request 3198.51.100.7

Requests spread across several home IPs

  1. Request 1203.0.113.11
  2. Request 2203.0.113.27
  3. Request 3203.0.113.42
The IP addresses shown are examples.

Steps

AI training data collection: steps

  1. Step 1

    Choose the sites and languages to collect, and check robots.txt

  2. Step 2

    Set the number of concurrent requests and the interval for each site

  3. Step 3

    Fetch documents with rotating sessions and save the text with its URL and date

  4. Step 4

    Filter out duplicates and empty documents and organize the dataset

Code

AI training data collection: code example

Example for Python, Scrapy. Put your access token where it says YOUR_TOKEN.

Python, Scrapy
import scrapy

class DocsSpider(scrapy.Spider):
    name = "docs"
    start_urls = ["https://example.com/articles"]
    custom_settings = {"CONCURRENT_REQUESTS_PER_DOMAIN": 4, "DOWNLOAD_DELAY": 1}

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url, meta={
                "proxy": "http://user-country-us:[email protected]:7777"
            })

    def parse(self, response):
        yield {"url": response.url, "html": response.text}

Check

AI training data collection: things to watch

  • Collect only public documents allowed by robots.txt and the terms of use

  • Leave pages that contain personal information out of the collection

  • Saving the fetch date and original URL makes it easy to check sources later

Countries

Countries often used for AI training data collection

More

Other proxy use cases

  • Proxies for price monitoring

    Price monitoring means collecting product prices and stock from many shops on a schedule and comparing them. With ProxySup residential proxies you see the prices that real visitors see in each country.

    View setup

  • Proxies for ad verification

    Ad verification means checking how your ads actually look to users in a target area. With ProxySup residential and mobile proxies you can check the same ad while switching country and city.

    View setup

  • Proxies for search result tracking

    Search result tracking means collecting which results appear, and in what position, for set keywords in each country and city. With ProxySup residential proxies you can check the same keyword while switching location.

    View setup

AI training data collection: set it up now

Sign up and top up your balance to use the setup above right away.

Create accountSee pricing