Web Scraping Best Practices for Lead Generation

Web scraping is powerful, but with great power comes great responsibility. Here's how to do it right.

Why Ethics Matter

Unethical scraping can lead to:

• IP bans and blocked access
• Legal issues under CFAA or GDPR
• Damaged reputation in your industry
• Overloaded servers that hurt other users

Scrappy is built with ethics in mind. We help you get the data you need while respecting websites and their users.

The Golden Rules

1. Respect robots.txt

Every website has a robots.txt file that specifies what can and can't be scraped.

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /public/

Scrappy checks a site's robots.txt and reports the result, and it honors a crawl delay when one is set. It does not block a source you have configured, so the decision to respect a disallow rule is yours. Read the report and act on it.

2. Rate Limit Your Requests

Hammering a website with requests will:

• Get you blocked
• Slow down their site for real users
• Potentially crash smaller servers

Scrappy's approach:

• Delays between requests
• A crawl delay from robots.txt is honored when present

3. Identify Yourself

Some sites appreciate knowing who's scraping them. Scrappy's cloud browser is a real Chrome browser and presents as one; it does not announce itself as a bot. If a site asks automated visitors to identify themselves or to stay away, that is for you to honor.

4. Cache When Possible

Don't re-scrape data you already have. Scrappy caches email validation results (180 days by default), so validating the same address again does not repeat the work. Schedule sources no more often than the page actually changes.

Legal Considerations

GDPR Compliance

If you're collecting data on EU citizens:

• Only collect data that's publicly available
• Document your legitimate interest
• Provide opt-out mechanisms
• Honor data deletion requests

Scrappy provides tools to help you meet these obligations (consent records, data export, account deletion, suppression lists). Using them does not make your use of the data compliant by itself.

CAN-SPAM and CCPA

When using scraped emails:

• Include your physical address
• Provide clear unsubscribe options
• Honor opt-out requests within 10 days
• Don't sell personal data without consent

Terms of Service

Some websites explicitly prohibit scraping in their ToS. While this is often unenforceable, it's good to be aware of it.

Technical Best Practices

Target the Right Pages

Don't scrape entire websites. Target specific pages:

• /team or /about - Team member info
• /contact - Contact details
• /leadership - Executive information

Handle Dynamic Content

Modern sites use JavaScript. Scrappy uses a cloud browser that:

• Executes JavaScript
• Waits for content to load
• Handles infinite scroll
• Bypasses basic bot detection

Extract Clean Data

Raw HTML is messy. Scrappy extracts:

• Valid email addresses only
• Phone numbers in consistent formats
• Social media links
• Company metadata

Validate Everything

Never trust scraped data blindly:

• Validate email syntax and deliverability
• Verify phone number formats
• Check for spam traps and honeypots

Common Mistakes to Avoid

Mistake 1: Scraping Too Fast

Wrong: 100 requests per second Right: 1-2 requests per second, with delays

Mistake 2: Ignoring Errors

Wrong: Keep retrying failed requests indefinitely Right: Exponential backoff with max retries

Mistake 3: Not Handling Edge Cases

Watch for:

• 404 pages
• Redirects
• CAPTCHAs
• Login walls
• Rate limit responses (429)

Mistake 4: Storing Sensitive Data

Never scrape or store:

• Passwords or credentials
• Payment information
• Health records
• Private communications

Scrappy's Built-in Protections

We've built these best practices into Scrappy:

| Feature | What It Does | |---------|--------------| | Request pacing | Delays between requests | | robots.txt check | Checks site rules and reports the result (not enforced) | | Error handling | Graceful retry with backoff | | Data validation | Cleans and validates all output | | GDPR tools | Consent records, export, deletion, suppression lists | | Proxy rotation | Distributes requests across IPs |

When to Use Scraping

Good use cases:

• Building B2B lead lists from public directories
• Competitive research from public sources
• Aggregating publicly available contact info

Bad use cases:

• Scraping private member areas
• Bypassing paywalls
• Collecting personal data without consent
• Overloading small business websites

The Bottom Line

Web scraping is legal and ethical when done right. Respect websites, follow the rules, and use data responsibly.

Scrappy makes it easy to scrape ethically while still getting the data you need to grow your business.