Question 1

What does Crawlbase Web Scraping (formerly ProxyCrawl) do?

Accepted Answer

Crawlbase (formerly ProxyCrawl) is a managed web scraping and crawling API that provides developers with a cloud-based proxy network and optional headless browser rendering to scrape any website without being blocked. Nigerian developers building data extraction pipelines, price monitors, news aggregators, or competitive intelligence tools can use Crawlbase to collect structured web data from Nigerian and international websites without maintaining their own proxy infrastructure, managing IP rotation, or handling anti-bot detection systems.

**What Crawlbase Does**

Crawlbase provides two primary scraping modes:

**1. Normal API (HTTP Proxy Scraping)**
Sends a standard HTTP request through a rotating pool of residential and datacenter proxies. Returns the full HTML of the target page. Suitable for sites that don't require JavaScript execution. Each successful request consumes one normal token from your monthly quota.

**2. JavaScript API (Headless Chromium)**
Loads the target URL in a full headless Chromium browser — executing JavaScript, loading dynamic content, and waiting for the page to fully render before returning the HTML. Essential for scraping React, Angular, or Vue.js applications (like many modern Nigerian e-commerce sites). Each JS request costs more tokens than a normal request.

**Additional Features**

- **CAPTCHA handling**: The JavaScript API attempts to solve common CAPTCHA types automatically
- **Async scraping**: Submit a URL for async processing and receive a callback when complete — useful for large-scale crawls
- **Storage API**: Store scraped HTML or structured data in Crawlbase's cloud storage for later retrieval
- **Geolocation proxies**: Request proxies from specific countries for geo-targeted content
- **Screenshot API**: Capture full-page screenshots of any URL (useful for visual monitoring)
- **Smart proxy rotation**: Crawlbase automatically selects the best proxy for each target domain based on success history

**How Developers Use Crawlbase**

After registering and obtaining tokens:

Normal scraping:
```
GET https://api.crawlbase.com/?token=YOUR_NORMAL_TOKEN&url=https://www.vanguardngr.com/news/
```

JavaScript rendering:
```
GET https://api.crawlbase.com/?token=YOUR_JS_TOKEN&javascript=true&url=https://www.jumia.com.ng/phones/
```

With callback for async processing:
```
GET https://api.crawlbase.com/?token=YOUR_TOKEN&url=TARGET_URL&callback=https://yourserver.com/webhook
```

All responses return the full page HTML plus metadata (status code, original URL, scraping mode used).

**Pricing**

- **Free Tier**: 1,000 requests/month (mix of normal and JS tokens). No credit card required.
- **Paid Plans**: Starting at $29/month for higher volumes. Normal and JavaScript requests are tracked separately — JS requests are typically billed at a higher rate than normal requests.
- **Annual Discount**: Available on paid plans.

The 1,000 free requests/month is among the most generous free tiers in the scraping API space, making Crawlbase suitable for development, testing, and low-volume production use cases.

**Nigerian Developer Use Cases**

**E-Commerce Price Monitoring**: Nigerian e-commerce platforms (Jumia, Konga) use React-rendered product pages. Crawlbase's JavaScript mode successfully renders these pages, enabling price monitoring for thousands of products across Nigerian retail sites.

**Competitive Intelligence**: Nigerian businesses can monitor competitor pricing, promotional campaigns, and product catalog changes by scraping public-facing competitor websites on a scheduled basis.

**Financial Data Collection**: Nigerian fintech companies can collect exchange rate data, loan product information, and savings rates from Nigerian bank websites using Crawlbase — aggregating data that isn't available through official APIs.

**Government Data Collection**: Nigerian government portals (BPP, NCC, CBN) publish regulations, circulars, and decisions as HTML pages. Crawlbase enables automated collection of these documents for compliance monitoring services.

**Real Estate Data**: Property listing sites in Nigeria (PropertyPro.ng, Nigeria Property Centre) publish structured listing data in HTML. Crawlbase can collect this data for market analysis and property valuation models.

**Anti-Detection Capabilities**

Nigerian websites, like most web properties, implement various bot-detection measures:
- User-agent checking (Crawlbase uses realistic browser user-agents)
- Rate limiting (Crawlbase distributes requests across its proxy pool)
- IP reputation checks (Crawlbase's residential proxies use legitimate residential IPs)
- JavaScript-based bot detection (Crawlbase's JS mode passes JavaScript challenges)

These capabilities mean that most Nigerian websites that don't require login will be accessible through Crawlbase.

**Legal and Ethical Considerations**

Web scraping must respect robots.txt files, terms of service, and Nigerian NDPR when collecting personal data. Crawlbase is a tool — it is the developer's responsibility to scrape only publicly accessible data, comply with website terms, and handle any personally identifiable information in compliance with applicable laws. Never use Crawlbase to scrape data from authenticated sections of websites without authorization.

**Company Background**

Crawlbase was founded as ProxyCrawl and rebranded to Crawlbase around 2021, reflecting a broader platform vision beyond proxy-based scraping. The company provides infrastructure for thousands of developers and businesses worldwide conducting legal web data collection. Their infrastructure is maintained across multiple global data centers for reliability and geographic coverage.

Crawlbase is one of the best-balanced scraping APIs for Nigerian developers — combining a generous free tier, JavaScript rendering capabilities for modern websites, built-in anti-detection, and straightforward pricing.

Question 2

How much does Crawlbase Web Scraping (formerly ProxyCrawl) cost?

Accepted Answer

Crawlbase Web Scraping (formerly ProxyCrawl) is free to start, with paid tiers for higher usage (freemium). Free: 1,000 requests/month. Paid: from $29/month. JavaScript (headless browser) requests cost more than plain HTTP requests. Annual discounts available.

Question 3

How do I authenticate with the Crawlbase Web Scraping (formerly ProxyCrawl) API?

Accepted Answer

API Token (query parameter: ?token=YOUR_TOKEN). Separate tokens for normal vs JavaScript requests.

Question 4

Is there a rate limit on Crawlbase Web Scraping (formerly ProxyCrawl)?

Accepted Answer

1,000 free requests/month. Paid plans for higher volumes. JavaScript requests billed at higher rate.

Question 5

Does Crawlbase Web Scraping (formerly ProxyCrawl) offer a sandbox or test mode?

Accepted Answer

Yes — Crawlbase Web Scraping (formerly ProxyCrawl) provides a sandbox or test environment so you can integrate and test before going live.

Question 6

What is Crawlbase Web Scraping (formerly ProxyCrawl) best suited for?

Accepted Answer

Nigerian developers building web scrapers, data extraction pipelines, or price monitors who need a managed proxy network and optional headless browser rendering to bypass blocks without managing their own proxy infrastructure.

Question 7

Is there anything else I should know before integrating Crawlbase Web Scraping (formerly ProxyCrawl)?

Accepted Answer

Formerly known as ProxyCrawl. Crawlbase provides two request types: normal (plain HTTP with proxy rotation) and JavaScript (headless Chromium for JS-rendered pages). Nigerian websites and global sites both supported. 1,000 free requests/month.

Question 8

Where can I find Crawlbase Web Scraping (formerly ProxyCrawl)'s documentation?

Accepted Answer

The official API documentation is at https://crawlbase.com/docs. The provider's website is https://crawlbase.com.

Question 9

Does Crawlbase Web Scraping (formerly ProxyCrawl) support Nigerian developers?

Accepted Answer

Yes — Crawlbase Web Scraping (formerly ProxyCrawl) explicitly supports Nigerian developers/use cases.

Crawlbase Web Scraping (formerly ProxyCrawl)

Features Included

Not Supported

Key Highlights

Required Access Documents

Use Cases

Ask question or give feedback

Sign in to participate

Discover similar API's

Ask question or give feedback

Sign in to participate