Go Web Scraper API Reference

The Go Web Scraper provides a RESTful API for accessing and manipulating scraped data programmatically. This documentation covers all the available endpoints and how to use them.

Base URL

When running locally:

http://localhost:8080/api

Authentication

Currently, the API doesn't require authentication as it's designed for local use.

Endpoints

Articles

Method	Endpoint	Description
GET	/api/articles	List all articles with pagination
GET	/api/articles/:id	Get a specific article by ID
GET	/api/articles/search	Search articles by keywords

List Articles

GET /api/articles?page=1&limit=20

Query Parameters:

• page: Page number (default: 1)
• limit: Number of items per page (default: 20)
• sort: Sort field (options: 'date', 'title', 'score')
• order: Sort order (options: 'asc', 'desc')

Response:

{
  "data": [
    {
      "id": 1,
      "title": "Introducing Go 2.0",
      "url": "https://example.com/go-2-release",
      "score": 120,
      "comments": 45,
      "author": "gopher",
      "date": "2025-03-15T14:22:18Z"
    },
    {
      "id": 2,
      "title": "Go Concurrency Patterns",
      "url": "https://example.com/go-concurrency",
      "score": 98,
      "comments": 32,
      "author": "rob_pike",
      "date": "2025-03-14T09:45:10Z"
    }
  ],
  "meta": {
    "current_page": 1,
    "per_page": 20,
    "total_items": 235,
    "total_pages": 12
  }
}

Get Article by ID

GET /api/articles/42

Response:

{
  "id": 42,
  "title": "Understanding Go Interfaces",
  "url": "https://example.com/go-interfaces",
  "score": 156,
  "comments": 67,
  "author": "go_enthusiast",
  "date": "2025-03-10T11:32:45Z",
  "content": "Go interfaces are a powerful feature that enables..."
}

Search Articles

GET /api/articles/search?q=golang+concurrency

Query Parameters:

• q: Search query (required)
• page: Page number (default: 1)
• limit: Number of items per page (default: 20)

Response: Same format as List Articles

Products

Method	Endpoint	Description
GET	/api/products	List all products with pagination
GET	/api/products/:id	Get a specific product by ID
GET	/api/products/search	Search products by keywords

Similar patterns apply for other product endpoints.

Export

Method	Endpoint	Description
GET	/api/export/articles	Export all articles as JSON or CSV
GET	/api/export/products	Export all products as JSON or CSV

Export Data

GET /api/export/articles?format=json

Query Parameters:

• format: Output format (options: 'json', 'csv', default: 'json')

Error Handling

All API errors return an appropriate HTTP status code and a JSON error response:

{
  "error": {
    "code": "not_found",
    "message": "Article with ID 999 not found"
  }
}

Common Error Codes

HTTP Status	Error Code	Description
400	invalid_request	The request is malformed or missing required parameters
404	not_found	The requested resource was not found
422	validation_error	Validation errors in the request parameters
500	internal_error	An unexpected error occurred on the server