Loading page
This workflow pulls news articles from NewsAPI, Mediastack, and CurrentsAPI on a scheduled basis. Each provider’s results are normalized into a consistent schema, then written into your database (NocoDB by default). Use case: automated agg…
Download this n8n workflow template and start using it instantly.
This workflow pulls news articles from NewsAPI, Mediastack, and CurrentsAPI on a scheduled basis.
Each provider’s results are normalized into a consistent schema, then written into your database (NocoDB by default).
Use case: automated aggregation of categorized news for content pipelines, research agents, or editorial queues.
Replace all placeholder keys:
call newsapi.org - Top Headlines → update API_KEY in URLcall newsapi.org - categories → update API_KEYcall mediastack → update "ACCESS_KEY" in JSONcall currentsapi → update "API_KEY" paramWorkflow uses NocoDB to store results.
You must:
create operationsIf you prefer Google Sheets, Airtable, or another DB:
All schedulers are disabled by default.
Enable the following so the workflow runs automatically:
You may change the run times, but all four must be scheduled for the workflow to function as designed.
Defined in:
newsapi.org categoriesmediastack categoriesEdit these arrays to pull only the categories you care about or to match your API plan limits.
Adjust article_limit in:
newsapi.org categoriesmediastack categoriescurrentsapi configSplit Out 1 (Split Out): Splits an array field into individual items so each can be processed separately.
Split Out 2 (Split Out): Splits an array field into individual items so each can be processed separately.
Split Out 3 (Split Out): Splits an array field into individual items so each can be processed separately.
Set Mediastack (Set): Sets or transforms workflow data — prepares, renames, or injects fields before the next step.
Add Mediastack item (Noco DB): Reads or writes records in a NocoDB database.
Add NewsAPI item (Noco DB): Reads or writes records in a NocoDB database.
call mediastack (HTTP Request): Makes an external API call (HTTP request) to send or retrieve data from a third-party service.
call currentsapi (HTTP Request): Makes an external API call (HTTP request) to send or retrieve data from a third-party service.
currentsapi config (Set): Sets or transforms workflow data — prepares, renames, or injects fields before the next step.
MediaStack (Schedule Trigger): Starts the workflow automatically on a schedule (cron / fixed interval).
CurrentsAPI (Schedule Trigger): Starts the workflow automatically on a schedule (cron / fixed interval).
call newsapi.org - Top Headlines (HTTP Request): Makes an external API call (HTTP request) to send or retrieve data from a third-party service.
Set CurrentsAPI (Set): Sets or transforms workflow data — prepares, renames, or injects fields before the next step.
NewsAPI - Top Headlines (Schedule Trigger): Starts the workflow automatically on a schedule (cron / fixed interval).
Split Out (Split Out): Splits an array field into individual items so each can be processed separately.
Add NewsAPI item1 (Noco DB): Reads or writes records in a NocoDB database.
mediastack categories (Set): Sets or transforms workflow data — prepares, renames, or injects fields before the next step.
newsapi.org categories (Set): Sets or transforms workflow data — prepares, renames, or injects fields before the next step.
Itemize Newsapi Categories (Code): Runs custom JavaScript to transform or manipulate the data flowing through the workflow.
Itemize Mediastack Categories (Code): Runs custom JavaScript to transform or manipulate the data flowing through the workflow.
call newsapi.org - categories (HTTP Request): Makes an external API call (HTTP request) to send or retrieve data from a third-party service.
NewsAPI - Categories (Schedule Trigger): Starts the workflow automatically on a schedule (cron / fixed interval).
Wait (Wait): Pauses the workflow for a set time or until a condition is met (used for polling / async jobs).
Wait1 (Wait): Pauses the workflow for a set time or until a condition is met (used for polling / async jobs).
Loop Over NewsAPI (Split In Batches): Loops over items in batches, processing each one in turn (used for iteration).
Loop Over Mediastack (Split In Batches): Loops over items in batches, processing each one in turn (used for iteration).
Set Newsapi.org top headlines (Set): Sets or transforms workflow data — prepares, renames, or injects fields before the next step.
Set Newsapi.org categories (Set): Sets or transforms workflow data — prepares, renames, or injects fields before the next step.
Add CurrentsAPI item (Noco DB): Reads or writes records in a NocoDB database.
This workflow contains community nodes that are only compatible with the self-hosted version of n8n. This workflow automatically monitors keyword rankings across search engines to track SEO performance and identify optimization opportunities. It saves you time by eliminating the need to manually check keyword positions and provides comprehensive ranking data for strategic SEO decision making. Overview This workflow automatically scrapes search engine results pages (SERPs) to track keyword rankings, competitor positions, and search features. It uses Bright Data to access search results without restrictions and AI to intelligently parse ranking data, track changes, and identify SEO opportunities. Tools Used n8n: The automation platform that orchestrates the workflow Bright Data: For scraping search engine results without being blocked OpenAI: AI agent for intelligent ranking analysis and SEO insights Google Sheets: For storing keyword ranking data and tracking changes How to Install 1. Import the Workflow: Download the .json file and import it into your n8n instance 2. Configure Bright Data: Add your Bright Data credentials to the MCP Client node 3. Set Up OpenAI: Configure your OpenAI API credentials 4. Configure Google Sheets: Connect your Google Sheets account and set up your ranking tracking spreadsheet 5. Customize: Define target keywords and ranking monitoring parameters Use Cases SEO Teams: Track keyword performance and identify ranking improvements Content Marketing: Monitor content ranking success and optimization needs Competitive Analysis: Track competitor keyword rankings and strategies Digital Marketing: Measure organic search performance and ROI Connect with Me Website: https://www.nofluff.online YouTube: https://www.youtube.com/@YaronBeen/videos LinkedIn: https://www.linkedin.com/in/yaronbeen/ Get Bright Data: https://get.brightdata.com/1tndi4600b25 (Using this link supports my free workflows with a small commission) #n8n #automation #keywordrankings #seo #searchrankings #brightdata #webscraping #seotools #n8nworkflow #workflow #nocode #ranktracking #keywordmonitoring #seoautomation #searchmarketing #organicseo #seoresearch #rankinganalysis #keywordanalysis #searchengines #seomonitoring #digitalmarketing #serp #keywordtracking #seoanalytics #searchoptimization #rankingreports #keywordresearch #seoinsights #searchperformance
by Yaron BeenDescription This workflow automates the process of scraping the latest discussions from HackerNews, transforming raw threads into human readable content using Google Gemini, and exporting the final content into a well-formatted Google Doc. Overview This n8n workflow is responsible for extracting trending posts from the HackerNews API. It loops through each item, performs HTTP data extraction, utilizes Google Gemini to generate human-readable insights, and then exports the enriched content into Google Docs for distribution, archiving, or content creation. Who this workflow is for Tech Newsletter Writers: Automate the collection and summarization of trending HackerNews posts for inclusion in weekly or daily newsletters. Content Creators & Bloggers: Quickly generate structured summaries and insights from HackerNews threads to use as inspiration or supporting content for blog posts, videos, or social media. Startup Founders & Product Builders: Monitor HackerNews for discussions relevant to your niche or competitors, and keep a pulse on the community’s opinions. Investors & Analysts: Surface early signals from the tech ecosystem by identifying what’s trending and how the community is reacting. Researchers & Students: Analyze popular discussions and emerging trends in technology, programming, and startups—enriched with AI-generated insights. Digital Agencies & Consultants: Offer HackerNews monitoring and insight reports as a value-added service to clients interested in the tech space. Tools Used n8n: The core automation engine that manages the trigger, transformation, and export. HackerNews API: Provides access to trending or new HN posts. Google Gemini: Enriches HackerNews content with structured insights and human-like summaries. Google Docs: Automatically creates and updates a document with the enriched content, ready for sharing or publishing. How to Install Import the Workflow: Download the .json file and import it into your n8n instance. Set Up HackerNews Source: Choose whether to use the HN API (via HTTP Request node) or RSS Feed node. Configure Gemini API: Add your Google Gemini API key and design the prompt to extract pros/cons, key themes, or insights. Set Up Google Docs Integration: Connect your Google account and configure the Google Docs node to create/update a document. Test and Deploy: Run a test job to ensure data flows correctly and outputs are formatted as expected. Use Cases Tech Newsletter Authors: Generate ready-to-use summaries of trending HackerNews threads. Startup Founders: Stay informed on key discussions, product launches, and community feedback. Investors & Analysts: Spot early trends, technical insights, and startup momentum directly from HN. Researchers: Track community reactions to new technologies or frameworks. Content Creators: Use the enriched data to spark blog posts, YouTube scripts, or LinkedIn updates. Connect with Me Email: ranjancse@gmail.com LinkedIn: https://www.linkedin.com/in/ranjan-dailata/ Get Bright Data: Bright Data (Supports free workflows with a small commission) #n8n #automation #hackernews #contentcuration #aiwriting #geminiapi #googlegemini #techtrends #newsletterautomation #googleworkspace #rssautomation #nocode #structureddata #webscraping #contentautomation #hninsights #aiworkflow #googleintegration #webmonitoring #hnnews #aiassistant #gdocs #automationtools #gptlike #geminiwriter
by Ranjan DailataThis workflow contains community nodes that are only compatible with the self-hosted version of n8n. This workflow automatically monitors competitor pricing changes and website updates to keep you informed of market movements. It saves you time by eliminating the need to manually check competitor websites and provides alerts only when actual changes occur, preventing information overload. Overview This workflow automatically scrapes competitor pricing pages (like ClickUp) and compares current pricing with previously stored data. It uses Bright Data to access competitor websites without being blocked and AI to intelligently extract pricing information, updating your tracking spreadsheet only when changes are detected. Tools Used n8n: The automation platform that orchestrates the workflow Bright Data: For scraping competitor websites without being blocked OpenAI: AI agent for intelligent pricing data extraction and parsing Google Sheets: For storing and comparing historical pricing data How to Install 1. Import the Workflow: Download the .json file and import it into your n8n instance 2. Configure Bright Data: Add your Bright Data credentials to the MCP Client node 3. Set Up OpenAI: Configure your OpenAI API credentials 4. Configure Google Sheets: Connect your Google Sheets account and set up your pricing tracking spreadsheet 5. Customize: Set your competitor URLs and pricing monitoring schedule Use Cases Product Teams: Monitor competitor feature and pricing changes for strategic planning Sales Teams: Stay informed of competitor pricing to adjust sales strategies Marketing Teams: Track competitor messaging and positioning changes Business Intelligence: Build comprehensive competitor analysis databases Connect with Me Website: https://www.nofluff.online YouTube: https://www.youtube.com/@YaronBeen/videos LinkedIn: https://www.linkedin.com/in/yaronbeen/ Get Bright Data: https://get.brightdata.com/1tndi4600b25 (Using this link supports my free workflows with a small commission) #n8n #automation #competitoranalysis #pricingmonitoring #brightdata #webscraping #competitortracking #marketintelligence #n8nworkflow #workflow #nocode #pricetracking #businessintelligence #competitiveanalysis #marketresearch #competitormonitoring #pricingdata #websitemonitoring #competitorpricing #marketanalysis #competitorwatch #pricingalerts #businessautomation #competitorinsights #markettrends #pricingchanges #competitorupdates #strategicanalysis #marketposition #competitiveintelligence
by Yaron BeenTurn any prompt into structured web data. Send a POST request with a natural language prompt and an optional JSON schema, and get back clean, structured results scraped from the web by an AI agent powered by Firecrawl. Use Cases Data Enrichment: Feed company names or URLs from your CRM and get back structured firmographic data (industry, funding, team size, tech stack). Lead Generation: Ask the agent to find pricing, contact pages, or product details for a list of competitors. Market Research: Extract structured pricing plans, feature comparisons, or product catalogs from any website. Content Aggregation: Pull structured news, events, or job postings from across the web on a schedule. Sales Intelligence: Enrich prospect lists with company info, recent news, or tech stack details before outreach. How It Works `` POST /webhook/scrape-agent ` 1. Receive Scrape Request receives a POST request with prompt and an optional output_schema. 2. Validate Output Schema checks the schema. If none is provided, it falls back to a permissive default. If the schema is malformed, it returns a clear error via Return Schema Error. 3. Research & Extract Web Data takes the prompt and uses the full Firecrawl toolkit to research the web: Search (/search): Finds relevant pages and sources across the web. Scrape (/scrape): Extracts clean, structured content from any URL. Interact (interactContext, interact, interactStop): Lets the agent interact with scraped pages in a live session. After scraping a page, the agent can click buttons, fill forms, navigate dynamic content, and extract data that static scraping cannot reach, all without managing sessions manually. This combination gives the AI agent complete web navigation capabilities. It can discover sources, read pages, and interact with dynamic content autonomously. 4. Format Response to Schema (Structured Output Parser) formats the agent's response to match the provided (or default) schema. 5. Return Structured Results sends the structured JSON back to the caller. Setup Requirements Firecrawl API Key: Sign up at firecrawl.dev and grab your API key. Connect it in the Firecrawl credential nodes. LLM Provider: Configure your Primary Chat Model and Fallback Chat Model nodes (e.g., OpenRouter, OpenAI, Anthropic). The template uses two model nodes for reliability, plus a separate Parser Chat Model for the output parser. n8n Instance: Self-hosted or cloud. Make sure the webhook node is set to accept POST requests. API Reference Endpoint ` POST https://your-n8n-instance/webhook/scrape-agent ` Request Body | Field | Type | Required | Description | |-------|------|----------|-------------| | prompt | string | Yes | Natural language instruction for the agent | | output_schema | object | No | JSON Schema defining the desired output structure | Response Returns a JSON object matching the provided schema, or a flexible object if no schema was given. --- Testing Examples 1. Basic Request (No Schema) The agent decides the output structure on its own. `bash curl -X POST "https://your-n8n-instance/webhook/scrape-agent" \ -H "Content-Type: application/json" \ -d '{ "prompt": "Find the latest pricing for Firecrawl" }' | jq ` Expected output: A JSON object with whatever structure the agent finds most appropriate for the data. Since no schema was provided, the internal default ({ "type": "object", "additionalProperties": true }) is used. 2. Request With a Custom Schema You define exactly the shape of data you want back. `bash curl -X POST "https://your-n8n-instance/webhook/scrape-agent" \ -H "Content-Type: application/json" \ -d '{ "prompt": "Find the latest pricing for Firecrawl", "output_schema": { "type": "object", "properties": { "source": { "type": "string" }, "plans": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "price": { "type": "string" }, "credits": { "type": "string" }, "highlights": { "type": "array", "items": { "type": "string" } } } } } } } }' | jq ` Expected output: `json { "output": { "source": "https://www.firecrawl.dev/pricing", "plans": [ { "name": "Free", "price": "$0 (one-time)", "credits": "500 credits (one-time)", "highlights": [ "Scrape up to 500 pages", "2 concurrent requests", "Low rate limits", "No credit card required" ] }, { "name": "Hobby", "price": "$16/month (billed yearly, save $38)", "credits": "3,000 credits / month", "highlights": [ "Scrape up to 3,000 pages", "5 concurrent requests", "Basic support", "$9 per extra 1k credits" ] } ] } } ` 3. Invalid Schema (String Instead of Object) `bash curl -X POST "https://your-n8n-instance/webhook/scrape-agent" \ -H "Content-Type: application/json" \ -d '{ "prompt": "Find the latest pricing for Firecrawl", "output_schema": "not a valid schema" }' | jq ` Expected output: `json { "error": true, "message": "Invalid output_schema: must be a JSON object with a valid 'type' property (object, array, string, number, boolean)", "example_schema": { "type": "object", "properties": { "name": { "type": "string" }, "price": { "type": "number" } } } } ` 4. Invalid Schema (Array Instead of Object) `bash curl -X POST "https://your-n8n-instance/webhook/scrape-agent" \ -H "Content-Type: application/json" \ -d '{ "prompt": "Find the latest pricing for Firecrawl", "output_schema": [1, 2, 3] }' | jq ` Expected output: Same error response as above. 5. Invalid Schema (Missing type Property) `bash curl -X POST "https://your-n8n-instance/webhook/scrape-agent" \ -H "Content-Type: application/json" \ -d '{ "prompt": "Find the latest pricing for Firecrawl", "output_schema": { "properties": { "name": { "type": "string" } } } }' | jq ` Expected output: Same error response as above. 6. Invalid Schema (Invalid type Value) `bash curl -X POST "https://your-n8n-instance/webhook/scrape-agent" \ -H "Content-Type: application/json" \ -d '{ "prompt": "Find the latest pricing for Firecrawl", "output_schema": { "type": "banana" } }' | jq ` Expected output: Same error response as above. --- Workflow Architecture ` Receive Scrape Request (POST) | v Validate Output Schema |--- Error --> Return Schema Error (error JSON) |--- Success --> Research & Extract Web Data (AI Agent) | |--- Primary Chat Model |--- Fallback Chat Model |--- Search & Scrape: | - /search with Firecrawl | - /scrape with Firecrawl |--- Interact Tool: | - Interact context with Firecrawl | - Execute interaction with Firecrawl | - Stop interaction with Firecrawl | v Return Structured Results | |--- Format Response to Schema (Output Parser) | |--- Parser Chat Model ` Schema Validation Logic The Validate Output Schema node runs this validation before passing data to the agent: If output_schema is missing or null, the default permissive schema is used: { "type": "object", "additionalProperties": true }. If output_schema is present, it must be a JSON object (not a string, array, or primitive). It must have a type property with a valid value: object, array, string, number, or boolean. If validation fails, the workflow returns an error response with a helpful message and example schema. Notes The Format Response to Schema node (Structured Output Parser) requires the schema to be passed as a JSON string. The expression {{ JSON.stringify($('Validate Output Schema').item.json.output_schema) }}` handles this conversion. The agent has access to Firecrawl's full toolkit: search, scrape, and interact. With all three connected, the agent has complete web navigation powers. It can discover sources via search, extract content via scrape, and interact with dynamic JavaScript-heavy pages via interact. The interact tools let the agent scrape a page first and then continue working with it in a live session, clicking buttons, filling forms, and navigating deeper, all without manual session management. The agent autonomously decides which tools to use based on the prompt. Response times vary depending on the complexity of the prompt and how many pages the agent needs to visit. Simple lookups take a few seconds; deep research can take longer.
by Firecrawl