Icon

JKISeason3-12_​tomljh

Getting Brewery Data

Level: Medium

Description: Did you know that August 2nd is International Beer Day? To celebrate one of the world’s oldest beverages, you are tasked with retrieving all worldwide brewery data from the OpenBreweryDB REST API, which provides data on the breweries in paginated responses. Build a loop that makes successive requests to each page of the API until all data is retrieved.

Not interested in beer? Feel free to use any API of your choice! Find more public APIs here: https://publicapis.io/

Author: Lada Rudnitckaia

Method 1: Use a loop to submit requests sequentiallyNote: Common methods, because most websites have restrictions. Method 2: Do not use loops and submit requests with high concurrency.Note: This requires the server to support high concurrent responses. Get MetadataCalculate the total number of pagesceil($total$/$per_page$)totalstr -> intThe total number of pages is the number of loop iterationsThe current page numberurlObtain the data of the current pageCheck the collected data. Most pages have 200 rows, and the last page has less than or equal to 200 rows.per_page:Change the default value of 50 to 200 to increase the execution speed.totalstr -> intGet Metadataper_page:Change the default value of 50 to 200 to increase the execution speed.Calculate the total number of pagesceil($total$/$per_page$)Generate rows in the table based on the number of pagesThe current page numberGet all urlObtain the data of the current pageConcurrency = 50 GET Request JSON Path Ungroup JSON to Table Table Rowto Variable Math Formula String to Number Counting Loop Start Math Formula(Variable) String Manipulation(Variable) GET Request Loop End GroupBy ConstantValue Column String to Number GET Request ConstantValue Column JSON to Table Math Formula One Row to Many Math Formula String Manipulation GET Request JSON Path Ungroup Column Filter Method 1: Use a loop to submit requests sequentiallyNote: Common methods, because most websites have restrictions. Method 2: Do not use loops and submit requests with high concurrency.Note: This requires the server to support high concurrent responses. Get MetadataCalculate the total number of pagesceil($total$/$per_page$)totalstr -> intThe total number of pages is the number of loop iterationsThe current page numberurlObtain the data of the current pageCheck the collected data. Most pages have 200 rows, and the last page has less than or equal to 200 rows.per_page:Change the default value of 50 to 200 to increase the execution speed.totalstr -> intGet Metadataper_page:Change the default value of 50 to 200 to increase the execution speed.Calculate the total number of pagesceil($total$/$per_page$)Generate rows in the table based on the number of pagesThe current page numberGet all urlObtain the data of the current pageConcurrency = 50GET Request JSON Path Ungroup JSON to Table Table Rowto Variable Math Formula String to Number Counting Loop Start Math Formula(Variable) String Manipulation(Variable) GET Request Loop End GroupBy ConstantValue Column String to Number GET Request ConstantValue Column JSON to Table Math Formula One Row to Many Math Formula String Manipulation GET Request JSON Path Ungroup Column Filter

Nodes

Extensions

Links

Download

To use this workflow in KNIME, download it from the below URL and open it in KNIME:

Download Workflow
Created by: Admin
Created at: 2024-07-31
On NodePit since: 2024-08-01
Last update: 2024-09-16
Created with KNIME version: v5.3.0
Tags: JKISeason3-12

Deploy, schedule, execute, and monitor your KNIME workflows locally, in the cloud or on-premises – with our brand new NodePit Runner.

Try NodePit Runner!