This tool helps you to build queries for the Netherlands Biodiversity API (NBA), execute them, and store them for re-use. It is not a true query-builder, but rather a collection of functions that makes it easier to understand what a query should look like, helps you construct and test them, and examine their resulting output. Before you start, it might be good to get acquainted with the NBA:
If you want to use the NBA directly, it can be accessed at
.
You can click and drag the header to move this window to a position of your liking. Click the 'X' top right to close. Click and hold marked items to highlight the corresponding item in the main screen.
Service list
List of available endpoints in the API. Endpoints are grouped by document type: specimen, taxon, multimedia, geo or other. For most endpoints, a short description of the service's function is shown when you hover with the mouse over the item in the list. Some items additionally display one or more icons:
- 👁: service accepts 'simple' type queries (using only querystring-parameters)
- –: service accepts no query
- ⁉: service has explain-option
'Simple type' queries have the form
parameter=value
(for instance:
collectionType=Aves
), whereas regular queries have a more complex form. There are templates available to help you on your way with more complex queries; see the section 'Inserting query-elements' below.
Explainable services can be called with an added 'explain' parameter that changes the regular output to a breakdown of the steps that lead to that output. Check
add explain to enable.
Server link: link to the root of the NBA server.
Query window
Window for writing the query you want to run. This can be either a JSON-object (for querySpec-queries) or a query-string (for human readable queries). Specify just the actual query, not the rest of the URL (or the '?') - these are inferred from the values selected in the service list. When you first open this page, two example queries will be automatically added to the list of saved queries to give you an idea what they look like.
Queries can be spread across multiple lines (for readability). Empty lines will be ignored, and lines that start with one of these character(s): will be treated as comment and ignored.
Execute query
Click the execute query-button to run your query. When you click the button, the program will, in short:
- Take the selected entry from the service list;
- If the service requires a query, take the contents of the query window, and encode them if required. If the selected service does not require a query, the contents of the query window are ignored;
- Put them all together to turn it into a NBA-query;
- Print the the complete resulting URL;
- Execute the query and print the formatted result in the result window.
Notes:
- The program will not check the validity of your query. If your query is of the complex type, the program will check whether it is valid JSON, but it does not perform any checks beyond that; these are left to the NBA itself. If errors occur there, these are displayed as output.
- The program does not check whether your query actually makes sense with the selected service. If for instance you combine a JSON-query with a human readable service, the program will execute the query all the same, which will generate an error as output.
- The URL printed in the resulting URL window is not literally what is executed. The http method used by the program for excution depends on the methods that the selected services accepts. Most services accept both GET and POST, in which case the program defaults to POST. In either case, the endpoint of the selected service and the query are posted back to a proxy server, which constructs a request and performs the actual execution. You can force the program to use a specific method by selecting a specific http request method in the additional options panel.
Query window manipulation
Right-click in the query window to see the edit, operators and queries menus. Hover over the menu title to access each menu's options.
Select and click an item from queries to insert it into the query window at the cursor position, or over selected text.
Inserting operators works the same way.
Type the (partial) name of a field in the type to find field name-input to find a field. As you type, a droplist will appear displaying all fields matching your search term (type a space to see all available fields). Only fields that match the document type of the selected service will be displayed. Click an item to insert it into the query window at the cursor position, or over the selected text. The droplist items include the name of the document type, as well as the fields' datatype and analyzers; these won't be inserted into the query window).
Fields are matched with each part of your search string, which is split by spaces. For instance, searching for ers gath name will match every line that has the strings 'ers', 'gath' and 'name' anywhere in their name, such as "gatheringEvent.gatheringPersons.fullName".
By default, datatypes and operators are excluded from the search process; by checking include datatypes and analyzers in search you can include them in your search (for instance if you are looking for a field of a specific datatype). Check show fields for all document types to see results for all document types, not just the one corresponding with the selected service.
Use decode query to decode the contents of the query window, useful when you want to investigate a query but only have the full url.
To copy the query to the clipboard, click copy query. Double clicking does the same and additionally adds {code}
tags around the copied content for readable pasting into JIRA.
Click copy query + to copy all query elements to the clipboard. (Note: holding down Ctrl while clicking any of the copy-links will additionally log the copied content to the browsers's console).
To clear the query window, click clear query.
If the query window has focus, use Ctrl + F11 to insert a comment char in query-window at the cursor; also works on multiple selected lines (adds them at the beginning of each line), and Ctrl + F12 to enclose selected text in the query-window in double quotes; also works with multiple words (detects word boundaries).
Resulting URL & run altered query
The resulting URL-window shows the resulting URL that the application sends to the NBA. You can directly edit the URL and execute the result by clicking run altered query (button becomes available once you've actually changed the URL). This is intended for quick debugging purposes; please note that your edits will be lost the next time you run execute query.
Query run options (available once you have executed a query)
- open directly: opens the last executed URL directly in a new browser window.
- full screen: show the results window full screen. Press Escape to close again.
- copy URL: copies the last executed URL to the windows clipboard.
- copy Scratchpad URL + query: copy a shareable link to Scratchpad including query and selected service.
- clear results: clear the result window.
- copy results: copy the results (can be limited by size; use download for full set). Follows the current result view (JSON or TSV)
- download results: download the results. Follows the current result view (JSON or TSV)
- switch result view: switch between JSON and table view.
- rotate table: rotate the results table (table view only).
result window
Results of your query from the NBA-server.
You can switch view between JSON and tabular data.
Saved queries are stored in the local storage of your browser. This means they will be available across browser sessions, but are subject to whatever the browser does to its own Local Storage. Emptying your cache will most likely not cause any problems, nor will upgrading your browser. However, re-installing or performing a factory reset or 'fresh up' of your browser will probably cause your saved queries to be deleted. There are, however, several options for exporting and backing up stored queries.
Saved queries will not be available when running a private or incognito session, but they will still be there when you start a regular session. Saved queries are not available across different types of browsers, but you can export them from one end import in another.
Query name to save
Enter the name under which you want to save your query. Max. Length is 64 characters. You can clear the input by clicking the clear query name-symbol.
Prefix your query with a name between straight bracket to create a group of queries. Creating queries [TEST] query a
and [TEST] query b
will create a group called 'TEST' containing queries 'query a' and 'query b'. You can collapse groups in the query list, which helps maintain overview.
Save query
Click save query to save the content of the query window with the query name to save. The application will also save a reference to the service that is selected in the service list at the moment you save the query. If, for whatever reason, you don't want the service to be saved as well, uncheck the save including service checkbox before you click save.
The query will be saved in your local browser storage, and will remain available across sessions. If you choose a name that already exists, the application will overwrite the existing query without warning.
The application will show a message stating the result of the save action, above the query name-input (see what a message looks like). It will also automatically update the list of saved queries.
please note, the application only saves the query and the service, not the server. So if you load a saved query, you'll have to manually select the appropriate server to run it against.
Saved queries
List of saved queries. Legend to the various symbols:
⟵ |
Load the saved query into the query window and set the saved service in the service list. Hover over the symbol to see the name of the saved service. Click while holding down Ctrl to cause the query to be executed after loading.
|
x |
Delete the query (asks confirmation). |
📋 |
Copy the query name to the query name to save input. Handy when you want to overwrite an existing saved query. Clicking one next to a group copies that group's name to the input, so you can more easily save a query as part of that group. |
≡ |
Open query notes (see below). |
≛ |
Same, but query already has some saved notes (hint indicates note length). |
⧉ |
Indicates a group of queries. Click to expand. The number indicates the number of queries in the group. |
When you have no saved queries, the application automatically creates a set of example queries.
Query notes
The application allows you to store notes with a query. Click on one of the ≡-symbols to open a popup with the notes for that query. You can have only one set of notes open at a given time. Clicking a second ≡-symbol while notes are already open will automatically save before opening the next set of notes. Notes are deleted automatically when you delete the corresponding query.
The note title is the same as the query the note belongs to. The number after the title indicates the size of the note (not sure if there's a maximum).
Type your text in the notes input area (which opens with existing notes if there are any) and click save notes to save the notes. To delete, clear the entire input and click save.
Click insert timestamp to insert the current date and time at the cursor position. Click insert line to insert the current date and time at the cursor position. Click close note to close the notes popup.
If there's a valid URL in the notes, you can select it to make a clickable link-icon ↗ appear in the notes window's header (obviously an actual hyperlink would be more practical, but the notes are in a textarea, which doesn't render html).
Query export
Click export all queries to have scratchpad automatically download a backup of all your queries to the browser's downloads-folder. See below for automated backups.
Additional options
Click the additional options-icons (▼ ▲) to toggle the maintenance options.
Use query backup file import to import queries from a previous backup file. By checking enable automatic monthly query-backup, the program will automatically back up your saved queries to the browser's downloads-folder (note: works only if you actually use the scratchpad once in a while).
Use export selected queries to export selected queries as a JSON-string. When you open additional options, checkboxes (☐ ☑) appear for each item in the saved queries list. Use the select all and select none links to change the selection in bulk. Select the queries you want to export, and click export selected queries. The corresponding JSON-string is written to the query window (notes, if any, are included. Services are not). Select and copy the JSON-string from the window and save it as backup in a regular txt-file, or share it with other users. Paste an exported JSON-string into the query window and click import queries to import the queries. All queries are imported; if another query with the same name already exist, a number is added to the name of the newly imported query.
Use delete selected queries in combination with the checkboxes in the saved queries list to delete queries in bulk (after confirmation).
Click query timer to bring up the query execution timer. The timer counts the delay between the moment the full query is launched, and the moment the results come back (more precise, between the moment the src-attribute of the result window is set, and the window's onload-event is triggered). The timer also keeps an average if you execute the same query repeatedly.
Click the output hashes link to bring up a list of output hashes. Each time you run a query, a hash of the resulting output is printed, providing an easy way to quickly see whether the results of two subsequent queries differ. The hashes themselves have no meaning.
Toggle enable Ctrl+s to enable the use of ctrl+s for saving of the current query (content of the query window), using the name of the currently highlighted saved query (= the last one you loaded or saved). Be aware, the program overwrites the stored query without warning, and there's no undo. Functionality is turned off by default.
- F2: open help-window (and close it again).
- Esc: close help- and links-windows.
- Ctrl+enter: run query.
- Ctrl+space: insert 2 spaces at cursor in query window.
- Ctrl+F7: clear query.
- Ctrl+F8: set focus on query.
- Ctrl+F9: set focus on type to find field name.
- Ctrl+F11: insert comment char at the beginning of the line in the query-window (works on multiple selected lines).
- Ctrl+Shift+F11: remove comment char from begin of selected lines in the query-window.
- Ctrl+F12: encloses selected text in query-window in double quotes; works with multiple words.
- Alt+Del: clear the result window.
- Ctrl+clicking copy query: dump info to browser console as well (works for all 'copy to clipboard' commands).
- Ctrl+clicking ⟵: immediately run query after loading.
- Ctrl+clicking add slashes: also removes newlines & multiple spaces.
- Shift+clicking add slashes: removes slashes from \".
- Ctrl+clicking datamodel entry: encloses value in double quotes when inserting.
- Ctrl+s: save current query with highlighted query-name (disabled by default).
- Shift+📋: also copy query(group) name to clipboard while copying it to the new name input.
- AltGr+F9: inserts the "size 100 (stand alone clause)" code bit in the query window.
- AltGr+F10: inserts basic query template in the query window.
When service select list is active, press a key to:
- s: jump to the first entry of the specimen-services in the service list.
- t: idem for the taxon services.
- m: idem for the multimedia services.
- g: idem for the geo services.
- d: idem for the metadata services.
- y: idem for the system services.
- v: idem for the static services.
This tool comes "as is". It was originally developed using Firefox 54.x, later in Chromium. This is version ().
-
v3.17 (2022.04.11)
Added monospace to CSS.
Added share-button for copying link to Scratchpad including query.
Ctrl+clicking once again adds quotes to inserted field names.
Changed keyboard service select to pressing keys (s,t,g,m,d,y,v) when service select list is active, rather than AltGr-combinations.
Error message has max width plus hidden overflow to avoid displacement.
Longitude and latitude are clickable, opening Google Maps with the corresponding coordinates. When only latitude or only longitude is present, the missing coordinate is assumed to be zero.
Ctrl+double clicking a key of a document in the resultset copies that key and its value(s) as a JSON-object to the clipboard.
-
v3.16.2 (2021.07.22)
Fixed payload conversion bug in proxy (NBAX-532, 2nd part)
-
v3.16.1 (2021.07.21)
Fixed double-nested sortFields bug (NBAX-531).
Added insert basic query template by pressing AltGr+F10.
-
v3.16 (2021.06.23)
"Stuck Ctrl-key" now automatically released when switching (blurring) windows, and when initial load completes (same for Shift and AltGr).
All types of errors are now displayed formatted correctly.
Full name of saved queries is now clickable (finally).
Fixed last entry check of session history (caused consecutive identical entries).
Fixed 'cut not working' bug.
Size indicator now shows the number of elements in the document for results with a single level (like those from a distinct query).
Max. columns in tabular view set to 100 (from 50), which allows most results to fit in their entirety without serious impact on performance; header interval now set to 25 (from 50).
http-links in JSON-output now also clickable (some left in CoL).
http(s)-links in tabular output clickable.
-
v3.15 (2021.04.30)
Session query history, remembers last 100 queries from session (link next to 'saved queries') (NBAX-425).
Query maintenance page for reordering and renaming queries and query groups (link in 'more options').
Links in output now clickable (JSON only).
"Hanging" Ctrl-key workaround (automatically resets to "up" after 10 seconds).
Resizing query window automatically resizes result window to fit screen.
Query window content is now trimmed before evaluated as empty for default query.
Clipboard copy of rotated output is now also rotated (bug fix).
countDistinct*-endpoints accept _querySpec (bug fix).
-
v3.14 (2021.03.31)
Displayed JSON-result is capped on 1MB, number of records rounded down to the nearest multiple of 10; table-view displays the same records; downloads are unaffected (NBAX-495).
'New window'-switch looks at what the service produces (zip & x-ndjson) (NBAX-494).
Single-level arrays are displayed correctly in table-view (NBAX-493).
Added full screen results option (⛶; Esc closes).
Can rotate tabular data (NBAX-492).
Services class correctly handles 'produces'-array (NBAX-444).
-
v3.13 (2021.03.04)
Added tabular data and .tsv-download.
Fixed non-updating record total in 'next batch'-function.
Log info form explain-query now appears in a dialog, JSON-result is treated like proper data.
Refactored and sanitized all libraries, implementing classes and namespaces.
Added User-Agent header to proxy.
Took out option of saving queries without service.
"Run altered query" button is disabled until query/URL is actually altered.
Messages on JSON parse errors are clickable, positioning the cursor at the location of the error.
Reduced the list of allowed operators in field-suggestion to a more compact format.
-
v3.12 (2020.12.02)
Fixed double [*]-selector in JSONpath input (NBAX-432).
Added "content-type:application/json" for POST-requests with an empty body to avoid HTTP 415-errors (NBAX-427).
Gave total result size more emphasis (NBAX-433).
Replaced internal NBA URL in error reporting from proxy (NBAX-434).
URL-window now correctly shows follow-up token requests (NBAX-428).
Moved JSONpath input box to JSONpath dialog (NBAX-431).
Added short JSONpath explanation and examples.
Changed POST-requests in proxy to content-type 'application/json', now ignoring endpoints that consume 'application/x-www-form-urlencoded'.
Removed simple type-duplicates of nameResolution- and batchQuery-requests.
Fixed broken queries for {id}-style endpoints (NBAX-441).
Now also parsing total result size from the debug ( __explain) output.
Replaced automatic opening of requests with unhandled content-type replies (like 'application/x-ndjson') by a clickable link.
-
v3.11 (2020.11.09)
Removed PURL-requests. Fixed 'always GET'-bug (oops), plus override in the proxy. Also added possibility of choosing method in the GUI. Inferred services are tested on loading. Added 'explain' option (for services that can handle it), including choice of level. Added service summaries (as mouseover-title in the services dropdown). Added 'next batch' button for batch queries (recognizes token in result). Added result download (including custom name and auto-increment counter). Added compression on all traffic from and to the proxy.
Interface fixes: replaced iframe with object (for non-JSON responses). Added total result size (even if there's none). Tidied service list. Took out access keys (patchy browser support). Result window now fits the screen w/o scrolling. Long saved query lists can now be scrolled. Context menu (right click) in query window now has proper 'insert elements into queries'-options (rather than just pasting). Replaced 'add slashes' with 'format query' ('add slashes' is still available in context menu). Ctrl + 'format query' gives a more compact format. Added searchType-operators for nameResolutionRequest to query window context menu.
Replaced original example queries with a set of more illustrative examples.
-
v3.10 (2020.05.18)
AltGr+F9 inserts size clause code. Fixed XJSON-window top. Fixed autobackup bug. Fixed notes window positioning bug. Changed 're-use (group)name'-icon, name is also copied to console.
- v3.9 (2020.03.04)
added skip/overwrite/append-option to query import. auto cleanup of orphaned notes. added monthly auto-backup of queries. fixed "2nd attempt query-import not responding"-bug. replaced shoddy 'run query'-arrows. removed dead F4-toggle and old server-select code. updated Jira code-tags. added query list toggle. added links to help dialog.
-
v3.8 (2019.12.19)
added support for "mixed content" aggregate queries. fixed 'undefined' bug in operator list. added more informative placeholders to query-window. removed 'suppress output' option. removed groupBy-query code examples. refactored underlying container.
-
v3.7 (2019.02.12)
AltGr+F1/F2/F3/F4/F5 to jump to the first entry in the service list of the specimen-services, the taxon-services, etc.
-
v3.6 (2018.12.28)
downloads are always forced in new tab. query file import repeat bug fixed. fixed month in save file name.
-
v3.5 (2018.11.07)
added 'quick queries download' option plus import of backups via file.
-
v3.4-server (2018.09.05)
fixed declaration of 'download' services.
-
v3.3-server (2017.10.23)
added loading of external query through _querySpec parameter.
-
v3.2-server (2017.10.03)
migrated to server. stripped multi-server capabilities.
-
v3.2 (2017.09.05)
operator list is generated automatically. added XPath for JSON functions. small fixes.
-
v3.1 (2017.07.18)
added grouping in context code menu. added 'noServices' option to servers.
-
v3.0 (2017.06.14)
Added some access keys. added
custom-styles.css
option. added query execution timer. Ctrl+space inserts multiple spaces. added querywindow context menu toggle (so you can turn off the browser's spelling checker). added Ctrl+s. added Shift+add slashes. added possibility of extra static services (in config.js
), including option for non-standard port (for PURL). added different result window for cross-domain-enabled servers, incl. copying and searching in results. implemented direct server requests for all ajax services, with proxy as backup. added support for non-JSON and -html responses. added manual setting of accept headers. added GET/POST-switch. added http response code monitor. added possibility of suppressing output (useful for better query timing). added output hashes (to quicky see if output has changed). bugfixes.
-
v2.10 (2017.05.15)
removes "just brackets" query before executing a human readable service. version-info is retrieved from this table. user can (de)select query-groups. query names are htmlEncoded before being saved. allows for default list of services from file for situations in which the proxy cannot be reached.
-
v2.9 (2017.05.09)
non-indexed fields are now excluded from document field search (including toggle). only searchable document fields that correspond with the selected service are shown (can be overridden). services are retrieved automatically from the server. page-elements are temporarily disabled while loading. bugfixes.
-
v2.8 (2017.05.01)
doubleclick copy query encloses the copied query in {code} tags for JIRA. Alt+F11 removes comment chars. moved code & operator selects to query window context menu. before executing, application removes superfluous comma's from the query. query is verified as valid JSON or not executed (can be overridden in
config.js
). cursor position (line, column) is displayed below the query-window (helps debugging JSON parse errors). list of document fields, allowed operators and analyzers is now retrieved dynamically from the selected server (through a remote proxy, for now). added 'search in Google' to query window context menu.
-
v2.7 (2017.03.27)
Ctrl+F11 inserts comment char in query-window, Ctrl+F12 encloses selected text in double quotes. Ctrl+F11 now also works on multiple selected lines. Ctrl+click datamodel entry adds "'s around inserted value. moved code- & operator-insert event to option onclick, so subsequent clicks on the same option also work. made config.js file entirely optional (all vars have defaults), added noExampleQueries option. datamodel-'type as you find' now works with partial & space-split search terms. Ctrl+F9 sets focus to field search input. fixed double-quote repeating strings enclosure bug.
-
v2.6.1 (2017.03.10)
Ctrl+add slashes also removes newlines & multiple spaces.
-
v2.6 (2017.03.10)
added datamodels & field-insert. removed 'insert' links from code bits and operator selectors.
-
v2.5 (2017.03.09)
added query export, import and bulk deletion.
-
v2.4.4 (2017.03.02)
added optional target to useful links. added additional tool
parse-mapping.html
. added 'add slashes'.
-
v2.4.3 (2017.03.01)
seperated config.js and services.js.
-
v2.4.2 (2017.02.09)
added labels for links window.
-
v2.4.1 (2017.01.31)
Alt+up/down changes selected server; Alt+delete clears the result window, plus link for the same; Ctrl+enter now works globally; escape closes help-window; added decode query function.
-
v2.4 (2017.01.26)
removed index numbers from, and added templates to server & service lists; added URL-awareness in notes' text; notes automatically change when a query is selected; notes window draggable; enabled copy to clipboard for query plus parameters; various small changes.
-
v2.3.4 (2017.01.16)
fixed server status toggle bug.
-
v2.3.3 (2017.01.12)
added Ctrl+click = load & execute.
-
v2.3.2 (2017.01.09)
moved message area to avoid width-issues; small style changes.
-
v2.3.1 (2017.01.09)
fixed nested query arrow bug.
-
v2.3 (2017.01.08)
queries can be saved with corresponding service.
-
v2.2 (2017.01.05)
added code bits and operators.
-
v2.1.1 (2017.01.05)
corrections to help text, added links pop-up.
-
v2.1 (2016.12.13)
included help text.
-
v1 … 2.0 (nov/dec. 2016)
initial versions.
- No paste option in query-window context menu.
- Non-indexed fields are absent from suggestion which prohibits them from being easily added the "fields" parameter.
- Field names with periods (like "identifications.defaultClassification.phylum") aren't clickable in JSONpath-screen.
- Check specified field names in query (condition and field-list) against list of available fields for that endpoint.
- Remember JSON or table-view as default (across sessions).
- Start using https://www.sitepoint.com/clipboard-api/ includes paste (and looks nice).
- Explain operators somewhere. Use notes.html?
- Separate inputs for {field}/{group}, name=value and full query (maybe; veers towards a query builder).
- Add non-searchable fields as non-clickable + "not indexed" in drop down.
Netherlands Biodiversity API (NBA)
Useful tools