RELEASE NOTES v 2.29.12 * ETL predict kingdom in default classification for CRS documents * Added gitleaks scanning RELEASE NOTES v 2.29.11 * DWC Archive Preparations Calculator RELEASE NOTES v 2.29.10 * build improvements RELEASE NOTES v 2.29.9 * upgrade to java 17 * multiple dependency upgrades * security improvements RELEASE NOTES v 2.29.8 * Data model update adding additional fields for multimedia RELEASE NOTES v 2.29.7 * Minor fixes RELEASE NOTES v 2.29.6 * Added offline DwcA creation RELEASE NOTES v 2.29.5 * Minor bugfix RELEASE NOTES v 2.29.4 * Log4j security update * Wildfly 26.0.0.Final RELEASE NOTES v 2.29.3 * Log4j security update RELEASE NOTES v 2.29.2 * Log4j security update RELEASE NOTES v 2.29.1 * Log4j security update RELEASE NOTES v 2.29 * Bugfix and library updates RELEASE NOTES v 2.28 * Bugfix and library updates RELEASE NOTES v 2.27 * ETL updates for importing Brahms and CRS source records * DWCA updates RELEASE NOTES v 2.26 * upgrade to Elastic 7.12.0 * multiple dependency upgrades * removed deprecated classes * refactoring and updated java documentation * modified RecordBasisCalculator * bugfix in bulk import RELEASE NOTES v 2.25 * upgrade to Elastic 7.11.2 * upgrade to Wildfly 21.0.2.Final * several minor Java library upgrades * ETL module accepts mineralogy and petrology records with "only" an informal name * bugfixes RELEASE NOTES v 2.24.1 * upgrade to Elastic 7.10.2 * bugfixes * removed services: - specimen/getIdsInCollection - specimen/getNamedCollections RELEASE NOTES v 2.24 * The Dutch Caribbean Species Register (DCSR) has been added as a new data source * Minor updates to the CRS ETL * Some minor bug fixes RELEASE NOTES v 2.23 * Introduction of the batchQuery. The batchQuery service enables you to retrieve documents in batches of configurable size. While the query service is more geared towards GUI builders and the dwcaQuery towards downloading entire datasets, the batchQuery service is more suitable for use in stand-alone programs that process the batches in real-time * Extended the ETL module with generating process reports (apart from the log files). The reports should improve quantitative as well as qualitative assesment of the ETL process * Several minor bug fixes RELEASE NOTES v 2.22 * New feature in this release is a query service that includes name resolution (queryWithNameResolution) improving the recall of a search using scientific and/or vernacular names * Fine-tuning the creation of DWCA download sets * Introduced caching for the DWCA getDataSet service to decrease download time * Fixed several minor bugs to increase general service performance RELEASE NOTES v 2.21 * Moved to Elasticsearch 7 and Wildfly 18 * Several changes to the ETL procedure for data from the source systems Brahms and CRS, all intended to increase the level of detail of taxonomic data * Extensions to the data model to achieve finer grained information storage and retrieval. I.e. TaxonomicRank has been extended with several ranks. * Several (minor) bug fixes RELEASE NOTES v 2.20.2 * A few minor modifications to the ETL module to address changes in some source systems and to improve the logging of imports * fixed some minor vulnarabilities RELEASE NOTES v 2.20.1 * Some minor bug fixes RELEASE NOTES v 2.20 * Moved to Java 11 * Several minor updates and bug fixes RELEASE NOTES v 2.19 * Moved from Apache commons-csv to Univocity for improved csv parsing * Prepared ETL module to enable incremental updates of the document store for selected sources * Changed data format of downloads from json to ndjson (newline delimited json) * Several minor updates and bug fixes RELEASE NOTES v 2.18 * NBA now uses Elasticsearch version 5.6.13 as its document store and search engine * The download service of Darwin Core Archives (DwCA) has been extended to include multimedia RELEASE NOTES v 2.17 * Several (minor) modifications to the data model to meet data requirements of two new data sets. * Several extensions needed for creating DwCA downloads. RELEASE NOTES v 2.16 * This release is for shear technical purposes: we have moved from ant as build tool to maven. This gives us improved dependency control, transparant configuration and better integrated (unit) testing. RELEASE NOTES v 2.15 * Several new fields have been added to the document types Specimen and MultiMediaObject allowing for extra information to be stored (NBAX-148, NBAX-149, NBAX-150, NBAX-151, NBAX-152, NBAX-153) * The servcie getFieldInfo has been updated with a description field. The description can contains more information about the type of field, like a reference to the metadate model (e.g. ABCD Version 2.06). RELEASE NOTES v 2.14 * NBAX-101 New feature in this release is a download service. Up until now, documents could be downloaded using the query service (limited to a maximum of 10.000 documents) or the DWCA downloaded service (limited to a specific set of fields and/or the csv-output format). The new download service does not have these limitations. It instead allows a user to define the set size and select fields according to the demand. * Fix of some technical bugs (NBAX-9, NBAX-102, NBAX-129, NBAX-130) RELEASE NOTES v 2.13 * Main improvement of this release is the modification of exisiting and the creation of new aggregation services. These services can be used, not to retrieve documents (like specimen documents, or taxon documents) from the document store, but summary data about sets of documents. Like, how many documents are there?, or what are distinct values being used in that field? - count - countDistinctValues - countDistinctValuesPerGroup - getDistinctValues - getDistinctValuesPerGroup RELEASE NOTES v 2.12 * Fix of know issue (see v 2.11.4). Conditions directed at fields in nested documents, are now related to other conditions in the same nested documents. Query results are now as expected. * Rigigorous testing methods have been added to the ETL module. These tests ensure consistency of data imports over code changes in both the ETL module itself, as in other modules as well. RELEASE NOTES v 2.11.4 * KNOWN ISSUE The implementation for nested queries has proven to be incorrect. Because of this, queries aimed at nested objects, like identifications or gatheringPersons, ignore the correlation between the inner fields of those objects. This means that searching with with more than one condition within nested objects, will sometimes produce an inaccurate result (generally, more documents than might be expected). RELEASE NOTES v 2.11 * NBAVTWO-478 The groupByScientificName services now use a query cache to increase response times of oft-used queries RELEASE NOTES v 2.10 * NBAVTWO-456 Missing type status no longer causes warning during import of specimen data sources * NBAVTWO-462 Fixed awkward feature where you could not query date fields in the format in which they are presented * NBAVTWO-465 Values for field scientificNameGroup now always lower case * NBAVTWO-470 Reuqesting more than the maximum number of buckets a.k.a. groups in the groupByScientificName services now results in error message * NBAVTWO-472 Values for field numberOfSpecimen will now be null in case of missing data (not 0) RELEASE NOTES v 2.9 * NBAVTWO-121 Date strings in queir condition now more strictly checked * NBAVTWO-374 System classification no longer set for specimens and multimedia * NBAVTWO-446 Fixed bug where dynamic DwCA download crash when printing longitude values * NBAVTWO-447 Fixed bug with strangely formatted dates in DwCA download * NBAVTWO-450 "fields" property of QuerySpec now ignored by DwCA module * NBAVTWO-451 Java client up-to-date again and provided with unit tests. RELEASE NOTES v 2.8 * NBAVTWO-20 Field export organization for vernacular names now set * NBAVTWO-282 DwCA download - synonyms now part of taxa.txt in stead of being written to separate file * NBAVTWO-431 Field subgenus now set for CRS multimedia * NBAVTWO-434 Field subgenus now set for CoL taxa * NBAVTWO-442 DwCA download - better error reporting; if anything goes wrong AFTER downloading has started a file named __ERROR__.txt will be put into the DwCA zip file, containing a Java stacktrace. * NBAVTWO-444 Fixed issue where simultaneious DwCA downloads caused corrupt DwCA files. RELEASE NOTES v 2.7 * NBAVTWO-159 There is a new quartet of operators: STARTS_WITH, NOT_STARTS_WITH, STARTS_WITH_IC, NOT_STARTS_WITH_IC * NBAVTWO-242 Imporved date handling * NBAVTWO-282 DwCA taxon download: taxa and synonyms now in one file (Taxa.txt) * NBAVTWO-311 There now is a REST service displaying all available REST services * NBAVTWO-376 Solved issue with constantScore conditions * NBAVTWO-384 type status now also set for multimedia * NBAVTWO-401 DwCA download - solved issue with meta.xml generation * NBAVTWO-403 DwcA specimen download: now with multimedia links * NBAVTWO-414 system classification no longer populated for specimens and multimedia * NBAVTWO-415 dwca/getDataSetNames - dataset names now sorted alphabetically * NBAVTWO-418 solved issue with groupByScientificName * NBAVTWO-419 solved issue with groupByScientificName * NBAVTWO-420 In queries specified throw separate URL query parameters, you can not search for null using special value @NULL@ * NBAVTWO-428 owner now set for multimedia * NBAVTWO-431 subgenus now set for mulimedia * NBAVTWO-434 subgenus now set for taxa * NBAVTWO-435 solved issue with meta.xml generation RELEASE NOTES v 2.6 * NBAVTWO-416 DwCA download service now more stable and not at risk of timing out RELEASE NOTES V 2.5 * NBAVTWO-375 constantScore setting on QueryCondition now passed on to Elasticsearch without exception * NBAVTWO-377 boost now always set on outermost ES query * NBAVTWO-381 You can now sort on the id field * NBAVTWO-383 NbaDao.find(String[] ids) now sets size the length of the ids array * NBAVTWO-387 querySpecial service now significantly faster b/c * NBAVTWO-388 Fixed bug in extracting QuerySpec for Specimen from QuerySpec for ScientificNameGroup RELEASE NOTES V 2.4 * NBAVTWO-7 Licence and copyright info for NSR is now provided by and set from NSR source files. * NBAVTWO-113 There is a service that allows you to retrieve the names of all DwCA datasets for occurence data: /specimen/dwca/getDataSetNames * NBAVTWO-116 There is a service that allows you to trieve the unitIDs of all specimens within a "special" collection (e.g. Arts, Sybold): /specimen/getIdsInCollection/{collection-name}. Use in combination with /specimen/getNamedCollections * NBAVTWO-131 Language now set for species description within NSR Taxon documents * NBAVTWO-147; NBAVTWO-148; NBAVTWO-149; NBAVTWO-150 There are two typical grouping queries exposed as service: //getDistinctValues/ //getDistinctValuesPerGroup/ See javadocs for INbaAccess * NBAVTWO-227 DwCA files for specimens now contain urls for associated multimedia in occurrence.txt * NBAVTWO-242 You can now search using "whole dates" (format "yyyy-MM") * NBAVTWO-282 For DwCA downloads of taxa synonyms are now written to taxa.txt rather than to a separate file. * NBAVTWO-289 collectors field number is now set * NBAVTWO-295; NBAVTWO-354 The ScientificNameGroup document type has been extended with the following fields: specimens.collectionType specimens.assemblageID specimens.identifications.preferred * NBAVTWO-353 type status is now also set for multimedia objects * NBAVTWO-356 The index.max_result_window configuration settings has been raised to 5,000,000. In other words you are now be able to specify a "from" parameter of 4,999,990 given the default "size" of 10 documents per page (or 4,999,925 with a page size of 75 documents) See also e.g. service /specimen/metadata/getSettings * NBAVTWO-368 You can no specify a query condition which is the equivalent of SQL's "WHERE 1 = 1" or "WHERE true": just leave field, operator and value all blank. See also javadocs for QueryCondition * NBAVTWO-370 URL-type fields no longer support the LIKE operator * NBAVTWO-380 There is a new metadata service that tells you which date formats you are allowed to use in queries: /metadata/getAllowedDateFormats. See also javadocs for INbaMetaData. RELEASE NOTES V 2.3 * NBAVTWO-263 There now is a metadata service that gives information about various constants and limits pertaining to the operation of the NBA and the underlying Elasticsearch index. * NBAVTWO-283 Data model change. The specimen's type status field has been moved to the specimen's identification. * NBAVTWO-285 Data model change. The TaxonomicIdentification object within a Specimen and MultiMediaObject document now contains data from the associated Taxon document. This enhances searchability. * NBAVTWO-291 There is new service for the ScientificNameGroup document: querySpecial. This service provides more intuitive query results than the stock query method. * NBAVTWO-325 If you sort on anything else than document relevance, the score field in the query result will be null (i.e. absent) rather than NaN * NBAVTWO-326 You can now explicitly sort on document relevance by specifying the "_score" sort field. * NBAVTWO-311 There now is a metadata service than returns all available NBA REST services. RELEASE NOTES V 2.2 * NBAVTWO-285 Query specimens using taxonomic attributes. * NBAVTWO-324 Allowed operators for SummaryXXX objects (objects within the ScientificNameGroup document) now the same as for the full-blown counterparts. * NBAVTWO-291 New document type: ScientificNameGroup; contains a "frozen" aggregation on taxon documents and specimen documents. Taxa and Specimens are grouped by their scientific name. * NBAVTWO-317 associatedTaxonReference and associatedSpecimenReference field now contain true Elasticsearch document IDs rather than source system IDs. * NBAVTWO-318 New metadata service: getFieldInfo, providing information about how/if a field can be queried. * NBAVTWO-309 Document type mapping changes for one document no longer require all document types to be re-created. * NBAVTWO-308 Index settings configurable per index. ======================================================= RELEASE NOTES V 2.06 * NBAVTWO-61 Sorting on nested fields now enabled. * A new service with endpoint /release-notes has been created. * NBAVTWO-291 A new document type, ScientificNameSummary, is introduced and exposed through the IScientificNameSummaryAccess API. * A import program and shell script have been created to populate the ScientificNameSummary index. The import program also runs as part of the import-all script. * The import-all shell script has been modified. It still imports all data sources, but now generates a different log file for each data source. * More fields can now be queried using the LIKE operator. As a rule of thumb: a name-ish fields can now be queried using the LIKE operator. * Less fields can now be queried using the MATCHES operator. All fields whose values are controlled and all fields that can possibly only contain a single word (e.g. rank names like "genus", "kingdom", etc.) are not analyzed using Elasticsearch's standard analyzer any longer and can therefore not be queried using the MATCHES operator. * NBA Version now runs on Elasticsearch V5 * NBAVTWO-292 Name of log file in etl import script now includes name of imported document type. * NBAVTWO-285 The specimen document type now contains a new object taxonomicEnrichments, which contains vernacular names and synonyms from associated taxa in our taxonomic systems (currently Catalogue of Life and Dutch Species Registry). This allows you to search for specimens using taxonomic attributes.