Connecting to the NTSPortal Elasticsearch Search API from R or Python
Connecting-to-NTSPortal-API.RmdAPI Connection, getting started
The API is called from the URL stored in the package option
ntsportal.elasticsearchHostUrl. In the case of the
NTSPortal-Bund server, this is https://elastic.bafg.de.
Users must 1) have been granted API access (please send a request to
ntsportal@bafg.de) and 2) must have created an API key for
themselves at ntsportal.bafg.de (gear icon->Security/API
keys, ‘control security privileges’ must be checked”)
Saving the login credentials is achieved using
ntsportal::connectNtsportal().
ntsportal::getTableByEsql() is the simplest function for
collecting data from NTSPortal in R. It uses the ES|QL API
(See documentation)
which can be tested in the Discover page of the NTSPortal
dashboard. The data is returned as a tbl_df object (similar
to a data.frame). The limitation of ES|QL is
that nested fields (like ms2 and
compound_annotation) cannot be queried or retrieved, for
this you must use ntsportal::getTableByQuery().
ntsportal::getTableByQuery() allows fetching records
from NTSPortal as a tbl_df and allows the retrieval of
nested fields. This function uses the Query DSL language (in the form of
a list) to pass the queries. The best way to get started
with Query DSL is to use the Elasticsearch “Developer Tools” console in
the NTSPortal Dashboards.
Currently the R/dbInterface-PythonDbComm.R methods are
used to implement the interface. The interface object is created using
getDbComm() but most user-facing functions automatically
create the interface using the stored credentials.
For an example of using the API in Python see Using the Elasticsearch Search API with Query DSL for retrieving documents.
The API documentation is found here. Detailed documentation is available for the different search languages offered, e.g. Query DSL.
Internal implementation
The API interface functions above use the Python package
elasticsearch to query the database (documentation). To
see how it works look at the code for
ntsportal::PythonDbComm(), which uses the
inst/pythonElasticComm/elasticSearchComm.py module to
create an elasticsearch client object in R. This client is used in many
of the interface functions found in
R/dbInterface-PythonDbComm.R.