Davrods Enhancements as part of the Grassroots Infrastructure
Introduction
One of the major aims of the Grassroots Infrastructure is to allow users to share their wheat data as easily and seamlessly as possible. To do this, we use iRODS which is a system allowing users to create files and directories as normal with extra features such as distributed storage viewable across different institutions and the ability to add metadata to all of the files and directories.
iRODS comes with command-line clients to allow access to its storage and Chris Smeele and Ton Smeele developed an Apache httpd module, Davrods to allow access to it using standard WebDAV commands. There were still some functionality that we wanted to add that was missing from Davrods, namely:
- Themeable listings
- Public access
- Metadata display
Themeable listings
The standard web pages produced by Davrods resembled the basic directory listings produced by Apache and we wanted to increase the functionality of these pages as well as making their look and feel customisable. To do this we based our approach upon that used by the autoindex module[4]. We added the ability to customise the HTML that would go in the head section of the web page as well as the parts before and after the iRODS directory listings. Along with this, the columns of the listings table were marked up to allow for easy styling. The link to get to the parent collection was replaced with a full navigation element with a full set of breadcrumbs.
Another concept that we replicated from the autoindex module was the ability to specify default icons to use for the data objects and collections along with more difficult icons depending upon their file types.
Figure 1: Editing the metadata within a browser.
Public access
iRODS workflow is based around the idea that users log in to see the data that they have permissions to access. Often for websites, there is a desire to give public access to browse a list of files and directories without the need to log in. So another feature that we added to Davrods was the ability to have a default user to be specified in the Apache server configuration that would be used to log into the iRODS system without the need for any user intervention. In effect, creating a public-facing website indistinguishable from one that allows a user to browse regular files and directories.
Metadata display
One of the major benefits of iRODS is its ability to add metadata as attribute-value pairs to any of the data objects or collections stored within it. The ability to view, edit and search across this data is a really useful feature for end users and so we wanted to add these capabilities to Davrods too. Two features we wanted available were to expose each of the metadata attribute-value pairs as hyperlinks to find data objects or collections also containing the same pair, as well as having a general search functionality to search across the entire metadata collection. Going further, adding the ability to add, edit or delete metadata from within in a web browser was another really useful feature that we wanted.
To do this we created a Representational state transfer (REST) API for querying and manipulating the iRODS metadata. The base address for this API is at /davrods/api/metadata though this can be changed to be anything you would like by altering the parameter in the Apache configuration. It contains the following functions:
get: This is for getting all of the associated metadata for an iRODS item. It takes two parameter, the first is id, which is the iRODS id of the data object or collection that you wish to get the metadata pairs for. The second parameter is output_format which specifies the format that the metadata will be returned in. It currently can take one of the following values:
- json: This will return the metadata as a JSON (JavaScript Object Notation) array with each entry in the array having attribute, value, and where appropriate, units keys for its key-value pairs.
- csv: This will return the metadata as a table of comma-separated values with the order of the columns being attribute, value, units. Each of these entries will be contained within double quotes to allow for commas within their values without causing errors.
- tsv: This will return the metadata as a table of tab-separated values with the order of the columns being attribute, value, units. Each of these entries will be contained within double quotes to allow for commas within their values without causing errors.
For example to get the metadata for a data object with the id of 1.10021 in a JSON output format, the URL to call would be
/davrods/api/metadata/get?id=1.10021&output_format=jsonsearch: This API call is for getting a list of all data objects and collections that have a given metadata attribute-value pair. It takes two parameters: key, which is the attribute to search for and, value, which specifies the metadata value. There is a third optional parameter, units for specifying the units that the metadata attribute-value pair must also have. So to search for all of the data objects and collections that have an attribute called volume with a value of 11, the URL to call would be
/davrods/api/metadata/search?key=volume&value=11add: This API call is for adding a metadata attribute-value pair to a data object of collection. It takes three parameters: id, which is the iRODS id of the data object or collection to add the metadata to, key, which is the attribute to add and, value, which specifies the metadata value to be added. As with the search call listed above, there is a fourth optional parameter, units for specifying the units that the metadata attribute-value pair will have. So to add an attribute called volume with a value of 11 to a data object with the id of 1.10021, the URL to call would be
/davrods/api/metadata/add?id=1.10021&key=volume&value=11edit: This API call is for editing a metadata attribute-value pair for a data object of collection and replacing one or more of its attribute, value or units. It takes the following required parameters: id, which is the iRODS id of the data object or collection to delete the metadata from, key, which is the attribute to edit, value, which specifies the metadata value to edit. Again, there is an optional parameter, units for specifying the units that the metadata attribute-value pair must also have to match. There must also be one or more of the following parameters to specify how the metadata will be altered: new_key, which is for specifying the new name for the attribute, new_value, for specifying the new metadata value and new_units for specifying the units that the metadata attribute-value pair will now have. So to edit an attribute called volume with a value of 11 and units of decibels for a data object with the id of 1.10021 and give it a new value of 8 and units of litres, the URL to call would be
/davrods/api/metadata/edit?id=1.10021&key=volume&value=11&units=decibels&new_value=8&new_units=litresdelete: This API call is for deleting a metadata attribute-value pair from a data object of collection. It takes three parameters: id, which is the iRODS id of the data object or collection to delete the metadata from, key, which is the attribute to delete for and, value, which specifies the metadata value to delete. As before, there is a third optional parameter, units for specifying the units that the metadata attribute-value pair must also have to match. So to delete an attribute called volume with a value of 11 and units of decibels from a data object with the id of 1.10021, the URL to call would be
/davrods/api/metadata/delete?id=1.10021&key=volume&value=11&units=decibels
To demonstrate this REST API, we have developed some client-side JavaScript functions so that users can call all of these routines comfortably and intuitively from within their web browser.
We have two example sites to show the enhancements that we have made. The first demo site https://wheatis.tgac.ac.uk/davrods/browse/reads/ is a website showcasing the anonymous access among with theming and read-only access to the metadata. The second demo site at https://grassroots.tools/davrods/ showcases the ability to add, edit and delete metadata attribute-value pairs for the available data objects and collections.
Download
The Davrods software is available to download at https://github.com/billyfish/davrods.