Wednesday 25 January 2017

Sampling-event standard takes flight on the wings of butterflies

Data collected from systematic monitoring schemes is highly valuable. That's because harvesting species data from a given set of sites repeatedly over time using a well-defined sampling effort opens the door to key ecological analyses including phenology, population trends, changes in community structure and other metrics related to a range of Essential Biodiversity Variables (EBVs).

A couple of years ago there was no faithful way to universally standardize data from systematic monitoring schemes. This meant that researchers using this kind of data would need to spend a lot of time deciphering it first. Their job would get even more complicated when trying to integrate data from various heterogeneous sources, each storing their data in different formats, units, etc.

Today, the situation looks much better thanks to a massive collaboration between GBIF, EU BON partners and the wider biodiversity community whose aim was to enable sharing of "sampling-event datasets".  

Indeed, one of the most successful outcomes from this collaboration has been the development of a standardized format for systematic butterfly monitoring schemes.

The format has been developed in close collaboration with the EU BON partners Israel Pe'er (GlueCAD- Biodiversity IT) and his son, Dr. Guy Pe'er, (UFZ), who works with systematic monitoring data.  The format can be adapted to many other types of systematic monitoring, for many taxonomic groups, as it ensures the following important conditions for researchers are met:
  • all visits to a given site are known, including those with no sightings, as this allows for analyses of species phenology, etc.
  • the range of species being recorded during sampling is explicit, as this allows for true absence to be determined.
  • the location hierarchies can be specified (e.g. the location is a fixed transect or subsection of a transect), as this allows users to group observations by location.
  • enough detailed information about the sampling effort and sampling area (e.g. units of measurement) are captured, as this allows users to calculate density or convert between units of abundance.
The Israeli Butterfly Systematic Monitoring Scheme (BMS-IL) dataset has already been published openly using this format. I'd like to invite everyone to explore this exemplar dataset from either the EU BON IPT or via

In the future, I hope that GEO BON's Guidelines for Standardized Global Butterfly Monitoring will incorporate a new recommendation that all monitoring programs use this standardized format for sharing their data. Without a doubt this will make researchers' jobs easier when integrating data from several butterfly monitoring programs for their analyses. It will also enable integrating the data with standardized sampling-event data from other disciplines as well.

Ideally, making the data openly available in a standardized format also leads to new collaboration. So far, BMS-IL data has been used to assess trends in the abundance and phenology of Israel's butterflies for the benefit of conservation or climate change research for example. I would like to encourage you to reach out to Israel and Guy Pe'er if you have any novel ideas on how to reuse their newly standardized data in order to help unlock its full potential.

Thursday 12 January 2017

IPT v2.3.3 - Your repository for standardized biodiversity data

GBIF is pleased to announce the release of IPT v2.3.3, now available for download from the IPT website.

This version looks and feels the same as 2.3.2 but is much more robust and secure. I'd like to recommend that all existing IPT installations be upgraded as soon as possible following the instructions listed in the release notes.

Additionally, a couple new strategic features have been added to the tool to enhance its potential. A description of these new features follows below.

Improved dataset homepage

Compared with general-purpose repositories such as Dryad or Figshare, the IPT ensures that uploaded biodiversity data gets disseminated in a standardized format (Darwin Core Archive - DwC-A), facilitating wider reuse and enabling the data to be indexed by aggregators such as

Interoperability comes at a small cost though, as depositors choosing to use the IPT must overcome a learning curve in understanding how to map their data to the Darwin Core standard.

To make this easier for depositors, a new set of Darwin Core Excel templates have recently been released. These new templates provide a simpler solution for capturing, formatting and uploading data to the IPT.

Similarly, users of the standardized data need to understand how to unpack a DwC-A and make sense of the data inside.  

Data Records section - RLS Global Reef Fish Dataset
To make this process easier for users, a new Data Records section has been added to the dataset homepage that provides an explanation of what the DwC-A format is with a graphic illustration showing the number of records in each file contained within it.

Overall this advancement will strengthen the IPT as a data repository, which is already capable of assigning DOIs to datasets to make them discoverable and citable. 

Translation into Russian 

Map of IPT installations in Russia - January 2017
Installed in 52 countries around the world, use of the IPT heavily is underrepresented across Russian speaking countries. Therefore to extend the IPT's reach in these areas, the user interface has been fully translated into Russian by a team of volunteer translators with the largest contribution made by Ivan Chadin from the Komi Science Centre of the Ural Branch of the Russian Academy of Sciences.

Map of data published by Russia - January 2017
At the time of writing there were already 18 datasets from Russia published by 5 IPTs installed across Pushchino, Moscow, St Petersburg and the Komi Republic. It will be exciting to watch this number grow over time in part thanks to this enormous volunteer contribution.


Once again I'd like to recognize all the volunteer translators that contributed their time and expertise to making this new version available in seven different languages:
  • Sophie Pamerlon (GBIF France) - Updating French translation
  • Yukiko Yamazaki (GBIF Japan (JBIF)) - Updating Japanese translation
  • Daniel Lins (Universidade de São Paulo, Research Center on Biodiversity and Computing - BioComp) - Updating Portuguese translation
  • Néstor Beltrán (Colombian Biodiversity Information System (SiB Colombia)) - Updating Spanish translation
  • Ivan Chadin (Institute of Biology of Komi Scientific Centre of the Ural Branch of the Russian Academy of Sciences), Max Shashkov (Institute of Physicochemical and Biological Problems in Soil Science, Russian Academy of Science) and Artyom Leostrin (Komarov Botanical Institute of the Russian Academy of Sciences (Saint-Petersburg)) - Adding Russian translation 
I'd also like to recognize a few volunteers that helped make significant improvements to the IPT codebase:
  • Bruno P. Kinoshita (National Institute of Water and Atmospheric Research (NIWA)) - Fixed issue #1241, ensuring the IPT can be installed on a server behind a proxy
  • Pieter Provoost (UNESCO) - Fixed issue #1248, improving the IPT's RSS feed
  • Tadj Youssouf (Security researcher, - Helped address a cross site scripting issue
Although the core development of the IPT happens at the GBIF Secretariat, the coding, documentation, and internationalization are a community effort and everyone is welcome to join in.

I look forward to seeing the IPT's community of volunteers and users continue to grow and hope you can unlock the full potential of this publishing tool and repository.