Launch of the Canadensys explorer

At Canadensys we already adopted and customized the IPT as our data repository. With the data of our network being served by the IPT, we have now built a tool to aggregate and explore these data. For an overview of how we built our network, see this presentation. The post below originally appeared on the Canadensys blog.

We are very pleased to announce the beta version of the Canadensys explorer. The tool allows you to explore, filter, visualize and download all the specimen records published through the Canadensys network.

The explorer currently aggregates nine published collections, comprising over half a million specimen records, with many more to come in the near future. All individual datasets are available on the Canadensys repository and via the Global Biodiversity Information Facility (GBIF) as well. The main functionalities of the explorer are listed below, but we encourage you to discover them for yourself instead. We hope it is intuitive. For the best user experience, please use an up-to-date version of your browser.

Happy exploring: http://data.canadensys.net/explorer

Functionalities

• The explorer is a one page tool, limiting unnecessary navigation.
• The default view shows all the data, allowing users to get an overview and explore immediately.
• Data can be queried by using and combining filters.
• Filters use smart suggestions, informing the user how often their search term occurs even before they search.
• The exact number of results is displayed, including the number of georeferenced records.
• The map view displays all georeferenced records for the current query, has different base layer options and can be zoomed in to any level.
• Points on the map can be clicked for more information.
• The table view displays a sortable preview of the records in the current query.
• Records in the table can be clicked for more information in the same way as on the map.
• The number of columns in the table responds to the screen width and can be controlled by the user in the display panel.
• Data for the current query can be downloaded as a Darwin Core archive. There is no limit on the number of records that can be downloaded.
• Users can download the data by providing their email address. Once the download package is generated, the user receives an email with a link to the data, information regarding the usage norms and a suggested citation.
• The interface and emails are available in French and English.

As this is a beta version, you may encounter issues. Please report them by clicking the feedback button on the right, which will open a report form.

Technical details

The Canadensys explorer was developed using the following open source tools:

• Spring, a Java framework used for the backend.
• ThreeTen, a date/time Java library, used for cleaning the eventDate field.
• GBIF parsers, a Java library used for cleaning the country field.
• ECAT common, a Java library used for parsing the scientificName field.
• Darwin Core archive reader, a Java library used to import the data in the backend and to create the user generated downloads.
• PostgreSQL, the database.
• PostGIS, a geospatial extension to the database.
• Mapnik, a tool to generate the maps and provide the map interactivity.
• Windshaft, a tile server.
• Freemarker, a Java template engine, used to structure to frontend.
• Backbone.js, a MVC structure for web applications, used for handling the filters.

Updating a customized IPT

This post originally appeared on the Canadensys blog and is a follow-up of the post Customizing the IPT.

As mentioned at the very end of my post about customizing the IPT, I face a problem when I want to install a new version of the GBIF Integrated Publishing Toolkit: installing it will overwrite all my customized files! Luckily Tim Robertson gave me a hint on how to solve this: a shell script to reapply my customization.

Here's how it works (for Mac and Linux systems only):

Comparing the customized files with the default files

First of all, I need to compare my customized files with the files from the new IPT. They might have changed to include new functionalities or fix bugs. So, I installed the newest version of IPT on my localhost, opened the default files and compared them with my files. Although there are tools to compare files, I mostly did this manually. The biggest change in version 2.0.3 was the addition of localization, for which I'm using a different UI, so I had to tweak some things here and there. It took me about 3 hours until I was satisfied with the new customized IPT version on my localhost.

I also subscribed to the RSS of the IPT Google Code website, to be notified of any changes in the code of "my" files, but I was just using this as a heads-up for coming changes. It is more efficient to change everything at once, when a stable version of IPT is out.

• RSS subscription for any changes in /webapp/WEB-INF/pages/inc, which contains most of my customized files
• RSS subscription for any changes in /webapp/styles/main.css, where I'm commenting out a lot of stuff so my CSS can kick in.

Setting up a file structure

This is how we've organized the files on our server. I've created a folder called ipt-customization, which contains all my customized files. That way, they can never be overwritten by a new IPT installation, which gets deployed in webapps. The folder also contains a script to apply the customization and a folder to backup the default files currently used by IPT.

• ipt-data
• webapps
  
  • ipt
• ipt-customization
  
  • backup-default
  • apply-customization.sh
  • revert-customization.sh
  • header.ftl
  • header_setup.ftl
  • menu.ftl
  • footer.ftl
  • main.css
  • custom.js

Creating the shell script

The apply-customization.sh script works in two steps:

1. Backup the default files, by copying them from IPT to the folder backup-default. The script will ask if I want to overwrite any previously backed up files. The last part is important if I'm running the script several times. In that case I do not want to overwrite the backups with the already customized files.
2. Overwrite the files currently used by IPT with the customized files, by copying them from my ipt-customization folder to the correct folder in IPT

# backup files of new IPT installation
cp -i ../webapps/ipt/WEB-INF/pages/inc/footer.ftl ../ipt-customization/backup-default/
cp -i ../webapps/ipt/WEB-INF/pages/inc/header_setup.ftl ../ipt-customization/backup-default/
cp -i ../webapps/ipt/WEB-INF/pages/inc/header.ftl ../ipt-customization/backup-default/
cp -i ../webapps/ipt/WEB-INF/pages/inc/menu.ftl ../ipt-customization/backup-default/
cp -i ../webapps/ipt/styles/main.css ../ipt-customization/backup-default/

# apply customization
cp footer.ftl ../webapps/ipt/WEB-INF/pages/inc/
cp header_setup.ftl ../webapps/ipt/WEB-INF/pages/inc/
cp header.ftl ../webapps/ipt/WEB-INF/pages/inc/
cp menu.ftl ../webapps/ipt/WEB-INF/pages/inc/
cp main.css ../webapps/ipt/styles/
cp custom.js ../webapps/ipt/js/

I also created a script revert-customization.sh, to revert the customization to the default IPT, in case something is broken. It moves the backed up files back to IPT:

# revert customization
cp backup-default/footer.ftl ../webapps/ipt/WEB-INF/pages/inc/
cp backup-default/header_setup.ftl ../webapps/ipt/WEB-INF/pages/inc/
cp backup-default/header.ftl ../webapps/ipt/WEB-INF/pages/inc/
cp backup-default/menu.ftl ../webapps/ipt/WEB-INF/pages/inc/
cp backup-default/main.css ../webapps/ipt/styles/
rm ../webapps/ipt/js/custom.js

Running the script

From the command line, I login to my server, navigate to the folder ipt-customization and make my script executable:

chmod +x apply-customization.sh

I only have to do this the first time I want to use my script. From then on I can use:

sh ./apply-customization.sh

To execute the script and customize my new version of IPT!

Customizing the IPT

One of my responsibilities as the Biodiversity Informatics Manager for Canadensys is to develop a data portal giving access to all the biodiversity information published by the participants of our network. A huge portion of this task can now be done with the GBIF Integrated Publishing Toolkit version 2 or IPT. The IPT allows to host biodiversity resources, manage their data and metadata, and register them with GBIF so they can appear on the GBIF data portal, which are all targets we want to achieve. Best of all, most management can be done by the collection managers themselves.

I have tested the IPT thoroughly and I am convinced the GBIF development team has done an excellent job creating a stable tool I can trust. This post explains how I have customized our IPT installation to integrate it with our other Canadensys websites.

Background

Our Canadensys community portal is powered by WordPress (MySQL, PHP), while our data portal - which before the IPT installation only consisted of the Database of Vascular Plants of Canada (VASCAN) - is a Tomcat application. We are using different technologies because we want to use the most adequate technology for a certain website. WordPress (or Drupal for that matter) is an excellent and easy-to-use CMS, perfect for our community portal, but not suitable for a custom made checklist website like VASCAN. To the user however, both websites look the same:

We do this by using the same HTML markup and CSS for both websites. If you want to learn HTML and CSS, w3schools provides excellent tutorials.

The HTML markup defines elements on a page (e.g. header, menu, content, sidebar, footer) and the CSS stylizes those elements (e.g. their position and color). The CSS is typically stored as one file (e.g. style.css) which is referenced in the <head> section of a page. For dynamic websites, the HTML is typically stored as different files, one for each section of a page (e.g. header.php, sidebar.php). Those files are combined as one page by the server if a page is requested. That way, changing a common element on all pages of a website (e.g. the header) can be done by changing just one file.

All of this also applies to the IPT. Here's how the IPT looks like without CSS:

Attempt 1 - Editing the CSS and logo

My first attempt at customizing the IPT was at the Experts Workshop in Copenhagen, by changing the CSS and logo only, which you can find in the /styles folder of your IPT installation:

/styles/main.css
/styles/logo.jpg

In 15 minutes, my IPT was Canadensys red and had a custom logo:

Attempt 2 - Editing the FreeMarker files

Even though my IPT now had its own branding, it was still noticeably different from the other Canadensys websites. The only way I could change that, was by editing the HTML as well. Luckily, the sections I wanted to change were all stored as FreeMarker files in the /inc folder:

/WEB-INF/pages/inc/header.ftl - the <head> section
/WEB-INF/pages/inc/menu.ftl - the header, menu and sidebar
/WEB-INF/pages/inc/footer.ftl - the footer
/WEB-INF/pages/inc/header_setup.ftl - the header during installation

I incorporated the HTML structure I use for the VASCAN website into menu.ftl (including the header, menu, container and sidebar), making sure I did not break any of the IPT functionality.

I started doing the same with main.css by replacing chunks of now unused IPT CSS with CSS I copied over from VASCAN, but I quickly realized that this wasn't the best option. Doing so would result in 2 CSS files: one for VASCAN and one for IPT, even though both web applications are under the same domain name with a lot of shared CSS. It would be easier if I only had to maintain a single stylesheet, used by both applications.

Attempt 3 - One styles folder for the data portal

I created a /common/styles folder under ROOT, where I placed my single common data portal stylesheet: /common/styles/common.css. This would be the CSS file I could use for IPT and VASCAN. I did the same for my favicon: /common/images/favicon.png.

I added a reference to both files in the header.ftl of my IPT (and VASCAN):

<link rel="stylesheet" type="text/css" href="${baseURL}/styles/main.css">
<link rel="stylesheet" type="text/css" href="http://data.canadensys.net/common/styles/common.css">
<link rel="shortcut icon" href="http://data.canadensys.net/common/images/favicon.png">

As you can see on the first line, I kept the reference to the default IPT stylesheet: ${baseURL}/styles/main.css (it's perfectly fine to reference more than one CSS file). This is where I would keep all the unaltered (=default) IPT CSS. In fact, I'm not removing anything from the default IPT stylesheet, I'm only commenting out the CSS that is unused or conflicting:

/* Unused or conflicting CSS */

The advantage of doing so, is that I now easily can compare this commented file with changes in the stylesheet of any new IPT version.

After I had done everything, my IPT now looked like this:

My IPT is now sporting the Canadensys header, footer and sidebar (only visible when editing a resource), making it indistinguishable from the other Canadensys websites. It is also using a more readable font-size (13.5px) and a fluid width.

Closing remarks

I have (re)designed quite a lot of websites, and very often I have been so frustrated with the HTML and CSS that I just started over from scratch. I didn't have that option here and it wasn't necessary either. I would like to thank the GBIF development team for creating such an easily customizable tool, with logical HTML and CSS. As a reminder, the whole customization has been done by editing only 5 files (links show default files):

/styles/main.css (custom file)
/WEB-INF/pages/inc/header.ftl
/WEB-INF/pages/inc/menu.ftl
/WEB-INF/pages/inc/footer.ftl
/WEB-INF/pages/inc/header_setup.ftl

Important: Remember that installing a new IPT version will overwrite all the customized files, so make sure to back them up! I will try to figure out a way to reapply my customization automatically after an update and post about that experience in a follow-up post. In the meantime, I hope that this post will help others in the customization of their IPT.