Developer Blog

Performance Evaluation of HBase

2012-02-28T17:09:00.000+01:00

In the last post Lars talked about setting up Ganglia for monitoring our Hadoop and HBase installations. That was in preparation for giving HBase a solid testing run to assess its suitability for hosting our index of occurrence records. One of the important features in our new Data Portal will be the "Download" function that lets people download occurrences matching some search criteria and currently that process is a very manual and labour intensive one, so automating it will be a big help to us. Using HBase it would be implemented as a full table scan, and that's why I've spent some time testing our scan performance. Anyone who has been down this road will probably have encountered the myriad opinions on what will improve performance (some of them conflicting) along with the seemingly endless parameters that can be tuned in a given cluster. The overall result of that kind of research is: "You gotta do it yourself". So here we go.

Because we hope to get some feedback from the HBase community on our results (ideally comparisons to other running clusters) we decided to use the PerformanceEvaluation class that ships with HBase. Unfortunately it's not completely bug free so I've had to patch it to work in the way we would like. From a stock cdh3u3 HBase I patched HBASE-5401 and HBASE-4688, and had a good, hard think about HBASE-5402. There are a bunch of things I learned about the PerformanceEvaluation (PE) class in the last few weeks, and the implications of 5402 are some of them. If anyone's interested I'll blog about those another time, but for now suffice it to say that in my final suite of tests I went with writing tables using "sequentialWrite" (which obviates the difficulties in 5402) and doing scans with the "scan" test.

There are many variables that can be changed in an HBase/Hadoop cluster setup but you have to start somewhere so here's our basic config that remained unchanged throughout the test (you can see more at its ganglia page):

Master (c1n2): HDFS NameNode, Hadoop JobTracker, HBase Master, and Zookeeper
Slaves (c2n1, c2n2, c2n3): HDFS DataNode, Hadoop TaskTracker, HBase RegionServer (6 GB heap)

Hardware:
c1n2: 1x Intel(R) Xeon(R) X3363 @ 2.83GHz (quad), 8GB RAM, 2x500G SATA 5.4K
c2n*: 2x Intel(R) Xeon(R) E5630 @ 2.53GHz (quad), 24GB RAM, 6x250G SATA 7.2K

The parameters I varied are:

number of rows in the test table (50M, 100M, 200M)
number of mappers per server (8, 10, 12, 14, 20 - more than 20 and we ran out of RAM)
snappy compression (on/off)
scanner caching (size)
block caching (on/off)

Test Methodology

The test methodology was to start with an empty HDFS (replication factor of 2), an empty HBase and no other tasks running on the machines. Then build a table using a command line like

hbase org.apache.hadoop.hbase.PerformanceEvaluation sequentialWrite 100

and then wait for the split/compaction cycle to complete, and then run a major compaction to encourage data locality. In tests using compression I would construct an empty table by hand, specifying compression=>'SNAPPY' for the info column family before running the sequentialWrite. I'd also wait for the region balancer to run, ensuring that all regionservers had an equal number of regions (+/- 1). Once that was complete I would run a scan test with a command like:

hbase org.apache.hadoop.hbase.PerformanceEvaluation scan 100

and note the real time taken as reported by the JobTracker as well as the total time spent in mappers as reported by PE itself. I ran each scan test 3 times unless the first 2 tests were very close, in which case I sometimes skipped the 3rd run. Note also that PE uses 10 byte keys and 1000 kilobyte values by default.

Two things were immediately obvious in my caching tests, which I'll talk about briefly before getting into the results proper.

Scanner Caching

The HBase API Scan class takes a setting called setCaching(int caching) which tells the scanner how many rows to fetch at a time from the server. This defaults to 1, which is optimal for single gets, but is decidedly suboptimal for big scans like ours. Unfortunately PE doesn't allow for explicitly setting this value on the command line and so I took the advice of the API Javadoc and passed in a value of 1000 for the property hbase.client.scanner.caching in the hbase-site.xml file that forms the configuration for the MapReduce job that is PE. It would seem, though, that this property is not being honoured by HBase and so my initial tests all ran with a cache size of 1, producing poor results. I subsequently hard-coded the setCaching(1000) on the scans themselves in the PE and saw significant improvements. Those differences are visible in the results, below.

Block Caching

When I say block caching I mean HBase blocks - not HDFS blocks. See the HBase guide or Lars George's excellent HBase: The Definitive Guide for many more details. You'll see many recommendations from different sources to turn block caching off during scans but it's not immediately obvious why that should be. It turns out that it has no performance implications for the scan itself, but will significantly impact any random reads (gets) that may be relying on recently loaded blocks. If the scan is using the block cache then it will load the cache with a block, read the block, and then load the next one, ignoring the first one for the rest of the scan. The earlier loaded blocks will be evicted quickly (as new blocks are loaded) leading to "cache churn". So I tested the block cache on and off and concluded that there was no difference in a dedicated scan test (no other activity in the cluster) and so for the rest of my tests I left the block cache on. In order to test this behaviour I again had to hard-code PE with setCacheBlocks(false) (it defaults to true).

The Results

The full spreadsheet is available for download from our google code site. I've just included the summary chart here, because it pretty much tells the story. Of note: the y-axis is Records / second which I calculate as the total records scanned divided by the total time for the job as reported by the JobTracker. This doesn't, therefore, take any of the PE generated per-mapper counts/times into account (which are still somewhat mysterious to me).

Conclusions (within our obviously limited test environment):

with bigger datasets, more mappers are better; with smaller datasets, somewhat fewer are better
compression slows things down. This is probably the most surprising (and contentious) finding. We would expect performance to improve with compression (at worst staying the same) because there is much less data to transfer when the files are compressed, and I/O is typically the limiting factor in HBase performance. I kept an eye on the cpus with ganglia during the uncompressed tests and sure enough, they spent approximately double the amount of time in io_wait as in the compressed tests.
ganglia revealed no obvious bottlenecks during the scan. You can look at our ganglia charts from the past few weeks and apart from a few purposefully degenerate tests there was no obvious component that was limiting things - cpus never went above 60% (io_wait never above 30%), and RAM and disks "looked ok". If anyone has insight here we'd really appreciate it.
total variability across all these tests was only about 20%. Either all tests are hitting the same, as yet unseen, hardware or configuration limit, or there really is no silver bullet to performance increases - tuning is an incremental thing.
there are still strange things going on :) I can't explain the fact that in the compressed test performance increases from 50 to 100M records, and then decreases dramatically with 200M records.

We'd love feedback on these results, both in terms of how we can improve our testing and in interpreting the results, as well as any numbers you may have from your clusters when running PE. Is 300k records per second even in the ballpark? It's crazy that these data don't exist out there already, so hopefully this post helps alleviate that somewhat.

The good news is that we'll be getting new cluster hardware soon, and then I think the next step is to load our real occurrences data into a table and run similar scanning tests against it to optimize those scans. Of course then we'll start writing new records at the same time, and then things will get.... interesting. Should be fun :)

Monitoring Hadoop and HBase

2012-02-08T15:57:00.000+01:00

We're getting serious in our Hadoop adoption. The first process (our so called "rollover") is now in production and it uses Hadoop, Hive, Oozie and various other parts of the Hadoop ecosystem.

Our next step is evaluating HBase and its performance on our (small and aging) cluster. To do that properly and to fix a rather embarrassing situation we first had to get proper monitoring up and running for our cluster. So far we've only had Cacti stats for OS level things (CPU, I/O, etc.) but we were missing actual Hadoop statistics.

So we've now set up Ganglia at GBIF and the best news is it's public and using the very latest Ganglia 3.3 which was released only a few days ago in February 2012. The setup was relatively painless. Ganglia was just nice to work with. To get monitoring of HBase working we had to apply HBASE-4854 because it's not included in our Hadoop distribution (CDH3u2). Thanks to Lars George for the hint.

So we can happily report that Ganglia 3.3 works perfect with the GangliaContext31 from Hadoop.

Now all that's left is learning what most of these stats mean and then trying to extract useful information from them. Any hints are more than welcome and the HBase community already offered to help. Thank you very much!

For anyone interested in a few notes and details about how we set Ganglia up keep on reading.

First we had to configure iptables. This accepts all multicast packets for UDP and IGMP (it took me a while to figure out this was missing...):

iptables -A INPUT -p igmp -d 224.0.0.0/4 -j ACCEPT
iptables -A INPUT -p udp -d 224.0.0.0/4 -j ACCEPT

We've split up our 18 machines in two Hadoop clusters so we wanted those separate in Ganglia as well. The only way I could get this to run is to have both clusters join different multicast addresses (mcast_join in gmond.conf). We're doing this using Puppet.

In gmetad.conf we then include one machine from each cluster:

data_source "hadoop-1" c1n1.gbif.org
data_source "hadoop-2" c1n2.gbif.org

In multicast mode all machines that are listening on the same multicast address know about all state from all machines in that "ring". So what gmetad has to do is to connect to one of the machines from each ring via TCP (not multicast, that's what the tcp_accept_channel is for in gmond.conf).

BioCASe now producing DarwinCore Archives

2012-01-19T14:08:00.001+01:00

Guest post from Jörg Holetschek, Botanic Garden and Botanical Museum Berlin-Dahlem.

The traditional way of sharing occurrence data with GBIF has been web-service-based for years. Data publishers have used one of the existing provider software packages (DiGIR, BioCASe or TAPIR Link) to expose their data as a DiGIR-, BioCASe- or TAPIR-compliant web service. Biodiversity networks such as GBIF used harvesters to crawl and index the records published by these services, an approach that works fine for small and medium-sized datasets, but runs into difficulties when record numbers hit the millions: Harvesting can take days and puts a heavy load on both the publisher and the crawler.

To overcome this, GBIF recently introduced DarwinCore Archives for storing all information of a dataset to be published in a single file. GBIF directly ingesting this file eliminates the time-consuming back-and-forth communication between data provider and harvester, speeding up the process and reducing load for both sides. GBIF’s IPT allows easy creation of such DarwinCore Archives and is a good option for providers that have already used the DarwinCore standard in the past or that want to share rather slim observation data.

However, sixty-two of GBIF’s data publishers are currently using BioCASe. In contrast to DarwinCore, BioCASe and its associated data standard ABCD are targeted mainly at rich data originating from specimens of natural history collections (even though it can be used for any type of occurrence data, including observations). Many of the BioCASe data providers also share their data with special interest networks such as GeoCASe, the DNA-Bank Network, or the EDIT Specimen network, all of them relying on BioCASe web services. Switching to the IPT and the associated DarwinCore standard is not an option for them.

For this reason, we decided to extend the BioCASe Provider Software with a feature to create DarwinCore Archives. This allows providers to continue using the rich ABCD schema (or one of its extensions) for the specific networks they’re connected to while using DarwinCore Archives to share their data with GBIF. In order to combine the richness of ABCD with the efficiency of downloadable archives, we created a hybrid in-between, the so called ABCD Archives, which can be used instead of the BioCASe web service for harvesting purposes (see figure below).

The first step – creating the ABCD archive – is implemented natively in the Provider Software in Python. The second step – transforming the ABCD archive into one or several DarwinCore archives – is done by Pentaho Data Transformation, an open source library also known as Kettle. In the current version 3.0, the transformation step is a stand-alone command-line application that can be downloaded separately; ultimately, it will be bundled with the Provider Software and integrated into the user interface.

The latest version of the Provider Software and the DarwinCore Creator can be downloaded from the BioCASe website. A detailed documentation of the new archiving features can be found in the PyWrapper Wiki. The wiki also stores a sample ABCD archive and a sample DarwinCore archive created by BioCASe.

Updating a customized IPT

2011-12-08T16:47:00.001+01:00

This post originally appeared on the Canadensys blog and is a follow-up of the post Customizing the IPT.

As mentioned at the very end of my post about customizing the IPT, I face a problem when I want to install a new version of the GBIF Integrated Publishing Toolkit: installing it will overwrite all my customized files! Luckily Tim Robertson gave me a hint on how to solve this: a shell script to reapply my customization.

Here's how it works (for Mac and Linux systems only):

Comparing the customized files with the default files

First of all, I need to compare my customized files with the files from the new IPT. They might have changed to include new functionalities or fix bugs. So, I installed the newest version of IPT on my localhost, opened the default files and compared them with my files. Although there are tools to compare files, I mostly did this manually. The biggest change in version 2.0.3 was the addition of localization, for which I'm using a different UI, so I had to tweak some things here and there. It took me about 3 hours until I was satisfied with the new customized IPT version on my localhost.

I also subscribed to the RSS of the IPT Google Code website, to be notified of any changes in the code of "my" files, but I was just using this as a heads-up for coming changes. It is more efficient to change everything at once, when a stable version of IPT is out.

RSS subscription for any changes in /webapp/WEB-INF/pages/inc, which contains most of my customized files
RSS subscription for any changes in /webapp/styles/main.css, where I'm commenting out a lot of stuff so my CSS can kick in.

Setting up a file structure

This is how we've organized the files on our server. I've created a folder called ipt-customization, which contains all my customized files. That way, they can never be overwritten by a new IPT installation, which gets deployed in webapps. The folder also contains a script to apply the customization and a folder to backup the default files currently used by IPT.

ipt-data
webapps
- ipt
ipt-customization
- backup-default
- apply-customization.sh
- revert-customization.sh
- header.ftl
- header_setup.ftl
- menu.ftl
- footer.ftl
- main.css
- custom.js

Creating the shell script

The apply-customization.sh script works in two steps:

Backup the default files, by copying them from IPT to the folder backup-default. The script will ask if I want to overwrite any previously backed up files. The last part is important if I'm running the script several times. In that case I do not want to overwrite the backups with the already customized files.
Overwrite the files currently used by IPT with the customized files, by copying them from my ipt-customization folder to the correct folder in IPT

# backup files of new IPT installation

cp -i ../webapps/ipt/WEB-INF/pages/inc/footer.ftl ../ipt-customization/backup-default/

cp -i ../webapps/ipt/WEB-INF/pages/inc/header_setup.ftl ../ipt-customization/backup-default/

cp -i ../webapps/ipt/WEB-INF/pages/inc/header.ftl ../ipt-customization/backup-default/

cp -i ../webapps/ipt/WEB-INF/pages/inc/menu.ftl ../ipt-customization/backup-default/

cp -i ../webapps/ipt/styles/main.css ../ipt-customization/backup-default/



# apply customization

cp footer.ftl ../webapps/ipt/WEB-INF/pages/inc/

cp header_setup.ftl ../webapps/ipt/WEB-INF/pages/inc/

cp header.ftl ../webapps/ipt/WEB-INF/pages/inc/

cp menu.ftl ../webapps/ipt/WEB-INF/pages/inc/

cp main.css ../webapps/ipt/styles/

cp custom.js ../webapps/ipt/js/

I also created a script revert-customization.sh, to revert the customization to the default IPT, in case something is broken. It moves the backed up files back to IPT:

# revert customization

cp backup-default/footer.ftl ../webapps/ipt/WEB-INF/pages/inc/

cp backup-default/header_setup.ftl ../webapps/ipt/WEB-INF/pages/inc/

cp backup-default/header.ftl ../webapps/ipt/WEB-INF/pages/inc/

cp backup-default/menu.ftl ../webapps/ipt/WEB-INF/pages/inc/

cp backup-default/main.css ../webapps/ipt/styles/

rm ../webapps/ipt/js/custom.js

Running the script

From the command line, I login to my server, navigate to the folder ipt-customization and make my script executable:

chmod +x apply-customization.sh

I only have to do this the first time I want to use my script. From then on I can use:

sh ./apply-customization.sh

To execute the script and customize my new version of IPT!

Bug fixing in the GBIF Data Portal

2011-12-05T14:28:00.000+01:00

Despite our current efforts to develop a new Portal v2, our current data portal at data.gbif.org has not been left unattended. Bug fixes are being done periodically from feedback sent to us from our user community. In order to keep our community informed, this post will summarize the most important fixes and enhancements done in the past months:

The data portal's main page now shows the total number of occurrence records with coordinates, along with the total count of records (non-georeferenced and georeferenced).

Decimal coordinate searches were not working properly. When a user wanted to refine their coordinate searches to use decimals, the data portal was returning an erroneous count of occurrence records. Issue was fixed. Details here.

Any feedback e-mail message sent from an occurrence or a taxon page now includes the original sender's email address in the CC field. Previously the sender's email address was not included in the feedback email, which represented a problem when the receiver replied to the email, but the sender never knew about the reply.

The Taxon Web Service's GET operation was returning errors when trying to request some specific taxons. The problem was detected and fixed.

On an occurrence detail page, when retrieving the original record from a data publisher, and the source data came from a Darwin Core Archive, it was not possible to retrieve a single record due to the single-file nature of a DwC Archive. (As opposed to a DiGIR request, in which you could extract just a single record). A fix was introduced so that the user can decide if he/she wants to download the complete archive (see an example).

When using the data portal's Web Services to produce KML output, there were some problems when the generated KML contained HTML elements and Google Earth tried to open the file for visualization (This is a standard problem of XML). A small fix was introduced to escape the conflicting HTML inside the KML output.

Other small GUI enhancements where also done.

Updates to the data portal's codebase is now done seldomly, but our goal is to fix any major issues that our user community reports. If you ever encounter problems, please don't hesitate to contact us at portal@gbif.org.

Important Quality Boost for GBIF Data Portal

2011-11-09T09:44:00.000+01:00

Improvements speed processing, “clean” name and location data, enable checklist publishing.

[This is a reposting from the GBIF news site]

A major upgrade to enhance the quality and usability of data accessible through the GBIF Data Portal has gone live.

The enhancements are the result of a year’s work by developers at the Copenhagen-based GBIF Secretariat, in collaboration with colleagues throughout the worldwide network.

They respond to a range of issues including the need for quicker ‘turnaround’ time between entering new data and their appearance on the portal; filtering out inaccurate or incorrect locations and names for species occurrences; and enabling species checklists to be indexed as datasets accessible through the portal.

After a testing period, the changes now apply to the more than 312 million biodiversity data records currently indexed from some 8,500 datasets and 340 publishers worldwide.

Key improvements include:

•    processing time for data has fallen from 3-4 days to around 36 hours, paving the way for more frequent ‘rollovers’ or index updates;
•    the ‘backbone taxonomy’ used by the GBIF Portal has been reworked with up-to-date checklists and taxonomic catalogues such as the Catalogue of Life 2011, improving search and download;
•    checklists describing species in particular geographic locations, taxonomic groups or thematic categories (eg. invasives) can now be published using a standard set of terms called the Global Names Architecture (GNA) Profile (see GNA guidelines) and thus become indexed and accessible via the Data Portal;
•    automated interpretation of the coordinates, country location and scientific names used in published records has been improved to screen out inaccuracies – for example, ensuring that records identified as coming from a particular country are shown as occurring within the borders and territorial waters of that country; and
•    a mechanism using the Hadoop open-source software system has been introduced to ensure that the Data Portal is able to cope with anticipated future growth in the volume of data.

The algorithms and dictionaries developed to improve interpretation of data published through the GBIF Data Portal are intended for future re-use by the wider biodiversity informatics community.

Commenting on the release of these substantive Data Portal improvements, GBIF Executive Secretary Nicholas King said: “These changes represent a major step forward in the usefulness of GBIF to science and society.

“They are a direct response to the feedback we have had from the data publishing and user communities, and will enable an even greater return on the long-term investment made over the past decade by GBIF Participant countries.”

IPT v.2.0.3 launched

The GBIF Secretariat has also issued a new release of the Integrated Publishing Toolkit (IPT), which enables biodiversity data updates to be ‘harvested’ automatically from databases published to the Internet.

IPT version 2.0.3 addressed 76 reported issues from the previous version, and includes translations into French and Spanish.

Instructions on installing the new version are available here.

Integration tests with DBUnit

2011-10-21T11:52:00.000+02:00

Database driven JUnit tests
As part of our migration to a solid, general testing framework we are now using DbUnit for database integration tests of our database service layer with JUnit (on top of liquibase for the DDL).

Creating a DbUnit test file
As it can be painful to maintain a relational test dataset with many tables, I've decided to dump a small, existing Postgres database into the DbUnit XML structure, namely FlatXML. It turned out to be less simple as I had hoped for.

First I've created a simple exporter script in Java that dumps the entire DB into XML. Simple.

The first problem I've stumbled across was a column named "order" which caused a SQL exception. It turns out DbUnit needs to be configured for specific databases, so I've ended up using three configurations to both dump and read the files.

Use Postgres specific types
Double quote column and table names
Enable case sensitive table & column names (now that we use quoted names, Postgres becomes case sensitive)

After that, reading in the DbUnit test file started out fine, but reached a weird NullPointerException that left me puzzled. After trying various settings I finally found a log warning that some columns might not be detected properly by DbUnit, as it only inspects the first XML record by default which can contain many null columns which subsequently then will be ignored. Luckily since version 4.3.7 of dbunit you can tell the builder that reads in the test files to scan all records first in memory, a feature know as column sensing. That got me a long way, but ultimately I've hit a much harder issue. Relational integrity.

The classic way to avoid integrity checks during inserts (including DbUnit) is simply to temporarily disable all foreign key constraints. On some databases this is simple. For example in MySQL you can simply execute SET FOREIGN_KEY_CHECKS=0 in your db connection. In H2 there is an equivalent of SET REFERENTIAL_INTEGRITY FALSE. Unfortunately there is nothing like that in PostgreSQL. You will have to disable all constraints individually and then painfully recreate them. In our case these were nearly a hundred constraints and I didn't want to go down that route.

The latest DBunit comes with a nice DatabaseSequenceFilter to automatically sort the tables being dumped in an order that respects the constraints. That worked very well for all constraints across tables, but of course failed to sort the individual records in tables which contain a self reference, for example the taxonomy table which has an adjacency list via parent_fk. Luckily I had only one table like this and that included already some nested sets indices (lft,rgt) that allowed me to sort the records in a parent first order. For this I had to issue a custom SQL query though, so I ended up dumping the entire database with all tables using the filter and in addition to export only one table with a custom sql that I then had to manually copy into the complete xml dump file. Voila, finally a working DbUnit test file!

DatabaseDrivenTest for MyBatis Services
In order to load the test data into a test DB for every JUnit test we decided to use a JUnit Rule definition that is executed before each test. The class responsible for most of the magic is DatabaseDrivenTest which is parameterized for the specific MyBatis Service to be tested. It is generic and can be used with any database system. The subclass DatabaseDrivenChecklistBankTest<T> then adds the database specific configurations and can be used as a Rule within the individual tests.

A simple & clean integration test example does look like this now:

public class ReferenceServiceMyBatisTest {

  @Rule
  public DatabaseDrivenChecklistBankTest ddt = new DatabaseDrivenChecklistBankTest(ReferenceService.class, "squirrels-full.xml");

  @Test
  public void testGet() {
    Reference ref = ddt.getService().get(37);
    assertEquals("Wilson, D. E. ; Reeder, D. M. Mammal Species of the World", ref.getCitation());
    assertEquals(100000025, ref.getUsageKey());
  }

Isn't that gorgeous? We only need to pass the dbunit test file and the service class to be tested to the JUnit Rule and then only need to bother with testing the service results! No additional setting up or tearing down is needed.

GBIF Portal: Geographic interpretations

2011-10-20T16:06:00.000+02:00

The new portal processing is about to go into production, and during testing I was drawing some metrics on the revised geographic interpretation. It is a simple issue, but many records have coordinates that contradict the country that the record claims to be in. Some illustrations of this were previously shared by Oliver.

The challenge of this is two fold. Firstly we see many variations in the country name which needs to be interpreted. Some examples for Argentina are given (there are 100s of variations per country):

Argent.
Argentina
Argentiana
N Argentina
N. Argentina
ARGENTINA
ARGENTINIA
ARGENTINNIA
"ARGENTINIA"
""ARGENTINIA""
etc etc

We have abstracted the parsing code into a separate Java library which makes use of basic algorithms and dictionary files to help interpret the results. This library might be useful for other tools requiring similar interpretation, or data cleaning efforts, and will be maintained over time as it will be in use in several GBIF tools.

The second challenge is that we need to determine if the point falls within the country. There is always room for improvement in this area, such as understanding changes over time, but due to the huge volume of outliers when using the raw data a check like this is required. Our implementation is a very basic reverse georeferencing RESTful web service that takes a latitude and longitude, and returns the proposed country and some basic information such as the title. Operating the service requires PostGIS and a Java server like Apache Tomcat. Currently we make use of freely available terrestrial shapefiles, and marine economic exclusion zones. It would be trivial to expand the service to use more shapefiles for other uses, and is expected to happen over time. Currently the GBIF service is an internal only processing service, but is expected to be released for public use in the coming months.

Improving the country name interpretation and making use of a more accurate geospatial verification service than previously will help improve data reporting at the national level using the GBIF portal as indicated here.

		# Records	# Georeferenced
Argentina	Previously	665,284	284,012
Argentina	Now	680,344	303,889
United States	Previously	79,432,986	68,900,415
United States	Now	81,483,086	70,588,182

Group synergy

2011-10-07T10:52:00.000+02:00

During the last few weeks we have been intensively designing and implementing what would come to be the new data portal. Oliver described nicely the new stage our team has entered in his last blog post Portal v2 - There will be cake. As my personal opinion, I think this has been truly a group experience as we have decided to change our paradigm of working. Normally we would have worked on different components each one of us and later try to integrate everything, but now we took the approach of just focusing on one subcomponent, all of us, and driving our efforts into it. From my point of view, the main advantage of this is that we avoid the Bus Factor element, that we as a small group of developers, are quite exposed to. Communication has increased among our team as we are all on the same page now.

As a general overview, the portal v2 will consist of different subcomponents (or sub-projects) that would need to interact between them to come up with a consolidated "view" for the user. Currently we have 3 different sub-projects on our tray, Checklist Bank, Registry, and Occurrence Store and our plan will be to have an API (exposed through web services) which will offer all data necessary (from these projects) for the portal to consume. The portal will then need to make use of a simple webservice client to communicate with this API.

Currently we have been working on the Checklist Bank sub-project. As Oliver pointed out in his previous post, some members of our team are more familiarized with certain sub-projects, and the checklist one does not escape from this reality. So for many, including me, it has been a learning experience. We have started development following very strict guidelines on API design and code conventions (which we document internally for our use). Even decisions that are sometimes taken in seconds by a single developer, are placed under group scrutiny so we are all on the same track. We have taken the commitment to apply the best coding practices.

Specifically on the checklist sub-project, we have come up with a preliminary API. Please note this API won't be exposed to the public as it is. It is subject to change as we try to refine it. It is just nice to show to the outside world what we have been working on.

I personally think we are in exciting times inside GBIF and that the final product of all this effort would be a great tool that would benefit the community in big ways. Expect more from us!

Portal v2 - There will be cake

2011-09-27T10:59:00.000+02:00

The current GBIF data portal was started in 2007 to provide access to the network's biodiversity data - at the time that meant a federated search across 220 providers and 76 million occurrence records. While that approach has served us well over the years, there are many features that have been requested for the portal that weren't addressable in the current architecture. Combined with the fact that we're now well over 300 million occurrence records, with millions of new taxonomic records to boot, it becomes clear that a new portal is needed. After a long consultation process with the wider community the initial requirements of a new portal have been determined, and I'm pleased to report that work has officially started on its design and development.

For the last 6 months or so the development team has been working on improving our rollover process, registry improvements, IPT development, and disparate other tasks. The new portal marks an important milestone in our team development as we're now all working on the portal, with as little distraction from other projects as we can manage. Obviously we're still fixing critical bugs and responding to data requests, etc, but all of us focusing on the same general task has already shown dividends in the conversations coming out of our daily scrums. Everyone being on the same page really does help.

And yes, we've been using daily stand-up meetings that we call "scrums" for several months, but the new portal marks the start of our first proper attempt at agile software development, including the proper use of scrum. Most of our team has had some experience with parts of agile techniques, so we're combining the best practices that everyone has had to make the best system for us. Obviously the ideal of interchangeable people with no single expert in a given domain is rather hard for us when Tim, Markus, Kyle and Jose have worked on these things for so long and people like Lars, Federico and I are still relatively new (even though we're celebrating our one year anniversaries at GBIF in the next weeks!), but we're trying hard to have non-experts working with experts to share the knowledge.

In terms of managing the process, I (Oliver) am acting as Scrum Master and project lead. Andrea Hahn has worked hard at gathering our initial requirements, turning them into stories, and leading the wireframing of the new portal. As such she'll be acting as a Stakeholder to the project and help us set priorities. As the underlying infrastructure gets built and the process continues I'm sure we'll be involving more people in the prioritization process, but for now our plates are certainly full with "plumbing". At Tim's suggestion we're using Basecamp to manage our backlog, active stories, and sprints, following the example from these guys. Our first kickoff revealed some weaknesses in mapping Basecamp to agile, and the lack of a physical storyboard makes it hard to see the big picture, but we'll start with this and re-evaluate in a little while - certainly it's more important to get the process started and determine our actual needs rather than playing with different tools in some kind of abstract evaluation process. Once we've ironed out the process and settled on our tools we'll also make them more visible to the outside world.

We're only now coming up on the end of our first, 2 week sprint, so it will take a few more iterations to really get into the flow, but so far so good, and I'll report back on our experience in a future post.

(If you didn't get it, apologies for the cake reference)

VertNet and the GBIF Integrated Publishing Toolkit

2011-09-15T09:08:00.000+02:00

(A guest post from our friends at VertNet, cross-posted from the VertNet blog)

This week we’d like to discuss the current and future roles of the GBIF Integrated Publishing Toolkit (IPT) in VertNet. IPT is a Java-based web application that allows a user to publish and share biodiversity data sets from a server. Here are some of the things IPT can do:

Create Darwin Core Archives. In our post about data publishing last week, we wrote about Darwin Core being the “language of choice” for VertNet. IPT allows publishers to create Darwin Core data records from either files or databases and to export them in zipped archive files that contain exactly what is needed by VertNet for uploading.

Make data available for efficient indexing by GBIF. VertNet has an agreement with its data publishers that, by participating, they will also publish data through GBIF. GBIF keeps our registry of data providers and uses this registry to find and update data periodically from the original sources to make it available through the GBIF data portal. IPT gives data publishers an easy means of keeping their data up-to-date with GBIF.

IPT can help with the data publishing process in other ways as well:

standardizing terms
validating records before they get published
adding default values for fields that aren’t in the original data

To get a better understanding of the capabilities, take a look at the IPT User Manual.

Why are we using IPT?

VertNet has a long waiting list of organizations (65 to date) that have expressed interest in making their data publicly accessible through VertNet. In the past, these institutions would have needed their own server and specialized software (DiGIR) for publishing to the separate vertebrate networks. We’d rather not require any of these participants to buy servers if they don’t have to. As an interim solution, we’re using the IPT to make data available online while we build VertNet. We have installed, at the University of Kansas Biodiversity Institute, an IPT that can act as a host for as many collections as are interested. The service is shared, yet organizations can maintain their own identity and data securely within this hosted IPT. This is a big win for us at VertNet, because there will be fewer servers to maintain and we can get more collections involved more quickly.

Going forward…

Well before completion, VertNet will support simple and sustainable publishing by uploading records from text files in Simple Darwin Core form. Because of this, the IPT will not be a required component of data publishing for VertNet. Rather, we see IPT as a great tool to facilitate the creation of Darwin Core Archives, which we will be able to use to upload data to VertNet.

Interested in publishing now with IPT?

We currently have two institutions sharing their collections with VertNet and GBIF through the VertNet IPT and we’re in the process of working with several more.

So, if you are or would like to be a vertebrate data publisher and would like to make your data accessible as Darwin Core Archives sooner rather than later, VertNet’s IPT might be the solution for you! Learn more about the process on the VertNet web site or email Laura Russell and Dave Bloom.

Posted by Laura Russell, VertNet Programmer; John Wieczorek, Information Architect; and Aaron Steele, Information Architect

Darwin Core Archives for Species Checklists

2011-08-26T15:03:00.004+02:00

GBIF has long had an ambition for supporting the sharing of annotated species checklists through the network. Realising this ambition has been frustrated by the lack of a data exchange standard of sufficient scope and simplicity as to promote publication of this type of resource. In 2009, the Darwin Core standard data set was formerly ratified by the TDWG, Biodiversity Information Standards. The addition of new terms, and a means of expressing these terms in a simplified and extensible text-based format, paved the way for the development of a data exchange profile for exchanging species checklists known as the Global Names Architecture (GNA) Profile. Species checklists, published in this format, can be zipped into single, portable, 'archive' files.

Here I introduce two example archives that illustrate the flexible scope of the format. The first represents a very simple species checklist while the second is a more richly documented taxonomic catalogue. The contents of any file can be viewed by clicking on the file icon or filename. A complete list of terms used in sharing checklists can be found here.

Example 1: **U.S. National Arboretum Checklist**
This checklist represents the most simple checklist archive. It consists of a document that describes the checklist and a second file with the checklist data itself. The checklist data consist of two columns. Note that by including column headers that match the standard DarwinCore term names, that no additional mapping document is needed.
EML.xml	The checklist is documented using an Ecological Metadata Language (EML) document.
Checklist.txt	The checklist itself is kept in this simple text file.

Example 2: **Catalog of Living Whales**
This checklist represents an annotated species checklist. In addition to the core species list ('whales.tab') there are numerous other data types consisting of Darwin Core extensions that conform to the GNA Profile. This more complex archive contains a resource map file ('meta.xml') that describes the files in the archive. An EML metadata document describes the catalog itself. This more complex archive uses a common identifier, taxonID, to link data in the extension files to the data records in the core species checklist ('whales.tab').
EML.xml	The checklist is documented using a Ecological Metadata Language (EML) document. It includes a title, contacts, citation information and more.
whale s.tab	The checklist itself is kept in this tab-delimited file.
meta.xml	The files in the archive are described in this resource map file.
distribution.tab	Distribution information conforming to the GNA Distribution extension are stored in this file.
references.tab	Bibliographic references are stored in this file and linked to 'whales.tab' via the taxonID
types.tab	Type specimen details are contained in this file.
vernaculars.tab	Common name information that conforms to the GNA Vernacular Extension are stored in this file.

Configuring Drupal and some modules for ticketing emails

2011-08-21T22:42:00.008+02:00

We at the Secretariat receive enquiries via helpdesk[at]gbif[dot]org, portal[at]gbif[dot]org and info[at]gbif[dot]org, everyday, or I would say, almost every hour. Some of them are provider-specific questions that need special attention from staff, while some others are FAQs. We have been thinking about better managing questions/issues, so by adding a little bit structure in the collaborative workflow, we can:

1. Make sure questions are answered with satisfaction; 2. Estimate how much man hours have been spent, or evaluate performance; 3. Improve efficiency on helpdesk activities.

To achieve these, we need softwares that meet these requirements:
1. Case management for incoming emails;
2. A Q&A cycle should be completed by solely using email. Web forms are good but not necessary in the beginning;
3. Easy configured knowledge base essays;
4. Graphical reports shows the helpdesk performance;
5. Automatic escalation of case status.

We looked for options from Open Source Help Desk List. While most of the sounding choices are tailored for software development cycle, some are commercial packages/services that indeed designed for enterprise help desk needs. While evaluating a few of those packages, I also found with Drupal and some modules, a solution that just meets our need is pretty out-of-box ready. The result is quite convincing and I can imagine the transition won't require too much learning of my colleagues.

Here is the recipe.

Materials and methods:

1. A mail server. All right I admit this is not something easy if you're not a system administrator. We use Dovecot to provide IMAP access to emails.
2. A Drupal installation. Installation instructions are here. As a wimp I choose version 6.
3. The Support module. Downloadable at http://drupal.org/project/support.
support_deadline, support_fields, support_timer, support_views, support_token, and support_nag are relevant modules that fit our purposes.
4. The CCK module. Downloadable at http://drupal/project/cck.
5. The Views module. Downloadable at http://drupal/project/views.
6. The Google Chart module. Downloadable at http://drupal.org/project/chart. Not "charts", which is a different module.
7. The Date module. Downloadable at http://drupal.org/project/date.
8. The Admin Menu module, for your administrative pleasure. Downloadable at http://drupal.org/project/admin_menu.
9. The Views Calc module, required by Support modules. Downloadable at http://drupal.org/project/views_calc.
10. Download all necessary modules to [drupalroot]/sites/all/modules directory. Enable them at [baseURL]/admin/build/modules.
11. You should see a "support ticketing system" menu by now. You need to

Add an email client with an email account you set on the dovecot mail server;
You probably want to change the email template at [baseURL]/admin/support/settings/mail;
Go through the general settings of the ticketing system at [baseURL]/admin/support/settings.

12. Send some testing emails to the testing email address.
13. Visit [baseURL]/admin/support/clients/1/fetch, see if the system retrieve email and create tickets successfully.

14. After more testing emails have been sent to the address and fetched. You can visit [baseURL]/admin/support/charts.

These are just some facets of my explorations so far. Some details are not covered, like permissions in Drupal. Probably a newbie would need a crash course of Drupal to start, but after that things will be easier and faster.

Using C3P0 with MyBatis

2011-08-12T14:29:00.001+02:00

The problem

In our rollover process, which turns our raw harvested data into the interpreted occurrences you can see on our portal, we now have a step that calls a Web Service to turn geographical coordinates into country names. We use this to enrich and validate the incoming data.

This step in our process usually took about three to four hours but last week it stopped working all together without any changes to the Web Service or the input data.

We've spent a lot of time trying to find the problem and while we still can't say for sure what the exact problem is or was we've found a fix that works for us which also allows us to make some assumptions about the cause of the failure.

It is a project called geocode-ws and it is a very simple project that uses MyBatis to call a PostgreSQL (8.4.2) & PostGIS (1.4.0) database which does the GISy work of finding matches.

Our process started out fine. The first few million calls to the Web Service were fine and returning reasonably fast but then at the end the process slowed down until it came almost to a complete stop with response times of over 10 minutes. That's when our Hadoop maps timed out and failed.

With hindsight we should have come to our final conclusion much earlier but it took us a while.

Looking for the problem

We've looked at the PostgreSQL configuration and tweaked it a lot. We added a lot more logging and we've made sure to log any long running statements (using the log_min_duration_statement option). We also made sure that our memory settings are sensible and that we don't run out of memory. Looking at io- and vmstats as well as our Cacti monitoring we could see that this wasn't the case though. PostgreSQL didn't seem to be the problem.

We also looked at the OS configuration itself as well as the connectivity between our Hadoop cluster and this Tomcat and PostgreSQL server but couldn't find the problem either.

Then we began improving our Web Service and implemented a JMX MBean to get more detailed information about the process. While our changes should have improved the code base they didn't fix the problem. Finally we enabled GC logging on our Tomcat server (something we should have done much earlier and we will probably do by default for our servers in the future). We didn't do it earlier because the Web Service didn't experience any symptoms of memory leak issues before and we didn't change anything there. It hadn't even been restarted in a while.

But as it turned out the problem was garbage collection. Unfortunately I can't provide pretty graphs because I've overwritten the GC logs but it was very easy to see (using the awesome GCViewer) a typical pattern of minor collections not reclaiming all space and growing memory usage up until the point where almost no memory could be reclaimed and most of the time was spent in Garbage Collection. We found the problem! This explained our time outs.

It still doesn't explain what was leaking though. And having spent that much time on it we quickly gave up trying to find the problem. We suspect some kind of combination between the MyBatis Connection Pool, the PostgreSQL JDBC driver and our configuration.

Our workaround (the MyBatis & C3P0 part)

So we looked around for other connection pools for use with MyBatis but unfortunately we couldn't find a ready made thing. There are implementations in the MyBatis-Guice project but they can only be used with the Annotation based configuration and we're using XML.

We ended up writing our own implementation of a C3P0 DataSourceFactory and it turned out to be very very easy: It is just one class (JavaDoc here) with one line of code in it.

This not only solved our apparent memory leak but the performance increased by a factor of two to three as well. We haven't had a problem with our setup since.

Conclusion

We didn't have the time to find the real problem but we found a solution that works for us. I suspect had we gone about this better we might have found the problem a lot sooner and perhaps identified the real reason for it.

Enable GC logging!
Enable JMX for Tomcat and set up your applications with useful metrics and logging
Even though the use of Profilers is heavily disputed they can often help. We've found YourKit to be excellent
Try to follow a logical route, change only one thing at a time, mock things to find a problem
Monitor and graph your systems

Indexing occurrences data - using Lucene technology

2011-08-03T08:35:00.000+02:00

The GBIF Occurrence Index collects, stores and parses data gathered from different sources to provide a fast and accurate access to biodiversity occurrence data. The purpose of having a GBIF Index is optimize speed, relevance and performance of search functionalities that will be implemented by the new GBIF portal architecture.

Currently, GBIF has been providing search functionalities in its Data Portal supported in a semi-denormalized index relational database design, which allows find occurrence information by specifying filters to refine the expected results. That design was envisioned to support use cases of the actual GBIF Data Portal (a Web application); for the next generation of the GBIF platform, a new set of requirements must be achieved and is possible that the current index will not be able to support them, the most relevant of those requirements are: scheduling of batch exports, full text search, realtime faceted search and probably new schemas of data sharing with other biodiversity networks.

For implementing this new Occurrence Index, several technologies are under evaluated, each technology taken into considerationfor specific features that make them an attractive option, those are:

Technology	Description
PostgreSQL	This relational data base contains several features that worth evaluate: query optimization for JOIN-like queries, flexible key-value store, partial indices and multicolumn indices
Lucene Index	At least four options are available for this implementation: pure Lucene Index, Katta, Apache Solr and ElasticSearch
MySQL	This is the current implementation of the index, a evaluation could help to determine if this technology will be able to support new use cases
Key-value systems	Several schema-less data stores are available: CouchDB, Mongo, PostgreSQL hstore. The main concern about these technologies is their capabilities to handle a considerable amount of records

In this post will show some preliminary results in the evaluation of Lucene-based indices, specifically: Lucene as itself, Katta, Apache Solr and ElasticSearch. The analysis will keep apart two concerns (in this post only the index creation is cited):

Index creation, means how the index is created, split (in shards) and merged if necessary.
Index use, refers in how the index performs in terms of usability (queries and search patterns), performance (response time) and througput.

Index Creation

Three scenarios were considered to the index creation phase:

Single process - n Indices: In this case a single process creates n-shards, the input data are split evenly; an IndexWriter is created for each shard. The case of n = 1 is considered part of this scenario, the # of shards is a parameter defined by the user, and is equal to the # of expected shards at the end of the process.
N threads - n Indices: The IndexWriter is a thread safe class, so it can be shared by several threads in order to create a single Index. The # of shards is defined by the user and internally is used to define the # of IndexWriters.
Distributed Index creation: in this case the index is created by splitting the input data into N shards, each shard is assigned to one process that contains a single IndexWriter which is responsible for the index creation.

Single process index creation

The process followed for this scenario is pretty straightforward:

The input is a row delimited file and each column is separated by a special character ('/001' in our case.
The # of shards input defines the number IndexWriters (only one IndexWriter can be opened for a Lucene Index).
Each row represents a Lucene document and is stored using one of the available index writers.
If multiple indices were created, at the final step those are merged into a single index (using "IndexWriter.addIndexesNoOptimize")

This process was tested using a 100 millions of records file. The entire process took 9200821 milliseconds (= 9200.821 seconds = 153.347016666666667 minutes = 2.555783611111111 hours) to finish. Some optimizations were implemented for this process, worth mention that the same sort of optimizations were applied for the multithread scenario:

Re-use the IndexWriter in multiple threads

Re-use the org.apache.lucene.document.Document and org.apache.lucene.document.Field instances. The Lucene fields are created in a static block and its value is changed for each new Document, then the document is added to the index. The intention of this is avoid the objects to be garbage collected.

  //Initialization
  static {
    for (int i = 0; i < accFieldsValues.length; i++) {
      fields[i] = new Field(accFieldsValues[i].name(), "", Store.YES, Index.ANALYZED);
    }
  }
 ...
 //Sets the field value
 fields[fieldsCount].setValue(stringTokenizer.nextToken());
 ...
 //Adds the same document instance with different values
 indexWriter.addDocument(doc);

The compound file format is turned off, this reduces the amount of files opened at the same time.
IndexWriter.autocommit is set to false: since the index doesn't provide searching during the creation time, this feature can be disable.

The flush is done by RAM, and the RAM usage is maximized:

 
LogByteSizeMergePolicy logByteSizeMergePolicy = new LogByteSizeMergePolicy(); logByteSizeMergePolicy.setMergeFactor(mergeFactor);
...
indexWriterConfig.setRAMBufferSizeMB(bufferSize);

Every N documents a entry is written in a log in order to notify the overall progress.

Multithreaded Index Creation

In terms of optimizations applied this scenario is very similar to the "Single process" scenario. However, the process is very different in terms of steps and the resulted index:

The input is a row delimited file and each column is separated by a special character ('/001' in our case).
The number of rows of the input file is known and is passed as input parameter.
The input file is split evenly in intermediate files, each file is assigned to a thread which will read it to create a Lucene Index.
The intermediate files are deleted after each index is created.
Depending of the number of shards desired , the indices are merged in smaller set of indices.

This process was run using: a pool of 50 threads and an input file with 100 million of rows. The execution time is detailed in the next table:

Phase	Time
Slicing (split the input file and distribute it in the threads)	1045948ms == 17.43 minutes
Indices creation	6890988 == 114.8498 minutes
Total time	132.2798 minutes = 2.204663333333333 hours

Katta for Index Creation

Katta is a distributed storage of indices, currently supports 2 types of indices: Lucene and Hadoop MapFiles. It uses ZooKeeper to coordinate the index creation, replication and the search across the nodes.

Main relevant features

An Katta index is basically a folder containing sub-folders (shards)
The client-node communication is implemented using HadoopRPC
Supports distributed scoring, each search query requires two network roundtrips: get document frequencies in all shards and second perform the query.
Provides functionality to merge indices (though is not a very comple task to implement using standard Lucene libraries)

Relevant issues found

Documentation is not extensive and lacks of necessary detail.
Small community and the development is very low: last commit was done in 2009-04-2.
Doesn't provide any help to create the indeces, index sharding must be done prior to import them into Katta.

Test environment configuration

A cluster of 4 Katta nodes in 2 servers was used.
The index was split in 8 shards.
The master configuration is replicated in each node using passphraseless ssh access between master and nodes.

The ZooKeeper server was embedded into the Katta master node.

 katta.zk.properties (file) ==>zookeeper.embedded=true

Each node contains 2 shards, each shard is replicated in 2 nodes.
The Lucene sharded index contains 100 millions of documents and was stored at the Hadoop distributed files system.

Index creation

Since Katta doesn't provide any functionality to create a Lucene from the scratch, the index was built using a multithread application and the were copied into Hadoop DFS. Then, the sharded index was imported into Katta using the command line:

bin/katta addIndex occurrence hdfs://namenode:port/occurrence/shardedindex/ 2

("2" means a replication factor of 2). Importing a index into Katta is just a matter of copy the file from Hadoop and update the Index status in the ZooKeeper server, so the index creation is external factor to Katta. In a next post the "Distributed Index Creation" scenario will be analyzed as well as the technologies ElasticSearch and Solr for index creation...

Customizing the IPT

2011-07-25T10:01:00.001+02:00

One of my responsibilities as the Biodiversity Informatics Manager for Canadensys is to develop a data portal giving access to all the biodiversity information published by the participants of our network. A huge portion of this task can now be done with the GBIF Integrated Publishing Toolkit version 2 or IPT. The IPT allows to host biodiversity resources, manage their data and metadata, and register them with GBIF so they can appear on the GBIF data portal, which are all targets we want to achieve. Best of all, most management can be done by the collection managers themselves.

I have tested the IPT thoroughly and I am convinced the GBIF development team has done an excellent job creating a stable tool I can trust. This post explains how I have customized our IPT installation to integrate it with our other Canadensys websites.

Background

Our Canadensys community portal is powered by WordPress (MySQL, PHP), while our data portal - which before the IPT installation only consisted of the Database of Vascular Plants of Canada (VASCAN) - is a Tomcat application. We are using different technologies because we want to use the most adequate technology for a certain website. WordPress (or Drupal for that matter) is an excellent and easy-to-use CMS, perfect for our community portal, but not suitable for a custom made checklist website like VASCAN. To the user however, both websites look the same:

We do this by using the same HTML markup and CSS for both websites. If you want to learn HTML and CSS, w3schools provides excellent tutorials.

The HTML markup defines elements on a page (e.g. header, menu, content, sidebar, footer) and the CSS stylizes those elements (e.g. their position and color). The CSS is typically stored as one file (e.g. style.css) which is referenced in the <head> section of a page. For dynamic websites, the HTML is typically stored as different files, one for each section of a page (e.g. header.php, sidebar.php). Those files are combined as one page by the server if a page is requested. That way, changing a common element on all pages of a website (e.g. the header) can be done by changing just one file.

All of this also applies to the IPT. Here's how the IPT looks like without CSS:

Attempt 1 - Editing the CSS and logo

My first attempt at customizing the IPT was at the Experts Workshop in Copenhagen, by changing the CSS and logo only, which you can find in the /styles folder of your IPT installation:

/styles/main.css

/styles/logo.jpg

In 15 minutes, my IPT was Canadensys red and had a custom logo:

Attempt 2 - Editing the FreeMarker files

Even though my IPT now had its own branding, it was still noticeably different from the other Canadensys websites. The only way I could change that, was by editing the HTML as well. Luckily, the sections I wanted to change were all stored as FreeMarker files in the /inc folder:

/WEB-INF/pages/inc/header.ftl - the <head> section

/WEB-INF/pages/inc/menu.ftl - the header, menu and sidebar

/WEB-INF/pages/inc/footer.ftl - the footer

/WEB-INF/pages/inc/header_setup.ftl - the header during installation

I incorporated the HTML structure I use for the VASCAN website into menu.ftl (including the header, menu, container and sidebar), making sure I did not break any of the IPT functionality.

I started doing the same with main.css by replacing chunks of now unused IPT CSS with CSS I copied over from VASCAN, but I quickly realized that this wasn't the best option. Doing so would result in 2 CSS files: one for VASCAN and one for IPT, even though both web applications are under the same domain name with a lot of shared CSS. It would be easier if I only had to maintain a single stylesheet, used by both applications.

Attempt 3 - One styles folder for the data portal

I created a /common/styles folder under ROOT, where I placed my single common data portal stylesheet: /common/styles/common.css. This would be the CSS file I could use for IPT and VASCAN. I did the same for my favicon: /common/images/favicon.png.

I added a reference to both files in the header.ftl of my IPT (and VASCAN):

<link rel="stylesheet" type="text/css" href="${baseURL}/styles/main.css">

<link rel="stylesheet" type="text/css" href="http://data.canadensys.net/common/styles/common.css">

<link rel="shortcut icon" href="http://data.canadensys.net/common/images/favicon.png">

As you can see on the first line, I kept the reference to the default IPT stylesheet: ${baseURL}/styles/main.css (it's perfectly fine to reference more than one CSS file). This is where I would keep all the unaltered (=default) IPT CSS. In fact, I'm not removing anything from the default IPT stylesheet, I'm only commenting out the CSS that is unused or conflicting:

/* Unused or conflicting CSS */

The advantage of doing so, is that I now easily can compare this commented file with changes in the stylesheet of any new IPT version.

After I had done everything, my IPT now looked like this:

My IPT is now sporting the Canadensys header, footer and sidebar (only visible when editing a resource), making it indistinguishable from the other Canadensys websites. It is also using a more readable font-size (13.5px) and a fluid width.

Closing remarks

I have (re)designed quite a lot of websites, and very often I have been so frustrated with the HTML and CSS that I just started over from scratch. I didn't have that option here and it wasn't necessary either. I would like to thank the GBIF development team for creating such an easily customizable tool, with logical HTML and CSS. As a reminder, the whole customization has been done by editing only 5 files (links show default files):

/styles/main.css (custom file)

/WEB-INF/pages/inc/header.ftl

/WEB-INF/pages/inc/menu.ftl

/WEB-INF/pages/inc/footer.ftl

/WEB-INF/pages/inc/header_setup.ftl

Important: Remember that installing a new IPT version will overwrite all the customized files, so make sure to back them up! I will try to figure out a way to reapply my customization automatically after an update and post about that experience in a follow-up post. In the meantime, I hope that this post will help others in the customization of their IPT.

Working with Scientific Names

2011-07-18T11:37:00.015+02:00

Dealing with scientific names is an important regular part of our work at GBIF. Scientific names are highly structured strings with a syntax governed by a nomenclatural code. Unfortunately there are different ones for botany, zoology, bacteria, virus and even cultivar names. When dealing with names we often do not know to which code or classification it belongs to, so we need to have a code agnostic representation as much as possible. GBIF came up with a structured representation which is a compromise focusing on the most common names, primarily the botanical and zoological names which are quite similar in its basic form.

The ParsedName class

Our ParsedName class provides us with the following core properties:

 genusOrAbove
infraGeneric
specificEpithet
rankMarker
infraSpecificEpithet
authorship
year
bracketAuthorship
bracketYear

These allow us to represent regular names properly. For example Agalinis purpurea var. borealis (Berg.) Peterson 1987 is represented as

 genusOrAbove=Agalinis
specificEpithet=purpurea
rankMarker=var.
infraSpecificEpithet=borealis
authorship=Peterson
year=1987
bracketAuthorship=Berg.

or the botanical section Maxillaria sect. Multiflorae Christenson as

 genusOrAbove=Maxillaria
infraGeneric=Multiflorae
rankMarker=sect.
authorship=Christenson

Especially in botany you often encounter names with authorships for both the species and some infraspecific rank or names citing more than one infraspecific rank. These names are not formed based on rules or recommendations from the respective codes and we ignore those superflous parts. For example Agalinis purpurea (L.) Briton var. borealis (Berg.) Peterson 1987 is represented exactly the same as Agalinis purpurea var. borealis above. In case of 4 parted names like Senecio fuchsii C.C.Gmel. subsp. fuchsii var. expansus (Boiss. & Heldr.) Hayek only the lowest infraspecific rank is preserved:

 genusOrAbove=Senecio
specificEpithet=fuchsii
rankMarker=var.
infraSpecificEpithet=expansus
authorship=Hayek
bracketAuthorship=Boiss. & Heldr.

Hybrid names are evil. They come in two flavors, named hybrids and hybrid formulas.

Named hybrids are not so bad and simply prefix a name part with the multiplication sign ×, the hybrid marker, or prefix the rank marker of infraspecific names with notho. Strictly this symbol is not part of the genus or epithet. To represent these notho taxa our ParsedName class contains a property called nothoRank that keeps the rank or part of the name that needs to be marked as with the hybrid sign. For example the named hybrid Pyrocrataegus ×willei L.L.Daniel is represented as

 genusOrAbove=Pyrocrataegus
specificEpithet=willei
authorship=L.L.Daniel
nothoRank=species

Hybrid formulas such as Agrostis stolonifera L. × Polypogon monspeliensis (L.) Desf., Asplenium rhizophyllum × ruta-muraria or Mentha aquatica L. × M. arvensis L. × M. spicata L. cannot be represented by our class. The hybrid formulas in theory can combine any number of names or name parts, so its hard to deal with them. Luckily they are not very common and we can afford to live with a complete string representation in those cases. Yet another "extension" to the botanical code are cultivar names, i.e. names for plants in horticulture. Cultivar names are regular botanical names followed by a cultivar name usually in english given in single quotes. For example Cryptomeria japonica 'Elegans'. To keep track of this we have an additional cultivar property, so that:

 genusOrAbove=Cryptomeria
specificEpithet=japonica
cultivar=Elegans

In taxonomic works you often have additional information in a name that details the taxonomic concept, the sec reference most often prefixed by sensu or sec. For example Achillea millefolium sec. Greuter 2009 or Achillea millefolium sensu latu. In nomenclatoral works one frequently encounters nomenclatoral notes about the name such as nom.illeg. or nomen nudum. Both these informations are hold in our ParsedName class, for example Solanum bifidum Vell. ex Dunal, nomen nudum becomes

 genusOrAbove=Solanum
specificEpithet=bifidum
authorship=Vell. ex Dunal
nomStatus=nomen nudum

Reconstructing name strings

The ParsedName class provides us with some common methods to build a name string. In many cases you dont want the complete name with all its details, so we offer some popular name string types out of the box and a flexible string builder that you can explicitly tell which parts you want to include. The most important methods are canonicalName(): builds the canonical name sensu strictu with nothing else but the three name parts at max (genus, species, infraspecific). No rank, hybrid markers or authorship information are included. fullName(): builds the full name with all details that exist. canonicalSpeciesName(): builds the canonical binomial in case of species or below, ignoring infraspecific information

The name parser

We decided at GBIF that sharing the complete name string is more reliable than trusting already parsed names. But parsing names by hand is a very tedious enterprise, so we needed to develop some parser that can handle the vast majority of all names that we encounter. After a short experimental phase with BNF and other grammars to automatically build a parser we decided to go back to start and start something based on good old regular expressions and plain java code. The parser has evolved now for nearly 2 years now and it might be the best unit tested class we have ever written at GBIF. It is interesting to take a look at the range of names we use for testing and also the test themseves to make sure its working as expected.

Parsing names

Using the NameParser in code is trivial. Once you create a parser instance all you need to do is call the parser.parse(String name) method to get your ParsedName object. As authorships are the hardest, ie most variable part of a name we have actually implemented two parsers internally. One that tries to parse the complete string and another fallback one that ignores authorships and only extracts the canonical name. The authorsParsed flag on a ParsedName instance tells you if the simpler fallback parser has been used. If a name cannot be parsed at all an UnparsableException is thrown. This is also the case for viral names and hybrid formulas, as the ParsedName class cannot treat these names. The exception itself actually has an enumerated property that you can use to know if the exception has been caused by a virus, hybrid or other name. As of today from 10.114.724 unique name strings that we have indexed only 116.000 names couldnt be parsed and these are mostly hybrid formulas.

Normalisation

Apart from the parse method the name parser also exposes a normalisation method to normalise any whitespace, commas, brackets and hybrid markers found in name strings. The parser uses this method internally before the actual parsing takes place. The string is trimmed, only single whitespace is allowed and spaces before commas are removed while it is enforced after a comma. Similar whitespace before opening brackets is added but removed inside. Instead of the proper multiplication sign for hybrids often a simple × followed by whitespace is used which is also replaced by this method.

Parsing Webservices

GBIF is offering a free webservice API to parse names using our name parser. We use JSON for the parsed results and it accepts single names as well as batches of names. For larger input data you have to use a POST request (GET requests are restricted in length), but for few names also a simple GET request with the names url encoded in the paramter "names" is accepted. Multiple names can be concatenated with the pipe | symbol. To parse the two names Symphoricarpos albus (L.) S.F.Blake cv. 'Turesson' and Pyrocrataegus willei ×libidi L.L.Daniel the parser service call looks like this: http://ecat-dev.gbif.org/ws/parser?names=Symphoricarpos%20albus%20(L.)%20S.F.Blake%20cv.%20'Turesson'|Stagonospora%20polyspora%20M.T.%20Lucas%20%26%20Sousa%20da%20Camara%201934 For manual usages we also provide a simple web client to this service that provides a form to enter names to be parsed and also accepts files with one name per line for upload. It is available as part of our tools collection at http://tools.gbif.org/nameparser/.

Source Code

All the code is part of a small java library that we call ecat-common. It is freely available under Apache 2 licensing as most of our GBIF work and you are invited to use our code at http://code.google.com/p/gbif-ecat/source/browse/trunk/ecat-common/, download the latest jar from our maven repository or include it in your maven dependencies like this:

 <repositories>
<repository>
<id>gbif-all</id>
<url>http://repository.gbif.org/content/groups/gbif</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>org.gbif</groupId>
<artifactId>ecat-common</artifactId>
<version>1.5.1-SNAPSHOT</version>
</dependency>
</dependencies>

Are you the keymaster?

2011-07-08T11:44:00.013+02:00

As I mentioned previously I'm starting work on evaluating HBase for our occurrence record needs. In the last little while that has meant coming up with a key structure and/or schema that optimizes reads for one major use case of the GBIF data portal - a user request to download an entire record set, including raw records as well as interpreted. The most common form of this request looks like "Give me all records for ", eg "Give me all records for Family Felidae".

So far I'm concentrating more on the lookup and retrieval rather than writing or data storage optimization, so the schema I'm using is two column families, one for verbatim columns, one for interpreted (for a total of about 70 columns). The question of which key to use for HTable's single indexed column is what we need to figure out. For all these examples we assume we know the backbone taxonomy id of the taxon concept in question (ie Family Felidae is id 123456).

Option 1
Key: native record's unique id

Query style: The simplest way of finding all records that belong to Family Felidae is scan all of them, and check against the Family column from the interpreted column family. The code looks like this:

    HTable table = new HTable(HBaseConfiguration.create(), tableName);

    byte[] cf = Bytes.toBytes(colFam);

    byte[] colName = Bytes.toBytes(col);

    byte[] value = Bytes.toBytes(val);

    Scan scan = new Scan();

    ResultScanner scanner = table.getScanner(scan);

    for (Result result : scanner) {

      byte[] testVal = result.getValue(cf, colName);

      if (Bytes.compareTo(testVal, value) == 0) doSomething;

    }

Because this means transferring all columns of every row to the client before checking if it's even a record we want, it's incredibly wasteful and therefore very slow. It's a Bad Idea.

Option 2
Key: native record's unique id

Query style: HBase provides a SingleColumnValueFilter that executes our equality check on the server side, thereby saving the transfer of unwanted columns to the client. Here's the code:

    HTable table = new HTable(HBaseConfiguration.create(), tableName);
    byte[] cf = Bytes.toBytes(colFam);
    byte[] colName = Bytes.toBytes(col);
    byte[] value = Bytes.toBytes(val);

    SingleColumnValueFilter valFilter = new SingleColumnValueFilter(cf, colName, CompareFilter.CompareOp.EQUAL, value);
    valFilter.setFilterIfMissing(true);

    Scan scan = new Scan();
    scan.setFilter(valFilter);
    ResultScanner scanner = table.getScanner(scan);

This is about as good as it gets until we start getting clever :)

Option 3
Key: concatentation of nub-taxonomy "left" with native record's unique id

Query style: We know that a taxonomy is a tree, and our backbone taxonomy is a well behaved (ie true) tree. We can use nested sets to make our "get all children of node x" query much faster, which Markus realized some time ago, and so thoughtfully included the left and right calculation as part of the backbone taxonomy creation. Individual occurrences of the same taxon will share the same backbone taxonomy id, as well as the left and right. One property of nested sets not mentioned in the wikipedia article is that when the records are ordered by their lefts, the query of "give me all records where left is between parent left and parent right" becomes "give me all rows starting with parent left and ending with parent right", which in HBase terms is much more efficient since we're doing a sequential read from disk without any seeking. So we build the key as leftId_uniqueId, and query as follows (note that startRow is inclusive and stopRow is exclusive, and we want exclusive on both ends):

    HTable table = new HTable(HBaseConfiguration.create(), tableName);

    Scan scan = new Scan();

    scan.setStartRow(Bytes.toBytes((left + 1) + "_"));

    scan.setStopRow(Bytes.toBytes(right + "_"));

    ResultScanner scanner = table.getScanner(scan);

Which looks pretty good, and is in fact about 40% faster than Option 2 (on average - depends on the size of the query result). But on closer inspection, there's a problem. By concatenating the left and unique ids with an underscore as separator, we've created a String, and now HBase is doing its usual lexicographical ordering, which means our rows aren't ordered as we'd hoped. For example, this is the ordering we expect:

1_1234

2_3458

3_3298

4_9378

5_3435

10_5439
100_9763

but because these are strings, HBase orders them as:

1_1234

10_5439

100_9763 

2_3458

3_3298

4_9378

5_3435

There isn't much we can do here but filter on the client side. For every key, we can extract the left portion, convert to a Long, and compare it to our range, discarding those that don't match. It sounds ugly, and it is, but it doesn't add anything appreciable to the processing time, so it would work.

Except that there's a more fundamental problem - if we embed the left in our primary key, it only takes one node added to the backbone taxonomy to force an update in half of all the lefts (on average) which means all of our primary keys get rewritten. At 300 million records and growing, that's not an option.

Option 4
Key: native record's unique id
Secondary index: left to list of unique ids

Query style: Following on from Option 3, we can build a second table that will serve as a secondary index. We use the left as a numeric key (which gives us automatic, correct ordering) and write each corresponding unique occurrence id as a new column in the row. Then we can do a proper range query on the lefts, and generate a distinct Get for each distinct id. Unfortunately building that index is quite slow, and is still building as I write this, so I haven't been able to test the lookups yet.

For those keeping score at home, I'm using Hbase 0.89 (from CDH3b4) which doesn't have built in secondary indexes (which 0.19 and 0.20 did).

I'll write more when I've learned more, and welcome any tips or suggestions you might have to aid in my quest!

The organisational structure and the endorsement process - if you're an IPT administrator

2011-06-30T15:14:00.014+02:00

During the Expert Workshop last week in Copenhagen, we had a session talking about configuring IPT to reflect different organisational structures. I think it's worth to explain about that part as a blog post here, since some of our readers would like to help deploy IPT in the GBIF Network. It's usually started by questions like this:

Why am I asked for a password of the organisation that I choose to register IPT? Why am I asked again when I want to add an additional organisation?

The short answer is, by having the password of the organisation, that means you have got the permission from that organisation and the organisation is aware of the fact that you're registering an IPT against it.

So, why is this the way of registering an organisation?

The organisational structure

Remember the GBIF Network is not only a common pool of sharing biodiversity data, to form such a pool, it's also the social network in which biodiversity data publishers interact. IPT, serves as the technical skeleton of the network, needs to tie to the organisational structure in order to properly accredit institutions or individuals, by helping to reflect those relationships among the organisations, hosted resources, IPTs and endorsing GBIF nodes/participants. The relationship can be seen on the GBIF Registry.

Take VertNet Hosting as an example. VertNet Hosting is an IPT installation that hosts data resources authored by those users in the IPT. In the second half of its page on the Registry, you see:

VertNet Hosting publishes uafmc_fish

University of Kansas Biodiversity Research Center has technical installation VertNet Hosting

Which means, this IPT has a public resource called uafmc_fish, which has been registered to the GBIF Network, and this IPT is registered against University of Kansas Biodiversity Research Center, which is the hosting organisation of IPT.

If you click the uafmc_fish link in the Registry page, it says:

University of Arkansas Collections Facility, UAFMC has data resource uafmc_fish

That means the uafmc_fish resource has been registered against University of Arkansas Collections Facility, UAFMC, which should have been added to the VertNet hosting IPT to be available for users to choose from, other than University of Kansas Biodiversity Research Center.

So the administrator of the VertNet Hosting IPT, in this case Ms. Laura Russell, had been asked twice the different passwords she needed to register her IPT and add another organisation.

The activities can be summarised as this graphic:

Which says, with Vertnet Hosting (IPT) registered against University of Kansas Biodiversity Research Center, uafmc_fish can be hosted by the IPT but tied to University of Arkansas Collections Facility, UAFMC. This means an organisation doesn't necessarily need to have the capacity to install IPT in order to host published resources. Any other IPT, with the agreement of hosting organisations, can host resources of others. This is why here the password is required.

The relationships of these units then are reflected on the Registry as:

By this design the relationship and accreditation of the GBIF Network is maintained.

Now, beyond these graphics, if you click University of Kansas Biodiversity Research Center and University of Arkansas Collections Facility, UAFMC in the Registry, you'll these two statements individually in each page:

USA endorses University of Kansas Biodiversity Research Center

USA endorses University of Arkansas Collections Facility, UAFMC

That means, these 2 organisations are endorsed by USA, which is a member of GBIF, and since they are endorsed, they are therefore available in the organisation drop-down list of IPT.

The endorsement process

Chances are, you're looking for your organisation in that list and you are pretty sure it's not there. What should you do?

Either you're registering IPT, or one of your user is requesting an organisation that is not available yet, you should try talk to administration level people and seek to get endorsed by a GBIF member. Normally, ➊ you or the representative of your institution write to helpdesk@gbif.org, provide some background information and at least a technical contact, ➋ the helpdesk will look for appropriate node for you to get endorsed. Upon ➌ positive feedbacks from an endorsing node, ➍ the helpdesk will inform you the availability of your organisation and the password.

This process runs administratively because we rely on the social level to ensure the responsibility for the registered IPT and published resources. This also makes the accreditation goes to correct persons or organisations.

Hope this helps IPT experts.

Synchronizing occurrence records

2011-06-24T21:10:00.002+02:00

This post should be read in the line of Tim’s post about Decoupling Components, as it takes for granted some information written there.

During the last week, I’ve been learning/working with some technologies that are related to the decoupling of components we want to accomplish. Specifically, I’ve been working with the Synchronizer component of the event driven architecture Tim described.

Right now, the synchronizer takes the responses from the resources and gets those responses into the occurrence store (MySQL as of today, but not final). But it has more to it: The responses from the resources come typically from DiGIR, TAPIR and BioCASe providers which render their responses into XML format. So how does all this data ends up in the occurrence store? Well, fortunately my colleague Oliver Meyn wrote a very useful library to unmarshall all these XML chunks into nice and simple objects, so on my side I just have to worry about calling all those getter methods. Also, the synchronizer acts as a listener to a message queue , queue that will store all the resource responses that need to be handled. All the queue’s nuts & bolts were worked out by Tim and Federico Méndez. So yes, it has been a nice collaboration from many developers inside the Secretariat and it’s always nice to have this kind of head start from your colleagues :)

So, getting back to my duties, I have to take all these objects and start populating the occurrence target store taking some precautions (e.g.: not inserting duplicated occurrence records, checking that some mandatory fields are not null and other requirements).

For now, it’s in development mode, but I have managed to make some tests and extract some metrics that show current performance and definitely leaves room for improvement. For the tests, first the message queue is loaded with some responses that need to be attended and afterwards I execute the synchronizer which starts populating the occurrence store. All these tests are done on my MacBook Pro, so definetely response times will improve on a better box. So here are the metrics:

Environment:

MacBook Pro 2.4 GHz Core2 Duo (4GB Memory)
Mac OS X 10.5.8 (Leopard)
Message Queue & MySQL DB reside on different machines, but same intranet.
Threads: synchronizer spawns 5 threads to attend the queue elements.
Message queue: loaded with 552 responses (some responses are just empty to emulate a real world scenario).
Records in total: 70,326 occurrence records in total in all responses

Results Test 1 (without filtering out records):

Extracting responses from queue
Unmarshalling
Inserting into a MySQL DB
202022 milliseconds (3 min, 22 secs)

Results Test 2 (filtering out records):

Extracting from queue
Unmarshalling
Filtering out records (duplicates, mandatory fields, etc)
Inserting into MySQL DB
over 30 minutes... (big FAIL)

So, as you see there is MUCH room for improvement. As I have just joined this project in particular, I need to start the long and tedious road of debugging why the huge difference, obviously the filtering out process needs huge improvement. Obvious solutions come to mind: increasing threads, improve memory consumption and other not so obvious solutions. I will try to keep you readers posted about this, and hopefully some more inspiring metrics, and for sure in a better box.

I hope to communicate further improvements later, see you for now.

Querying Solr using a pure AJAX application

2011-06-20T15:20:00.021+02:00

This is the third (and final) post related to the GBIF Metacatalogue Project. The first 2 were dedicated to explain how the data is harvested and how that information is stored in Apache Solr. Those post can be consulted in:

One the nicest features of Solr is that most of its functionalities are exposed via Rest API. This API can be used for different operations like: delete documents, post new documents and more important to query the index. In cases when the index is self-contained (i.e: doesn't depend of external services or storages to return valuable information) a very thin application client without any mediator is viable option. In general terms, "mediator" is a layer that handles the communication between the user interface and Solr, in some cases (possibly) that layer manipulates the information before send it to user interface. Metadata Web application is a perfect example of the scenario just described: it's basically an independent storage of documents that can be used to provide free and structured search. All the information is collected from several sources and then is store in the Solr index, even the full XML documents are stored in the Solr Index. AJAX Solr is a Javascript framework that facilitates querying a Solr server and the display of results in a Web Application; to implement the remote communication only requires a way of sending requests to Solr, in our case, we used JQuery. In AJAX Solr the information is displayed to the end-user by widgets like: facets widgets, list of results, cloud widgets, etc. Widgets

Widgets

In this context widgets are user interface componenents whose functionality doesn't depend on other widgets, each one has an specific responsabilty. All the communication (between Solr and the UI) is handled by Manager whose main responsability is send the requests and communicates the responses to the widgets. The image below shows some widgets examples: From the implementation point of view the code below shows how the manager is created and the way of attach widgets to it:

$(function () {
  Manager = new AjaxSolr.Manager({
    solrUrl: solrServerUrl
  });
  /*Adds a listener widgets to the Manager*/
  Manager.addWidget(new AjaxSolr.ResultWidget({
    id: 'result',
    target: '#docs' //<-- element where the result will be displayed     }));
 Manager.addWidget(new AjaxSolr.PagerWidget({    
     id: 'pager',    
     target: '#pager',    
     prevLabel: '<',    
     nextLabel: '>...

Then, we "simply" add the desired params to perform the query:

  Manager.store.addByValue('facet.field', 'providerExact');
  Manager.store.addByValue('facet.date', 'endDate');
  Manager.store.addByValue('q', '*:*');
  Manager.doRequest();

Other libraries

Some other components were used in order to provide a better user experience, those are:

JQuery/JQuery UI (http://jquery.com/): AJAX Solr requires a library to implement the AJAX requests. JQuery was chosen for this purpose. Additionally, several JQueryUI widgets are extensively used for a richer user experience.
SyntaxHighlighter(http://alexgorbatchev.com/SyntaxHighlighter/): this is a code syntax highlighter developed in JavaScript, This component is used for displaying the XML view of a metadata document.

The prototype application is available here; this application is 99.9% free of server-side code, there's only one line of code with server dependency, and is for indicate the Solr server url:

var solrServerUrl = <%="'" + config.getServletContext().getInitParameter("solrServerUrl") + "'"%>;

However, that line can be modified easily to deploy the same application in other web server technology rather than a Servlet container (Tomcat, Jetty, etc.).

Simple wallboard display with Scala and Lift at GBIF

2011-06-17T14:28:00.000+02:00

This week we hit 300 million indexed occurrence records. As you can see in the picture we have got a monitor set up that shows us our current record count. It started as an idea a few weeks ago but while at the Berlin Buzzwords conference (we were at about 298 million then) I decided it was time to do something about it.

I've been playing around with Scala a bit in the last few months so this was a good opportunity to try Lift, a web framework written in Scala. In the end it turns out that very little code was needed to create an auto-updating counter. There are three components:

We've got a DBUpdater object that uses Lift's Schedule (used to be called ActorPing which caused some confusion for me) to update its internal count of raw occurrence records every ten seconds. The beauty is that there is just one instance of this no matter how many clients are looking at the webpage.
The second part is a class that acts as a Comet adaptor called RawOccurrenceRecordCount which waits for updates from the DBUpdater and passes these on to the clients.
The last part is the Bootstrap code that schedules the first update of the DBUpdater and sets up the database connection and other stuff.

To get to this point, though, took quite some time as I have to say that the documentation for Lift is very lacking especially in explaining the basic concepts (I've read Simply Lift, bits and pieces in the Wiki and am halfway through Exploring Lift) for beginners like me. I'm really looking forward to Lift in Action and really hope it serves as a better introduction than the currently available documentation.

That said I liked the end product very much and I hope to be able to extend the work a bit more to incorporate more stats for our wallboard display but so far I haven't managed to call JavaScript functions from my Comet Actor. That's next on my list. Ideas for a wallboard are piling up and I hope to be able to continue doing it in Lift and Scala.

Buzzword compliance

2011-06-15T12:05:00.080+02:00

Over the last few years a number of new technologies have emerged (inspired largely by Google) to help wrangle Big Data. Things like Hadoop, HBase, Hive, Lucene, Solr and a host of others are becoming the "buzzwords" for handling the type of data that we at the secretariat are working with. As a number of our previous posts here have shown, the GBIF dev team is wholeheartedly embracing these new technologies, and we recently went to the Berlin Buzzwords conference (as a group) to get a sense of how the broader community is using these tools.

My particular interest is in HBase, which is a style of database that can handle "millions of columns and billions of rows". Since we're optimistic about the continued growth of the number of occurrence records indexed by GBIF, it's not unreasonable to think about 1 billion (10^9) indexed records within the medium-term, and while our current MySQL solution has held up reasonably well so far (now closing in on 300 million indexed records) it certainly won't handle an ever-growing future.

I'm now in the process of evaluating HBase's ability to respond to the kinds of queries we need to support, particularly downloads of large datasets corresponding to queries in the data portal. As in most databases, schema design is quite important in HBase, as is the selection of a "primary key" format for each table. A number of the talks at Berlin Buzzwords addressed these issues and I was very happy to hear from some of the core contributers to HBase and their conclusion that figuring out the right setup for any particular problem is far from trivial. Notable among the presenters were Jean-Daniel Cryans from StumbleUpon (a fellow Canadian, woot!) and Jonathan Gray from Facebook (with luck their slides will be up at the Buzzwords slides page soon). Jonathan's presentation especially gives a sense of what HBase is capable of given the truly huge amount of data Facebook drives through it (all of Facebook's messaging is held in HBase).

Apart from learning a number of new techniques and approaches to developing with HBase, more than anything I'm excited to dive into the details knowing such a strong and supportive community is out there to help me when I get stuck. You can follow along my testing and deliberations on the wiki page for our occurrence record project.

Getting started with Avro RPC

2011-06-09T22:59:00.011+02:00

Apache Avro is a data exchange format started by Doug Cutting of Lucene and Hadoop fame. A good introduction to Avro is on the cloudera blog so an introduction is not the intention of this post.

Avro is surprisingly difficult to get into, as it is lacking the most basic "getting started" documentation for a new-comer to the project. This post serves as a reminder to myself of what I did, and hopefully to help others get the hello world working quickly. If people find it useful, let's fill it out and submit it to the Avro wiki!

Prerequisites: knowledge of Apache Maven

Start by adding the Avro maven plugin to the pom. This is needed to compile the Avro schema definitions into the Java classes.

<plugin>
  <groupId>org.apache.avro</groupId>
  <artifactId>avro-maven-plugin</artifactId>
  <version>1.5.1</version>
  <executions>
    <execution>
      <id>schemas</id>
      <phase>generate-sources</phase>
      <goals>
        <goal>schema</goal>
        <goal>protocol</goal>
        <goal>idl-protocol</goal>
      </goals>
      <configuration>
        <excludes>
          <exclude>**/mapred/tether/**</exclude>
        </excludes>
        <sourceDirectory>${project.basedir}/src/main/avro/</sourceDirectory>
        <outputDirectory>${project.basedir}/src/main/java/</outputDirectory>
        <testSourceDirectory>${project.basedir}/src/test/avro/</testSourceDirectory>
        <testOutputDirectory>${project.basedir}/src/test/java/</testOutputDirectory>
      </configuration>
    </execution>
  </executions>
</plugin>

Now add the dependency on Avro and the Avro IPC (Inter Process Calls)

<dependency>
  <groupId>org.apache.avro</groupId>
  <artifactId>avro</artifactId>
  <version>1.5.1</version>
</dependency>
<dependency>
  <groupId>org.apache.avro</groupId>
  <artifactId>avro-ipc</artifactId>
  <version>1.5.1</version>
</dependency>

Now we create the Avro Protocol file, which defines the RPC exchange. This file is stored in /src/main/avro/nublookup.avpr and looks like so:

{"namespace": "org.gbif.ecat.ws",
 "protocol": "NubLookup",
 "types": [
     {"name": "Request", "type": "record",
      "fields": [
        {"name": "kingdom", "type": ["string", "null"]},
        {"name": "phylum", "type": ["string", "null"]},
        {"name": "class", "type": ["string", "null"]},
        {"name": "order", "type": ["string", "null"]},
        {"name": "family", "type": ["string", "null"]},
        {"name": "genus", "type": ["string", "null"]},
        {"name": "name", "type": ["string", "null"]}
      ]
     },
     {"name": "Response", "type": "record",
      "fields": [
        {"name": "kingdomId", "type": ["int", "null"]},
        {"name": "phylumId", "type": ["int", "null"]},
        {"name": "classId", "type": ["int", "null"]},
        {"name": "orderId", "type": ["int", "null"]},
        {"name": "familyId", "type": ["int", "null"]},
        {"name": "genusId", "type": ["int", "null"]},
        {"name": "nameId", "type": ["int", "null"]}
      ]
     }  
 ],
 "messages": {
     "send": {
         "request": [{"name": "request", "type": "Request"}],
         "response": "Response"
     }
 }
}

This protocol defines an interface called NubLookup, that takes a Request and returns a Response. Simple stuff.

From the command line issue a compile:

$mvn compile

This will generate into src/main/java and the package I declared in the .avpr file (org.gbif.ecat.ws in my case).

Now we can test it using a simple Netty server which is included in the Avro dependency:

public class Test {
  private static NettyServer server;
  
  // A mock implementation
  public static class NubLookupImpl implements NubLookup {
    public Response send(Request request) throws AvroRemoteException {
      Response r = new Response();
      r.kingdomId=100;
      return r;
    }
  }
  
  public static void main(String[] args) throws IOException {
    server = new NettyServer(new SpecificResponder(
        NubLookup.class, 
        new NubLookupImpl()), 
        new InetSocketAddress(7001)); 

      NettyTransceiver client = new NettyTransceiver(
          new InetSocketAddress(server.getPort()));
      
      NubLookup proxy = (NubLookup) SpecificRequestor.getClient(NubLookup.class, client);
      
      Request req = new Request();
      req.name = new Utf8("Puma");
      System.out.println("Result: " + proxy.send(req).kingdomId);

      client.close();
      server.close();
  }
}

I am evaluating Avro to provide the high performance RPC chatter for lookup services while we process the content for the portal. I'll blog later about the performance compared to the Jersey REST implementation currently running.

MySQL: A speed-up of over 9000 times using partitioning

2011-06-03T19:05:00.003+02:00

I wanted to write about a MySQL performance optimization using partitioning as I recently applied it to the Harvesting and Indexing Toolkit’s (HIT) log table. The log table was already using a composite index (indexes on multiple columns), but as this table grew bigger and bigger (>50 million records) queries were being answered at a turtle’s pace. To set things up, imagine that in the HIT application there is a log page that allows the user to tail the latest log messages in almost real time. Behind the scenes, the application is querying the log table every few seconds for the most recent logs, and the effect is a running view of the logs. The tail query used looks like this:

mysql> select * from log_event where id >= ‘latest id’ and datasource_id = ‘datasource_id’ and level >= ‘log level’ order by id desc;

In effect this query asks: “give me the latest logs for datasource with id X having having at least a certain log level”. Partitioning basically divides a table into different portions that are stored and can be queried separately. The benefit is that if a query only has to hit a small portion instead of the whole table, it can be answered faster. There are different ways that you can partition tables in MySQL, and you can read about them all in the MySQL reference manual. I first experimented using Key partitioning using the table ID. Unfortunately, because different logs for a datasource could be spread across different partitions, the tail query would have to hit all partitions. To check how many partitions the query hits, I used the following query:

mysql> explain partitions select * from … ;

This resulted in an even slower response than without partitioning, so Tim thought about it from a different angle. He discovered a nice solution using Range partitioning by datasource ID instead. This way the table would get divided into ranges of datasources that are contiguous but not overlapping. A range size of 1000 was used, so the 1st partition would contain all logs for datasources with IDs between 0 – 999, the 2nd partition would contain all logs for datasources with IDs between 1000 – 1999 and so on. Part of the command used to apply Tim’s partitioning strategy (having 36 partitions) is displayed below:

ALTER TABLE log_event
ADD PRIMARY KEY(id,bio_datasource_fk)
PARTITION BY RANGE (bio_datasource_fk) (
    PARTITION p0 VALUES LESS THAN (1000),
    PARTITION p1 VALUES LESS THAN (2000),
    .
    .
    PARTITION p36 VALUES LESS THAN MAXVALUE
);

Checking how many partitions the tail query would hit, I confirmed that it only ever uses a single partition. The result was impressive, and initial tests resulted in a speed-up of over 9000 times! Important to note is that the primary key must include all fields in the partition. Therefore because we were partitioning using the datasource id, this field had to be included in the primary key before partitioning would work. Also, an index on the id was also added to further optimize the query - why not right? The speed-up might be dramatic now, but as more log messages get written to a partition and it starts to swell, I envisage having to either delete old logs or repartition the table again using smaller range sizes in order to sustain good performance. There is a trade-off between the number of partitions and performance, so some tweaking is needed in every case I guess. Lastly, I’ll reiterate that improper partitioning can actually make things worse. Perhaps it could work for you too, but please apply with caution.

Ordered updates with Postgres

2011-06-01T19:55:00.010+02:00

When updating a postgres table you sometimes want the update to happen in a specific order. For example I found myself in a situation when I wanted to assign new sequential ids to records in the alphabetical order given by a text string column.

With postgres 8.4 the solution using an updateable, ordered view didn't work (anymore?). After experimenting a little I found that clustering a table according to the desired order is a simple solution that works exactly as hoped for. Clustering changes the actual order of the table data instead of only adding a new index. And apparently postgres uses this native order for updates.

CREATE TABLE idupd (id int, name varchar(128));

CREATE INDEX idupd_idx ON idupd (name);

CLUSTER idupd USING idupd_idx;

CREATE SEQUENCE idupd_seq;

SELECT setval('idupd_seq', 100);

UPDATE idupd set id=nextval('idupd_seq');

Decoupling components

2011-05-30T16:20:00.001+02:00

Recent blog posts have introduced some of the registry and portal processing work under development at GBIF. Here I'd like to introduce some of the research underway to improve the overall processing workflows by identifying well defined components and decoupling unnecessary dependencies. The target being to improve the robustness, reliability and throughput of the data indexing performed for the portal.

Key to the GBIF portal is the crawling, processing and indexing of the content shared through the GBIF network, which is currently performed by the Harvesting and Indexing Toolkit (HIT). Today the HIT operates largely as follows:

Synchronise with the registry to discover the technical endpoints
Allow the administrator to schedule the harvest and process of an endpoint, as follows:

Initiate a metadata request to discover the datasets at the endpoint
For each resource initiate a request for the inventory of distinct scientific names
Process the names into ranges
Harvest the records by name range
Process the harvested responses into tab delimited files
Synchronise the tab delimited files with the database "verbatim" tables
Process the "verbatim" tables into interpreted tables

Logically the HIT is depicted:

Some of the limitations in this model include:

The tight coupling between the HIT and the target DB mean we need to stop the harvesting when we are going to perform very expensive processing on the database
Changes to the user interface for the HIT require the harvester to be stopped
The user interface console is driven by the same machine that is crawling, meaning the UI becomes unresponsive periodically.
The tight coupling between the HIT and the target DB preclude the option of storing in multiple datastores (as is current desire as we investigate enriching the occurrence store)

The HIT can be separated into the following distinct concerns:

An administration console to allow the scheduling, oversight and diagnostics of crawlers
Crawlers that harvest the content
Synchronisers that interpret and persist the content into the target datastores

An event driven architecture would allow this to happen and overcome the current limitations. In this model, components can be deployed independently, and message each other through a queue when significant events occur . Subscribers to the queue determine what action if any to take on a per message basis. The architecture under research is shown:

In this depiction, the following sequence of events would occur:

Through the Administration console, the administrator schedules the crawling of a resource.
The scheduler broadcasts to the queue that the resource is to be crawled rather than spawning a crawler directly.
When capacity allows, a crawler will act on this event and crawl the resource, storing to the filesystem as it goes. On each response message, the crawler will broadcast that the response is to be handled.
Synchronizers will act on the new response messages and store them in the occurrence target stores. In the above depiction, there are actually 2 target stores, each of which would act on the message indicating there is new data to synchronise.

This architecture would have significant improvements to the existing setup. The crawlers would only ever stop when bug fixing in the crawlers themselves occurs. Different target stores can be researched independently of the crawling codebase. The user interface for the scheduling can be developed, and redeployed without interrupting the crawling.

As an aside, during this exercise we are also investigating improvements in the following:

The HIT (today) performs the metadata request, but does NOT update the registry with the datasets that are discovered, only the data portal. The GBIF registry is "dataset aware" for the datasets served through the Integrated Publishing Toolkit and ultimately we intend the registry to be able to reconcile the multiple identifiers associated with a dataset. For example, it should be possible in the future to synchronise with the like of the Biodiversity Collections Index which is a dataset level registry.
The harvesting procedure is rather complex, with many points for failure; it involves inventories of scientific names, processing into ranges of names and a harvest based on the name ranges. Early tests suggest a more simpler approach of discrete name ranges [Aaa-Aaz, Aba-Abz ... Zza Zzz] yield better results.

Watch this space for results of this investigation...

The Phantom Records Menace

2011-05-27T15:31:00.007+02:00

For a data administrator, going to the web test interface of data publisher can be incredibly useful if one needs to compare the data that was collected using the Harvesting and Indexing Toolkit: HIT and what is available from the publisher. In a perfect world transfer of records would happen without a glitch but when we eventually get less (or more!) than we asked for the search/test interfaces can be a real help (for instance the PyWrapper quering utilities)

Sometimes GBIF will index a resource that for no apparent reason turns in fewer records than what is expected from the line count that the HIT performs automatically. In this particular case there appears to be several identical records on top of that – which we are made aware of by the HIT that warns us that there are multiple records with the same “holy triplet”: Institution code, collection code and catalogue number.

Now what happens when a request goes out for this name range: Abies alba Mill. - Achillea millefolium L. followed by a request for Achillea millefolium agg. - Acinos arvensis (Lam.) Dandy? Those of you with good eyesight will have spotted that the request asks for Achillea millefolium L. before Achillea millefolium agg. This is because this particular instance or configuration of pywrapper returns a name range that is sorted according to the character values you find in UTF-8 and ASCII/Latin-1 which orders all upper-case characters before the lower-case ones. Whether this is an artifact of the underlying database system or the pywrapper itself, or even a specific version of the wrapper is not yet known, but the scenario exists today and consumers should be aware of this. The HIT then builds requests based on this name range and if the requests by chance divide between “Achillea millefolium L. and Achillea millefolium agg.” you will be receiving overlapping responses - that is two responses that contain parts of each other’s records – because the response is not based on a BINARY select statement and therefore returns the records alphabetically sorted without giving precedence to upper-case letters. This behavior can be replicated by going to the pywrapper interface and searching these name ranges. Fortunately the HIT removes the redundant records during the synchronizing process. However, the record count is based on the line count at the point where the records are received from the access point. This is why the record count in the HIT is inflated and as you see this kind of error can be am bit difficult to spot.

2011 GBIF Registry Refactoring

2011-05-23T16:50:00.003+02:00

For the past couple of months, I have been working closely with another GBIF developer (and also fellow blog writer) Federico Mendez, on development tasks on the GBIF's Registry application. This post provides an overview of the work being done on this matter.

First, I will like to explain the nuts and bolts of the current Registry application (the one online), and then the additions/modifications it has "suffered" during 2011 (modifications have not been deployed). As stated on The evolution of the GBIF Registry blog post, in 2010 the Registry entered a new stage on its development by moving to a single DB, enhanced web service API, and a web user interface. On top of this, an admin-only web interface was created so that we could do internal curation of the data inside the Secretariat.

Hibernate's framework was chosen as the preferred persistence framework and the Data-Access-Object (DAO) classes were coded with the HQL necessary to provide an interface to the Hibernate persistence mechanism. The Business tier consisted of several Manager classes that relied on the DAOs to get the required data. These Managers also were the ones responsible for populating the Data-Transfer-Objects (DTOs) so that they could be passed to the Presentation tier. This last tier made use of plain Java Server Pages (JSPs), along with JQuery, Ajax, CSS among others. Then, at the start of this year 2011, a decision was made to improve the application's underlying implementation in some aspects:

Use of MyBatis data mapper framework. This involved walking away from Hibernate's Object-Relational Mapping (ORM) approach. Our use of Hibernate involved HQL, adding an extra latency component when converting HQL to SQL, but in MyBatis we use direct SQL mapped statements making it quicker to access the DB. (I will share some benchmarking on my next blog post, to justify this remark)

We found out that using a DTO pattern represented somewhat of an overkill for an application that didn't had such complexity at the model level. We could trim some code complexity by passing the model objects straight to the presentation tier. So we did, and all DTOFactories & DTO objects were gone.

Several codebase improvements were introduced mainly by Federico, cutting down huge amounts of lines and making it easier to add new functionality with less effort (e.g. heavy use of Java's generics)

At the web service level, the Struts2 Rest plugin was replaced by the Jersey library. I personally found the Struts2 Rest plugin lacking documentation (1 year ago) so the Registry's use of it was kind of ad hoc. My next blog post will include more reasoning about this decision.

We now make use of the Guice dependency injection framework. Beforehand, we were making use of Spring's ability for this. Also, these injections are made through annotations now; with Spring we were using XML based injection.

The Registry project is now divided into different libraries. In particular:

registry-core: Business & persistence logic
registry-web: All related to the web application (Struts2)
registry-ws: All the web service stuff
There are also some libraries Federico has created to manage the interaction between the Registry and all technical installations (DiGIR, Tapir, BioCase, etc) of those publishers sharing data with GBIF. These are extremely important libraries as they are the ones who keep the Registry up to date.

(2011 refactoring)

I must emphasize again that these changes are not yet deployed, this in an ongoing project but if you are really interested to see the progress being made, please feel free to visit the project's site. Also, these changes won't affect the current web services API, or the DB structure. Merely the changes are to improve the underlying codebase.

Indexing bio-diversity metadata using Solr: schema, import-handlers, custom-transformers

2011-05-19T20:14:00.007+02:00

This post is the second part of OAI-PMH Harvesting at GBIF. In that blog was explained how different OAI-PMH services are harvested. The subject of this post is introduce the overall architecture of the index created using the information gathered from those services. Let's start by justifying why we needed a metadata index at GBIF, one of the main requirements we had was allow "search datasets by a end-users". To enable this, the system provides two main search functionalities: Full Text Search and Advanced Search. For both functionalities the system will display a list of data sets containing the following information: title, provider, description (abstract) and hyperlink to view the full metadata document in the original format (DIF, EML, etc.) provided by the source; all that information was collected by the harvester. The results of any search had to be displayed with two, amog others, specific features: highlight the text that matched the searh criteria, and group/filter the results by facets: providers, dates and OAI-PMH services. In order to provide nice searh features we couldn't leave the responsability to the capabilities of a database, so we decided implement a index support the searh requirements by building a index capable of facilitate the user needs. An index is like a single-table database without any support for relational queries with only purpose to support search and not be the primary source of data. The structure of the index is de-normalized and contain just the data needed to be searched. The index was implemented using Solr which is an open source enterprise search. It has numerous other features such as search result highlighting, faceted navigation, query spell correction, auto-suggest queries and “more like this” for finding similar documents. The metadata application stores a subset of the available information in the metadata documents as Solr fields and a special field (fullText) is used to store the whole XML document to enable full text search, the schema fields are:

id: full file name is used for this field.
title: title of the dataset,
provider: provider of the dataset
providerExact: same as the previous field, but uses String data type for facets and exact match search
description: description or abstract of the dataset
beginDate: begin date of the temporal coverage of dataset
endDate: end date of the temporal coverage of dataset, when the input format only supports one dataset date, beginDate and endDate will contain the same value
westBoundingCoordinate: Geographic west coordinate
eastBoundingCoordinate: Geographic east coordinate
northBoundingCoordinate: Geographic north coordinate
southBoundingCoordinate: Geographic south coordinate
fullText: The complete text of the XML metadata document
externalUrl: Url containing specific information about the dataset; in the case of the
serverId: Id of the source OAI-PMH Service; this information is taken from the file system structure and is used for the facets search.

The XML documents gathered by the harvester are imported into Solr using data import handlers for each input format (EML, DIF,etc.). An example of one of the data import handlers is the following used for index dublin core xml files:

The data import handlers are implemented using three main features available in Solr:

FileDataSource: allows fetching content from files on disk.
FileListEntityProcessor: an entity processor used to enumerate the list of files.
XPathEntityProcessor: used to index the XML files, it allows defining of Xpath expressions to retrieve specific elements.
PlainTextEntityProcessor: reads all content from the data source into a single field; this processor is used to import the whole XML file into one field.
DateFormatTransformer: parses date/time strings into java.util.Date instances; it is used for the date fields.
RegexTransformer: helps in extracting or manipulating values from fields (from the source) using Regular Expressions.
TemplateTransformer: used to overwrite or modify any existing Solr field or to create new Solr fields; it is used to create the id field.
org.gbif.solr.handler.dataimport.ListDateFormatTransformer: this is a custom transformer to handle non-standard date formats that are common in input dates; it can handle dates with formats like: 12-2010, 09-1988, and (1998)-(2000). It has three important attributes: i) separator that defines the character/string to be used as separator between year and month fields, ii) lastDay to define if the date to be used with a particular year value (e.g., 1998) should be the first or the last day of the year: if the year is being interpreted as a beginDate, then the value is set to yyyy-01-01 and lastDay is set to false; if the year is interpreted as an endDate then the value is set to yyyy-12-31 and the lastDay value is set to true, iii) selectedDatePosition to define which date is being processed when a range of dates is present in the input field; for example:

imports the “dc/date” into the begin date using “_” as separator ; selectedDatePosition=”1” states the date to be processed is the first one in the range of dates and lastDay is thus set to false. The implementation of this custom handler is available on google code site. The web interface can be visited in this url, in a next blog I'll exaplained how this user interface was implemented using some a simple ajax framework.

Software quality control at GBIF

2011-05-17T13:40:00.000+02:00

We've not only set up Hadoop here at GBIF but also introduced a few other new things. With the growing software development team we've felt the need to put some control measures in place to guarantee the quality of our software and to make the development process more transparent both for us at GBIF and hopefully for other interested parties as well.

GBIF projects have always been open source and hosted at their Google Code sites (e.g. GBIF Occurrencestore or the IPT). So in theory it was always possible for everyone to check every commit and review it. We've set up a Jenkins server however that does continuous integration for us which means that every time a change is made to one of our projects it is checked out and a full build is being run including all tests, code quality measurements (I'm going to get back to those later), web site creation (e.g. Javadocs) and publishing of the results to our Maven repository.

This is the first step in our new process. Every commit is checked in this way and we've had great success improving the stability of our builds in this way. Our Jenkins server is publicly visible at the URL http://hudson.gbif.org (background on the Hudson name in the URL can be found on Wikipedia).

As part of the process Jenkins also calls a code quality server called Sonar. Our Sonar instance is public as well. Take a look at the metrics for the IPT for example. You'll see a lot of information about our code, good and bad. We're not yet using this information extensively but are looking into useful metrics to incorporate them more closely into our development process. One example are some Coding Conventions to make the code consistent and easier to understand for everybody.

Once the build has finished the Sonar stage the results of the build are pushed to our Maven repository (which is running a Nexus server). That means we now have up to date SNAPSHOT builds of all our projects available (to use in our and your projects).

At the moment we don't have a lot of code contributions from outside of the GBIF to our projects but we hope that by making our development process more transparent we can encourage others to take a look as well.

We're always open for suggestions, questions and comments about our code base.

Here be dragons - mapping occurrence data

2011-05-16T08:51:00.000+02:00

One of the most compelling ways of viewing GBIF data is on a map. While name lists and detailed text are useful if you know what you're looking for, a map can give you the overview you need to start honing your search. I've always liked playing with maps in web applications and recently I had the chance to add the functionality to our new Hadoop/Hive processing that answers the question "what species occurrence records exist in country x?".

Approximately 82% of the GBIF occurrence records have latitude and longitude recorded, but these often contain errors - typically typos, and often one or both of lat and long reversed. Map 1, below, plots all of the verbatim (i.e. completely unprocessed) records that have a latitude and longitude and claim to be in the USA. Note the common mistakes, which result in glaring errors: reversed longitude produces the near-perfect mirror over China; reversed latitude produces a faint image over the Pacific off the coast of Chile; reversing both produces an even fainter image off Australia; setting 0 for lat or long produces tell tale straight lines over the Prime Meridian and the equator.


Map 1: Verbatim (unprocessed) occurrence data coordinates for the USA

One of goals of the GBIF Secretariat is to help publishers improve their data, and identifying and reporting back these types of problems is one way of doing that. Of course the current GBIF data portal attempts to filter these records before displaying them. The current system for verifying that given coordinates fall within the country they claim is by overlaying a 1 degree grid on the world map, and identifying each of those grid points as belonging to one or more countries. This overlay is curated by hand, and is therefore error prone, and its maintenance is time consuming.

The results of doing a lookup against the overlay are shown in Map 2, where a number of bugs in the processing are still visible: parts of the mirror over China are still visible; none of the coastal waters that are legally US territory (i.e. Exclusive Economic Zone of 200 nautical miles off shore) are shown; the Aleutian Islands off the coast of Alaska are not shown; and, some spots around the world are allowed through, including 0,0 and a few seemingly at random.

Map 2: Results of current data portal processing for occurrences in the USA

My work, then, was to build new processing into our Hive/Hadoop processing workflow that addresses these problems and produces a map that is as close to error free as possible. The starting point is a webservice that can answer the question "In what country (including coastal waters) does this lat/long pair fall?". This is clearly a GIS problem, and in GIS-speak this is a reverse geocode, and something that PostGIS is well equipped to provide. Because country definitions and borders change semi-regularly, it seemed wisest to use a trusted source of country boundaries (shapefiles) that we could replace whenever needed. Similarly we needed the boundaries of Exclusive Economic Zones to cover coastal waters. The political boundaries come from Natural Earth, and the EEZ boundaries shapefile come from the VLIZ Maritime Boundaries Geodatabase.

While not an especially difficult query to formulize, a word to the wise: if you're doing this kind of reverse geocode lookup, remember to build your query by scoping the distance query within its enclosing polygon, like so

where the_geom && ST_GeomFromText(#{point}, 4326) and distance(the_geom, geomfromtext(#{point}, 4326)) < 0.001

This buys an order of magnitude improvement in query response time!

With a thin webservice wrapper from Jersey, we have the GIS pieces built. We opted for a webservice approach to allow us to ultimately expose this quality control utility externally in the future. Since we process in Hadoop, we experienced huge stress on this web service - we were DDOS'ing ourselves. I mentioned a similar approach in my last entry, where we alleviated the problem with load balancing across multiple machines. And in case anyone is wondering why we didn't just use Google's reverse-geocoding webservice, the answer is twofold - first, it violates their terms of use, and second, even if we were allowed, they hold a rate limit on how many queries you can send over time, and that would have brought our workflow to its knees.

The last piece of the puzzle is adding the call to the webservice from a Hive UDF and adding it to our workflow, which is reasonably straight forward. The result of the new processing is shown in Map 3, where the problems of Map 2 are all addressed.

Map 3: Results of new processing workflow for occurrences in the USA

These maps and the mapping cleanup processing will replace the existing maps and processing in our data portal later this year, hopefully in as little as a few months.

You can find the source of the reverse-geocode webservice at the Google code site for the occurrence-spatial project. Similarly you can browse the source of the Hadoop/Hive workflow and the Hive UDFs.

The GBIF Spreadsheet Processor - an easy option to publish data

2011-05-11T23:37:00.002+02:00

Most of data publishers in the GBIF Network use software wrappers to make data available on the web. To set up those tools, usually an institution or an individual needs to have certain degrees of technical capacity, and this more or less raises the threshold for publishing biodiversity data.

Imaging an entomologist who deals with collections and monographs everyday, the only thing s/he does on a PC is Word or Excel. S/he's got no student to help with, but keen to share the data before s/he retires. What is s/he going to do?

One of our tools is built to support this kind of scenario - the GBIF Darwin Core Archive Spreadsheet Processor, usually we just call it "the Spreadsheet Processor."

The Spreadsheet Processor is a web application that one can:

Use templates provided on the web site;
Fill and upload(or email) the xls file;
Get a Darwin Core Archive file as the result.

This is a pretty straight-forward approach to prepare data for publishing, because the learning curve is flat if users already know how to use Excel, how to upload a file on a web site.

When the spreadsheet template is uploaded to the page, the web app first parses the values in the metadata sheet to generate an eml.xml, and then the occurrence or checklist sheet to generate an meta.xml and csv file. These files are then collected and zipped according to Darwin Core Archive standard - ready to download.

With a DwC-A file, the data is in a standardized format and ready to be published. In the example scenario above, this entomologist can either only share them among colleagues, or, send them to the nearest GBIF node which hosts IPT. Since IPT can digest a DwC-A file and publish it, the entomologist doesn't need to know the usage of IPT. To update it, s/he can revise the spreadsheet, create and send DwC-A to the node again.

P.S. This manual explains how to publish and register data in DwC-A format.

Reworking the HIT, after reworking the Portal processing

2011-05-10T09:39:00.018+02:00

If GBIF reworks the Portal processing, then what would be the knock-on effect on the Harvesting and Indexing Toolkit (HIT)? This blog serves to talk a little about the future of the HIT, and very little about the new Portal processing (saved for later blogs).

To provide some background, the HIT has three major responsibilities:

harvesting specimen and occurrence data from data publishers,
writing that data in its raw form to the database, and
transforming raw data into its processed form running quality assurance routines (such as date and terrestrial point validation) and tying it to the backbone "nub" taxonomy.

When it is complete, the new Portal processing is actually going to do step 3. In the new processing, data will be extracted from the MySQL database into HBase (using sqoop) where quality assurance routines can be run much more quickly. Running outside of the MySQL database means that there won't be any more competition between steps 2 and 3 - step 3 constantly locking the raw data table in order to run its routines. That will mean the HIT will be able to write raw data uninterrupted to the database.

Lately I can tell you that the HIT has been having some frustrations trying to process large datasts. For example, a dataset with 12 million records, processing 10,000 records at a time, would lock the raw table for 10 minutes while scanning through the more than 280 million raw records in order to generate its record set. No raw data can be written at that time, thereby bringing the massively parallel application to its knees. Perhaps now you can understand why the rework of the Portal processing is so urgently needed.

For the few adopters of the HIT that will still require the application with its current functionality please rest assured that the project will just maintain a separate trimmed-down version when the time comes to adapt it. It will always remain an open-source application that anyone in the community can customize for their own needs.

Improving Hive join performance using Oozie

2011-05-06T16:02:00.001+02:00

In the portal processing we are making use of Apache Hive to provide SQL capabilities and Yahoo!'s Oozie to provide a workflow engine. In this blog I explain how we are making use of forks to improve the join performance of Hive, by further parallelizing the join beyond what Hive provides natively.

Please note that this was adopted using Hive version 0.5 but in Hive 0.7 there are significant improvements to joins

For the purposes of this explanation, let's consider the following simple example, where a table of verbatim values is being processed into four tables in a star schema:

To generate the leaves of the star, we have three simple queries (making use of a simple UDF to produce the increment IDs):

CREATE TABLE institution_code AS

SELECT rowSequence(), institution_code

FROM verbatim_record

GROUP BY institution_code;

CREATE TABLE collection_code AS

SELECT rowSequence(), collection_code

FROM verbatim_record

GROUP BY collection_code;

CREATE TABLE catalogue_number AS

SELECT rowSequence(), catalogue_number

FROM verbatim_record

GROUP BY catalogue_number;

To build the core of the star the simple approach is to issue the following SQL:

CREATE TABLE parsed_content AS

SELECT v.id AS id, ic.id AS institution_code_id,

cc.id AS collection_code_id, cn.id AS catalogue_number_id

FROM verbatim_record v

JOIN institution_code ic ON v.institution_code=ic.institution_code

JOIN collection_code cc ON v.collection_code=cc.collection_code

JOIN catalogue_number cn ON v.catalogue_number=cn.catalogue_number;

What is important to note is that the JOIN is across 3 different values, and this results in a query plan with three sequential MR jobs, a very large intermediate result set, which is ultimately passed through the final Reduce in the Hive planning.

By using Oozie (see the bottom of this post for pseudo workflow config), we are able to produce three temporary join tables, in a parallel fork, and then do a single join to bring it all back together.

# parallel join 1

CREATE TABLE t1 AS

SELECT v.id AS id, ic.id AS institution_code_id

FROM verbatim_record v JOIN institution_code ic ON v.institution_code=ic.institution_code;

# parallel join 2

CREATE TABLE t2 AS

SELECT v.id AS id, cc.id AS collection_code_id

FROM verbatim_record v JOIN collection_code cc ON v.collection_code=cc.collection_code

# parallel join 3

CREATE TABLE t3 AS

SELECT v.id AS id, ic.id AS institution_code_id

FROM verbatim_record v JOIN catalogue_number cn ON v.catalogue_number=cn.catalogue_number;

CREATE TABLE parsed_content AS

SELECT v.id AS id, t1.institution_code_id

t2.collection_code_id, t3.catalogue_number_id

FROM verbatim_record v

JOIN t1 ic ON v.id=t1.id

JOIN t2 cc ON v.id=t2.id

JOIN t3 cn ON v.id=t3.id;

Because we have built the join tables in parallel, and join on the foreign key only, Hive compiles to a single MR job, and runs much quicker.

In reality our tables are far more complex, and we use a Map side JOIN for the institution_code since it is small, but for our small cluster and the following table sizes we saw a reduction from several hours to 40 minutes to compute these tables.

verbatim_record: 284 million
collection_code: 1.5 million
catalogue_number: 199 million
institution_code: 8 thousand

All of this work can be found here.

Pseudo workflow config for this:

Line terminating characters breaking Darwin Core Archive

2011-05-04T14:48:00.005+02:00

Hi, I am Jan K. Legind the new data administrator at the GBIF Secretariat and one of my responsibilities is to ensure that datasets from publishers get indexed so that the data can be made available through the GBIF Portal. I am a historian by training and I have worked with archival data collection and testing prior to joining GBIF.

Recently I have been bug hunting a large dataset (DwC - Archive) that from a casual glance would look OK at the publisher side, but upon hitting the parser several records would be rejected because of the occurrence of line terminating characters in the records themselves (Hex value 0A). On top of that the individual record would be replaced by one empty line due to the illegal line termination AND another empty line would be added to that due to the tail end of the record appearing to the parser as the start of a new record, which of course would not be well-formed (thus being replaced with blank line number two). The parser will see a line that has too few columns and drop it. Since the line was bisected the tail end will also be conceived of as an individual line with an insufficient number of columns.

Here is an example of a record that would be replaced by an empty line:

The line terminating characters seems to have been escaped but without achieving the desired result. The secondary effect of this error is that the record count is miscalculated since the parser merely counts the lines and therefore ends up with a larger number than the publisher expected (remember that the line terminating character breaks the data file by producing two lines with an incorrect number of columns). Incidentally this example can sometimes explain why we harvest MORE than 100% of the target records.

By using the Integrated Publishing Toolkit (IPT) illegal characters can be avoided and the publishers will benefit from a faster transition into data appearing live in the GBIF portal. http://www.gbif.org/informatics/infrastructure/publishing/

Fortunately I am working in a joint effort with the publisher’s team on ironing out the bumps on this resource so we can get the data published fast and prevent future errors of this sort.

GBIF Data Portal

2011-05-02T11:16:00.007+02:00

The current GBIF Data Portal was designed and implemented in 2005/2006, around the time I first joined the GBIF Secretariat in Copenhagen. As I am not a developer myself, but have been involved with the Data Portal for a long time, I thought I would take the opportunity to give a bit of a summary view of some of the components discussed in other posts here, looking at them more from the perspective of the Data Portal.

The GBIF Data Portal has been in operation more or less in its current form since mid 2007. From the time it was designed, the Portal's focus is on providing discovery of and access to primary species occurrence data (specimens in museums, observations in the field, culture strains and others). Since the launch, bug fixes and some minor changes were made, but development stopped due to new priorities. We did receive a lot of input on data content and functionality, though, both from data publishers and data users, and also through a number of reports and analyses.

Towards the end of 2010, a new development phase started, initiating version 2 of the GBIF Data Portal. This was the time to start taking care of all the known shortcomings and improvement requests, e.g. a more robust and reliable backbone taxonomy, improvement of data quality, better attribution of contributors, and others. However, this is not just a matter of adding some data or changing the user interface: a lot of those points first require considerable reworking of internal processing and workflows between the Data Portal and related components, blogged about in other contributions here:

quicker indexing and more frequent rollovers (publication cycles) from the non-public indexing database to the public web portal can only be achieved through a complete re-working of the rollover processing workflow.
a reliable taxonomic backbone required a review and re-implementation of name parsing routines, integrating lookup services, and following that, a complete regeneration of the taxonomic backbone
the demand for better attribution of data owners and service providers can only be met after having moved on to a new registry, better modelling the GBIF network structure, players and interactions. This is especially the case where datasets are aggregated or hosted, and both the owning and the service providing institution need to receive proper credit for their contributions
extended and improved metadata are needed to assess suitability of a dataset for specific applications (e.g. modelling), and to allow discovery of collections that are not digitised or not published

In 2011, GBIF Data Portal development focuses on consolidating and integrating these re-worked components, and on including both names (checklist) and metadata sources into the search functionality. The implied changes on the Portal user interface side are quite fundamental. With other known and future requirements on user interface functionality, the time has now come to replace the old Portal code base. At present, we are working with an external team to develop wireframes for key Portal pages, based on functionality requests from GBIFS regarding the integration of the new data areas and following evaluation of a number of sources (task group reports, reviews, participant reports etc). Those wireframes will aid further discussions on functionality starting from July, and also build the basis for implementation in 2011 and after. Once there is a public version available to look at, we will give an update.

The evolution of the GBIF Registry

2011-04-29T14:20:00.003+02:00

The GBIF Registry has evolved through time to become an important tool in GBIF's day to day work. But before going into this post, a basic understanding of the GBIF Network model should be provided. GBIF is a decentralised network that has several network entities that are related in some way between each other. At the top level, there are GBIF Participant Nodes, which typically are countries or thematic networks that coordinate their domain. These Nodes endorse one or more Organisations or Institutions inside their domain, and each Organisation possesses one or more Resources exposed through the GBIF Network. Also, each Resource typically comes associated to a Technical Access Point which is the url to access its data. There are also other entities such as IPT Installations which are deployed inside specific organisations, but are not resources by themselves. They publish Resources that might be owned by other organisations. A quick view on the GBIF's network model can be seen:

Not long ago, this complexity was modeled using an Universal Description, Discovery and Integration (UDDI) system. This system served a purpose at the time, despite its limited data structures types (e.g. businessEntity, businessService, bindingTemplate, tModel). A BusinessEntity was associated with an Organisation/Institution, a BusinessService was associated to a Resource and a BindingTemplate was associated with the technical access point to access the data from that specific resource. A tModel was used to associate the BusinessEntity(Organisation) with a specific Node inside the GBIF Network. A quick view on how the network information was kept on this Registry :

The main disadvantages (for our concerns) of the UDDI Specification:

Its lack of contact information at the BusinessService(Resource) level (contacts can only be added at the BusinessEntity(Organisation) level)
Lack of more descriptive metadata on Organisation and Resources (lacking fields such as the address, homepage, phone of the organisation - sure you could provide all of this information through a complex use of UDDI's capabilities, but will result in unnecessary complexity to extract this information for third-party tools.
Limited to a fixed specification and to a fixed API (although, the UDDI client libraries available are quite straightforward to use)
General purpose specification, not easily adaptable for modeling the complexity of GBIF's network.
Our software dated back to the beginning of the past decade (Systinet WASP UDDI).
Third party consumers will need to know how to talk UDDI

In 2009, we tried overcoming some of our Registry limitations by trying an "UDDI on steroids" approach, which consisted still of an UDDI system (jUDDI in our case) and an external database which would hold some extra data (e.g. Resource contact information, organisation's address, homepage or phone, etc.). The main advantage was the creation our own APIs so that third-party tool developers, who wanted to consume the GBIF's network information, didn't need to know the nuts and bolts of UDDI specs anymore. We offered the community a simple API and its proper documentation, and we dealt with the inner workings of it all.

Further in this evolution, our Registry took the next step and we removed the UDDI component and were left only with a DB which gave us complete freedom to model the network. We now had a system on hand which offered the possibility to create any kind of entities on the Network (Nodes, Organisations, Resources, Technical Installations) and any relation among them. Along with this new approach, came the web application (http://gbrds.gbif.org) and a far better API which offered the possibility to consume the data in XML or JSON format. These APIs are easy to follow and are well documented (http://code.google.com/p/gbif-registry/wiki/TableOfContents). Among the new features:

Create any kind of entities
Create any kind of relation among them
More detailed metadata (for entities and contacts)
Ability to tag entities
Individual credentials for each Institution/Organisation to provide the ability to add new or delete existing resources under their own Organisations (this is currently only available through the APIs or via admin management)
Enhanced maintenance features (for admins)

[Evolution of GBIF's Registry]

Development is still ongoing and many exciting features are expected in the future. The status of development can be checked out here.

OAI-PMH Harvesting at GBIF

2011-04-27T07:00:00.014+02:00

GBIF has been my first experience in the bio-informatics world; my first assignment was developing an OAI-PMH harvester. This post will introduce OAI-PMH protocol and how we are gathering XML documents from different sources, in a next post I'll give a introduction to the Index that we have built using those documents.  
The main goal for this project was develop the infrastructure needed across the GBIF network to support the management and delivery of metadata that will enable potential end users to discover which datasets are available, and, to evaluate the appropriateness of such datasets for particular purposes. In the GBIF context, resources are datasets, loosely defined as collections of related data, the granularity of which is determined by the data custodian/provider.  

OAI-PMH (Open Archives Initiative Protocol for Metadata Harvesting) is a platform independent framework for metadata publishers and metadata consumers as well. The most important concepts of this protocol are:
• Metadata: provides information on such aspects as the ‘who, what, where, when and how’ pertaining to a resource. For the producer, metadata are used to document data in order to inform users of their characteristics, while for the consumer, metadata are used to both discover data and assess their appropriateness for particular needs ('fitness for purpose’).
• Repository: an accessible server that is able to process the protocol verbs.
• Unique identifier: is an unambiguous identifier of an item (document/record) inside the repository.
• Record: is metadata expressed in a specific format.
• Metadata-prefix: specifies the metadata format in OAI-PMH requests issued to the repository (EML 2.1.0, Dublin Core, etc.)

GBIF-Metadata Network Topology 
The metadata catalogue will primarily be used as the central catalogue in the GBIF Data Portal for the global GBIF network, which, in turn, will broker information to wider initiatives such as EuroGEOSS, OBIS, etc. Such initiatives are basically OAI-PMH service providers that will be contacted by GBIF metadata harvester.

The GBIF metadata catalogue service undertakes both harvesting and serving roles; aggregating metadata from other OAI-PMH repositories and serving metadata via OAI-PMH to other harvesting services. The harvested metadata are stored in a local file system. The system can apply XSLT transformation to create a new document based of the content of the existing one (e.g., transforming an EML document to an ISO19139 one).

OAI-PMH Harvester
The harvester is a standalone Java application, it makes extensive use of the open source project “OAIHarvester2” which supports OAI-PMH v1.1 and v2.0. The source code of this project was not modified but extended to handle the harvested XML payload. The payload is delivered as a single file of aggregated xml documents (one per metadata resource).
This component was implemented by modifying the OAICat (http://www.oclc.org/research/activities/oaicat/default.htm) web application. The main changes, made to achieve specific objectives, are:
• Dynamic load of file store. The default behaviour of the server is to load the file list at the server start-up. Since the harvester can modify the file store, the server loads the file list every time a ListIdentifiers or ListRecords verb is requested.
• Support multiple XSL transformations for an input format. The reference implementation only supports one transformation, in our implementation an input document can be published using multiple formats; for example: an EML document can be published using Dublin Core and DIF, if a XSL transformation is configured for each output format.

More detail about this project is available at the google-code project site: http://code.google.com/p/gbif-metadata/. In a future post I’ll explain how the information gathered by the harvester was used to build a search index using Solr and how a Web application uses the Index to enable end-users the search of metadata.

Cleanup of occurrence records

2011-04-20T13:19:00.001+02:00

Lars here, like Oliver I've started here in October 2010 and have no biology background either so my first step here at GBIF was to set up the infrastructure Tim was mentioning before, but I've written about that already (at length).

To continue the series of blog posts that was started by Oliver, and in no particular order, I'll talk about what we are doing to process the incoming data - which is the task I was given after the Hadoop setup was done.

During our rollover we're processing Occurrence records. Millions of them, about 270 millions at the moment and we expect this to grow significantly over the next few months and years. It is only natural that there is bound to be bad data in there for various reasons. These might be everything from simple typos to misconfigured publishing tools and transfer errors.

The more we know about the domain and the data the more we are obviously able to fix. Any input is appreciated on how we could do better on this part of our processing.

For fields like kingdom, phylum, country name or basis of record we do a simple lookup in a dictionary to look for common mistakes and replace those with the proper versions. Other fields like class, order, family, genus and author have way too many distinct values for us to prepare a dictionary with all the possible errors and their correct forms. That is why we only apply a few safe clean up procedures here (e.g. remove blacklisted names or invalid characters).

Scientific names are additionally parsed by the NameParser in the ECAT project which does all kinds of fancy magic to try to infer a correct name. Altitudes, depths and coordinates get treatment as well by looking at common unit markers and errors we've seen in the past.

And last but not least we also try to make most out of the dates we get. As everyone who ever dealt with date strings knows this can be one of the hardest topics in an internationalized environment. In theory our input data consists of three nicely formatted fields: year, month and day. In reality though a lot of dates are just in the year field. We've got all kinds of delimiters (with "/" and "-" being among the most common ones), abbreviations ("Mar") and database export fragments ("1978.0" because it was a floating point variable in the database), missing data and more.

Additionally we obviously have to deal with different time formats. Is "01/02/02" the first of February or the second of January? In most cases we can only guess.

Having said that: We've rewritten large parts of the date handling routines and are continuing to improve them as we know that this is an important part of our data. Feedback on how we're doing here is greatly appreciated!

I'm really hoping to have a chance to compile a few statistics about our incoming data quality once we've tested all of this in production.

Reworking the Portal processing

2011-04-18T15:49:00.002+02:00

The GBIF Data Portal has provided a gateway to discover and access the content shared through the GBIF network for some years, without major change. As the amount of data has grown, GBIF have scaled vertically (e.g. scaling up) to maintain performance levels; this is becoming unmanageable with the current processing routines due to the amount of SQL statements issued against the database. As GBIF content grows, the indexing infrastructure must change to scale out accordingly.

I have been monitoring and evaluating alternative technologies for some time and a few months ago GBIF initiated the redevelopment of the processing routines. This current area of work does not increase functionality offered through the portal (that will be addressed following this infrastructural work) but rather aims to:

Reduce the latency between a record changing on the publisher side, and being reflected in the index
Reduce the amount of (wo)man-hours needed to coax through a successful processing run
Improve the quality assurance by inclusion of

Checking that terrestrial point locations fall within the stated country using shapefiles
Checking coastal waters using Exclusive Economic Zones

Rework all the date and time handling
Use dictionaries (vocabularies) for interpretation of fields such as Basis of Record
Integrate checklists (taxonomic, nomenclatural and thematic) shared through the GBIF ECAT Programme to improve the taxonomic services, and the backbone ("nub") taxonomy.
Provide a robust framework for future development
Allow the infrastructure to grow predictably with content and demand growth

Things have progressed significantly since my early investigations, and GBIF are developing using the following technologies:

Apache Hadoop: A distributed file system, and cluster processing using the Map Reduce framework

GBIF are using the Cloudera distribution of Hadoop

Sqoop: A utility to synchronize between relational databases and Hadoop
Hive: A data warehouse infrastructure built on top of Hadoop, and developed and open-sourced by Facebook. Hive gives SQL capabilities on Hadoop. [Full table scans on GBIF occurrence records reduce from hours to minutes]
Oozie: An open-source workflow/coordination service to manage data processing jobs for Hadoop, developed then open-sourced by Yahoo!

[GBIF are researching using HBase, the Hadoop database to allow an increase in the richness in the indexed content, and will be the subject of future blogs. See the project site]

The processing workflow looks like the following (click for full size):

The Oozie workflow is still being developed, but the workflow definition can be found here.

Lucene for searching names in our new common taxonomy

2011-04-18T02:00:00.000+02:00

Oliver here - I'm one of the new developers at GBIF, having started in October, 2010. With no previous experience in biology or biological classification you can bet it's been a steep learning curve in my time here, but at the same time it's very nice to be learning about a domain that's real, valuable and permanent, rather than yet another fleeting e-commerce, money-trading or "social media" application!

One of the features of GBIF's Data Portal is allowing searching of primary occurrence data via a backbone taxonomy. For example let's say you're interested in snow leopards and would like to plot all current and historical occurrences of this elusive cat on a world map. Let's further say that Richard Attenborough suggested to you that the snow leopard's scientific name is "Panthera uncia". You would ask the data portal for all records about Panthera uncia and expect to see all occurrences of snow leopards. Unfortunately biologists aren't agreed on how to classify the snow leopard - some argue that it belongs in the genus Panthera, while others argue that it should belong to its own genus, Uncia, and naturally the GBIF network has records under both names. You would just like to see all of those records and never mind the details - and that's just the tip of the iceberg when it comes to building a backbone taxonomy to match the 260 million+ occurrence records in the GBIF network.

Indeed, the backbone taxonomy (we call it our "Nub Taxonomy") in use by the current data portal has been one of the biggest sources of criticism of the GBIF data portal - it doesn't cover enough of the names in our occurrence records, and it doesn't handle the tricky stuff (as above) as well as it should. One of the reasons for that is the current backbone taxonomy was built based on the Catalogue of Life 2007, a similar vintage International Plant Names Index (IPNI), and then augmented with the classifications from any unmatched occurrence records. This has led to a classification hierarchy which is less reliable than we (and the GBIF network) would like.

Markus Döring is the GBIF software team's taxonomy expert and he has employed a new strategy for building an improved Nub Taxonomy by building it exclusively on well-known and respected taxonomies already out there - things like the most recent Catalogue of Life, IPNI, and more, but without using the classifications as given in the occurrence data. After the Nub Taxonomy is built, the occurrence records then need to be matched to it. As the first step to integrating the new Nub Taxonomy into the data portal, my job in the last little while has been to build a searchable index of all the names in our Nub Taxonomy and a web service that can accept a scientific name (from an occurrence record) and match it to the index, while understanding the implications of homonyms and synonyms, as well as tolerating misspellings. And of course, make it fast :)

Since what we're talking about here is string matching with a tolerance for messy input (e.g. spelling mistakes, different violations of nomenclatural rules) the place to start is Lucene. Our Nub Taxonomy has about 8 million unique names, and our 260 million occurrence records are also comprised of roughly 8 million unique names. Our use case is somewhat out of the ordinary for Lucene in that we can build the index once and after that it becomes read-only until the next update of our Nub Taxonomy (e.g. to reflect an update in the Catalog of Life), and it only takes a few minutes to build the index, so it's not all that important for it to be persistent. That means we can optimize for search speed and not worry so much about indexing performance. Lucene has just the index storage implementation for this need - RAMDirectory. For the most part this worked just fine, but no matter how hard I hit the index, I couldn't get cpu usage to 100% - the best I could do was about 80%. I found that very irksome and spent some time testing different Directory implementations, web service stacks, and everything in between. None of the other Directory implementations (all file based in some way) showed any improvements, nor did eliminating the web stack. Finally by attaching a profiler to the Tomcat instance running the webservice while running with RAMDirectory we were able to see thread blocking increasing proportional to the number of requesting threads. That led us to the Lucene source code where we found a synchronized() block that we deemed the culprit. With the cause at least found I decided not to waste time trying to fix the problem for what would be nominal gain, but instead decided to use two Tomcat installations and load balance between them with Apache. With the Tomcats running on quite powerful machines we are now seeing approximately 1000 lookups/sec (including a bunch of business logic beyond the Lucene lookup), which we think is pretty good, and sufficient for our purposes.

This is all being used from within our Oozie orchestrated Hive/Hadoop workflow (which Lars will talk more about soon) but once we're confident that it's behaving properly and stably we will also offer this web service (or something similar) for public consumption. More importantly the new Nub Taxonomy will be available in the GBIF data portal very soon and with it we hope to have eliminated most of the problems people have found with our current backbone taxonomy.

The first drafts of the Data Publishing Manuals are available for feedbacks

2011-04-15T17:26:00.014+02:00

Since Darwin Core had been officially ratified by Biodiversity Information Standards (TDWG) in November 2009, a few tools were developed by GBIFS to leverage the standard data format, a.k.a the Darwin Core Archive, to facilitate data mobilisation. These tools include Darwin Core Archive Assistant, GBIF Spreadsheet Processor and some validators that users can use to produce standard-compliant files for data exchanging or publishing purposes. Also, IPT has upgraded to version 2 recently to fully support data publishing in metadata, occurrence data and taxonomic data using Darwin Core Archive.

Accompanying these development efforts, a suite of document are also prepared to instruct users on, not only the usage of individual software tool, but how to make data available within the GBIF Network. For those tool options we have in the biodiversity information world, we organised these materials according to which kind of content that users want to publish, and present a document map for users to follow. So, if you go to the Informatics section of the GBIF web site, there are pages called "publishing" under "Discovery/Metadata," "Primary Data" and "Name Services." Maps are there ready to guide you through the way for publishing your data. Every node in the maps are clickable and will lead you to those individual manuals.

The intention of using a map as a guide is to suggest a route that user can have basic understanding about data publishing before they play with software tools, so readers are not given a bunch of documents and don't know where to start, or hesitate to finish reading all of these before starting. We also try to keep each manual as compact as possible, with emphasis on steps, rather than just theories.

In addition to users with biodiversity background, we'd like invite developers to evaluate these draft materials, too. Any comments are welcome, especially whether these manuals help in explaining the data publishing workflow to the users you serve.

Can IPT2 handle big datasets now?

2011-04-13T10:27:00.018+02:00

One of IPT1's most serious problems was its inability to handle large datasets. For example, a dataset with only half a million records (relatively small compared to some of the biggest in the GBIF network) caused the application to slow down to such a degree that even the most patient users were throwing their hands up in dismay.
Anyways, I wanted to see for myself whether the IPT’s problems with large datasets have been overcome or not in the newest version: IPT2.

Here’s what I did to run the test: First, I connected to a MySQL database and used a “select * from … limit …” query to define my source data totalling 24 million records (the same number of records as a large dataset coming from Sweden). Next, I mapped 17 columns to Darwin Core occurrence terms and once this was done I was able to start the publication of a Darwin Core Archive (DwC-A). The publication took just under 50 minutes to finish, processing approximately 500,000 records per minute. Take a look at the screenshot below that was taken after the successful publication. Important to note is that this test was run on a Tomcat server with only 256MB of memory. In fact, special care was taken during IPT2 design to ensure it could still run on older hardware that didn’t have a lot of memory. It’s worth noting that this is one of the reasons why IPT2 is not as feature rich as the IPT1 was.

So just how does the IPT2 handle 24 million records coming from a database while running on a system with so little memory? The answer is that instead of returning all records at once, they are retrieved in small result sets only having about 1000 records each. These results sets are then streamed to file and immediately written to disk. The final DwC-A generated was 3.61GB in size, so some disk space is obviously needed too.

Therefore in conclusion I feel that he IPT2 has successfully overcome its previous problems handling large datasets. I hope other adopters will now give it a shot themselves.

The GBIF Development Team

2011-04-11T11:30:00.001+02:00

Recently the GBIF development group have been asked to communicate more on the work being carried out in the secretariat. To quote one message:

"IMHO, simply making all these discussions public via a basic mailing list could help people like me ... have a better awareness of what's going on... We could add our comments / identify possible drawbacks / make some "scalability tests"... In fact I'm really eager to participate to this process" (developer in Belgium)

To kick things off, we plan to make better use of this blog and have set a target of posting 2-3 times a week. This is a technical blog, so the anticipated audience include developers, database administrators and those interested in following details of the GBIF software development. We have always welcomed external contributers to this blog and invite any developers working on publishing content through the GBIF network, or developing tools that make use of content discoverable and accessible through GBIF to write posts.

Today we are pleased to welcome Jan Legind to the team who will be working as a data administrator to help improve the frequency of the network crawling (harvesting) and the indexing processes. Jan will be working closely with the data publishers to help improve the quality and quantity of content accessible through GBIF.

The GBIF development group has expanded in the past 6 months, so I'll introduce the whole team working in the secretariat and contracted to GBIF:

Developers (in order of appearance in the team): Kyle Braak, José Cuadra, Markus Döring (contracted in Germany), Daniel Amariles & Hectór Tobón (contracted at CIAT in Colombia), Federico Méndez, Lars Francke and Oliver Meyn.
Systems architect: Tim Robertson
Systems analyst: Andrea Hahn
Informatics liason: Burke (Chih-Jen) Ko
Systems admins: Ciprian Vizitiu & Andrei Cenja
Data administrator: Jan Legind

The current focus of work at GBIF include the following major activities:

Developing and rolling out the Integrated Publishing Toolkit.
Integrating the checklist (taxonomic, nomenclatural and thematic) content into the current Data portal.
Developing a processing framework to automate the steps needed to apply quality control and index content for discovery through the Data portal.

Specifically to reducing the time taken and complexity in initiating a rollover of the content behind the index
Reworking all quality control (geographic, taxonomic and temporal)
Automating the process

Initiating a redesign of the data portal user interface to provide richer discovery and integration across dataset metadata, checklists and primary biodiversity data.
Reducing the time between publishing content onto the network and discovery through the Data portal. This includes providing specific support to those who are experiencing problems with large datasets in particular, and assisting in migration to the DarwinCore-Archive format.
Technical and user documentation of the publishing options available

Let the blogging begin.

[Please use #gbif in twitter hashtags]

Setting up a Hadoop cluster - Part 1: Manual Installation

2011-01-26T16:20:00.000+01:00

Introduction

In the last few months I was tasked several times with setting up Hadoop clusters. Those weren't huge - two to thirteen machines - but from what I read and hear this is a common use case especially for companies just starting with Hadoop or setting up a first small test cluster. While there is a huge amount of documentation in form of official documentation, blog posts, articles and books most of it stops just where it gets interesting: Dealing with all the stuff you really have to do to set up a cluster, cleaning logs, maintaining the system, knowing what and how to tune etc.

I'll try to describe all the hoops we had to jump through and all the steps involved to get our Hadoop cluster up and running. Probably trivial stuff for experienced Sysadmins but if you're a Developer and finding yourself in the "Devops" role all of a sudden I hope it is useful to you.

While working at GBIF I was asked to set up a Hadoop cluster on 15 existing and 3 new machines. So the first interesting thing about this setup is that it is a heterogeneous environment: Three different configurations at the moment. This is where our first goal came from: We wanted some kind of automated configuration management. We needed to try different cluster configurations and we need to be able to shift roles around the cluster without having to do a lot of manual work on each machine. We decided to use a tool called Puppet for this task.

While Hadoop is not currently in production at GBIF there are mid- to long-term plans to switch parts of our infrastructure to various components of the HStack. Namely MapReduce jobs with Hive and perhaps Pig (there is already strong knowledge of SQL here) and also storing of large amounts of raw data in HBase to be processed asynchronously (~500 million records until next year) and indexed in a Lucene/Solr solution possibly using something like Katta to distribute indexes. For good measure we also have fairly complex geographic calculations and map-tile rendering that could be done on Hadoop. So we have those 18 machines and no real clue how they'll be used and which services we'd need in the end.

Environment

As mentioned before we have three different server configurations. We've put those machines in three logical clusters c1, c2 and c3 and just counting up in those (our master for example is currently running on c1n1):

c1 10: Intel(R) Xeon(R) CPU X3363 @ 2.83GHz, 2x6MB (quad), 8 GB RAM, 2 x 500GB SATA 7.2K
c2 3: 2 x Intel(R) Xeon(R) CPU E5630 @ 2.53GHz (quad), 24 GB RAM, 6 x 250 GB SATA 5.4K
c3 5: Intel(R) Xeon(R) CPU X3220 @ 2.40GHz (quad), 4 GB RAM, 2 x 160 GB SATA 7.2K
CentOS 5.5
The machines are in different racks but connected to only one switch

We realize that this is a very heterogeneous cluster configuration. We also realize that some people highly discourage use of old machines or machines with little RAM but the c1 and c3 clusters were old unused machines and this way they still serve a purpose and we've had no problems so far using this setup.

Goal

These were the goals we set out to achieve on our cluster and these are also all the things I'll try to describe in this or a following post:

Puppet for setting up the services and configuring machine state
CDH3 (Beta 3)
- Hadoop HDFS + MapReduce incl. Hadoop LZO
- Hue
- Zookeeper
- HBase
Easily distributable packages for Hadoop, Hive and Pig to be used by the employees to access the cluster from their own workstations
Benchmarks & Optimizations

Be warned: This is going to be a very long post and unfortunately it is the nature of these things that some of the information is bound to be outdated pretty quickly so let me know if something has changed and I'll alter the post.

Manual Installation

Before we use Puppet to do everything automatically I will show how it can be done manually. I think it is important to know all the steps in case something goes wrong or you decide not to use Puppet at all. When I talk about "the server" I always mean "all servers in your cluster" except when noted otherwise. I highly recommend not skipping this part even if you want to use Puppet.

Operating System

For now I'll just assume a vanilla CentOS 5.5 installation. There's nothing special you need. I recommend just the bare minimum, everything else needed can be installed at a later time. A few words though about things you might want to do: Your servers probably have multiple disks. You shouldn't use any RAID or LVM on any of your slaves (i.e. DataNodes/TaskTracker). Just use a JBOD configuration. In our cluster all disks are in a simple structure:

/mnt/disk1
/mnt/disk2
...

There are also two tweaks for your slaves you can do:

Mount your data disks with noatime (e.g. /dev/sdc1 /mnt/disk3 ext3 defaults,noatime 1 2 which btw. implies nodiratime)
By default there are a certain number of blocks reserved on ext (not familiar with others) file systems (check by running tune2fs -l /dev/sdc1 and look at the Reserved block count). While this is useful on system disks so that critical processes can still write some data when the disk is full otherwise this is wasted space on our data disks. By default 5% of a HDD are reserved for this. I recommend setting this down to 1% by running: tune2fs -m 1 <device> on all your data disks (i.e. tune2fs -m 1 /dev/sdc1) which frees up quite a bit of disk space. You can also set it to 0% if you want though I went with 1% for our cluster. Keep the default setting for your system disks though!

On your NameNode however use any means you feel necessary to secure your data. You know your requirements better than I do. Use RAID and/or LVM however you like. We don't have any special resources so our NameNode is running on one of our regular servers at the moment. We might change that in the future.

A note on Cloudera's Package system & naming

Cloudera provides the various components of Hadoop in different Packages but they follow a simple structure: There is one hadoop-0.20 package which contains all the jars, config files, directories, etc. needed for all the roles. And then there are packages like hadoop-0.20-namenode which are only a few kilobytes and they only contain the appropriate start- and stopscripts for the role in question.

1. Common Requirements

Most of the commands in this guide need to be executed as root. I've chosen the easy route here and just logged in as root. If you're operating as a non-privileged user remember to use su, sudo or any other means to ensure you have the proper rights.

Repository

Cloudera documentation

As all the packages we're going to install are provided by Cloudera we need to add their repository to our cluster:

curl http://archive.cloudera.com/redhat/cdh/cloudera-cdh3.repo > /etc/yum.repos.d/cloudera-cdh3.repo

Java installation

Cloudera documentation
Java downloads
We're using JDK 6 Update 23

You have to download the JDK from Oracle's website yourself as license issues prevent it from being added to the repositories. Chose the correct system (probably Linux x64) and make sure to download the file ending in -rpm.bin (i.e. jdk-6u23-linux-x64-rpm.bin). You might have to do this from a client machine because you need a browser that works with the Oracle site. So on any one machine execute the following:

unzip jdk-6u23-linux-x64-rpm.bin

You should now have a bunch of .rpm files but you only need one of them: jdk-6u23-linux-amd64.rpm. Copy this file to your servers and install it as root using rpm:

rpm -Uvh ./jdk-6u23-linux-amd64.rpm

Time
While not a hard requirement it makes a lot of things easier if the clocks on your servers are synchronized. I added this part at the last minute because we just realized that ntpd was disabled on three of our machines (c2) by accident and had some problems with it. It is worth taking a look at the clocks now and set up ntp properly before you start.

DNS
It doesn't matter if you use a DNS server or hosts files or any other means for the servers to find each other. But make sure this works! Do it now! Even if you think everything's set up correctly. Another thing that you should check is if the local hostname resolves to the public IP address. If you're using a DNS server you can use dig to test this but that doesn't take into account the /etc/hosts file so here is a simple test to see if it is correct:

ping -c 1 `hostname`

This should resolve to the public IP and not to 127.0.0.1.

Firewall
Hadoop uses a lot of ports for its internal and external communications. We've just allowed all traffic between the servers in the cluster and clients. But if you don't want to do that you can also selectively open the required ports. I try to mention them but they can all be changed in the configuration files. I might also miss some due to our config so I'd be glad if someone could point those out to me.

Packages
We're going to use lzo compression, the Hadoop native libraries as well as hue so there are a few common dependencies on all machines in the cluster which can be easily installed:

rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
yum install -y lzo hue-plugins hadoop-0.20-native

Directories
We also need some directories later on so we can just create them now:

mkdir <data disk>/hadoop
chown root:hadoop <data disk>/hadoop

Cloudera uses the alternatives system to manage configuration. In /etc/hadoop/conf is the currently activated configuration. Look at the contents of /etc/hadoop and you'll find all the installed configurations. At the moment there is only a conf.empty directory which we'll use as our starting point:

cp -R /etc/hadoop/conf.empty /etc/hadoop/conf.cluster

Now feel free to edit the configuration files in /etc/hadoop/conf.cluster but we'll go through them as well later in this post. The last step is to activate this configuration:

/usr/sbin/alternatives --install /etc/hadoop-0.20/conf hadoop-0.20-conf /etc/hadoop-0.20/conf.cluster 50

LZO

hadoop-gpl-compression project
Todd Lipcon's hadoop-lzo & Kevin Weil's hadoop-lzo projects

Due to licensing issues the LZO bindings for Hadoop cannot be distributed the same way as the rest of the packages. So this - once again - involves a few manual steps. After these bindings were removed from Hadoop itself a few versions ago they moved tho the hadoop-gpl-compression project on Google Code which (as far as I know) still works but hasn't seen any development for over a year. Thankfully though Twitter's Kevin Weil and Cloudera's Todd Lipcon have picked up the project and maintained it. They regularly sync their github repositories so both should have almost the same code. I'm going to use Todd's version here as it should be better synced with CDH releases. You have to download the code from the repository, build the native libraries as well as the jar file and distribute those files on your cluster. You need to do this only on one machine which ideally should run the same OS version as the servers in your cluster. When you're finished you can just copy the result to all servers. We're using version 0.4.9 so we use this to download and build:

rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
yum install -y lzo-devel
wget --no-check-certificate https://github.com/toddlipcon/hadoop-lzo/tarball/0.4.9
tar xvfz toddlipcon-hadoop-lzo-0.4.9-0-g0e70051.tar.gz
wget http://www.apache.org/dist/ant/binaries/apache-ant-1.8.2-bin.tar.bz2
tar jxvf apache-ant-1.8.2-bin.tar.gz
cd toddlipcon-hadoop-lzo-0e70051
JAVA_HOME=/usr/java/latest/ BUILD_REVISION="0.4.9" ../apache-ant-1.8.2/bin/ant tar

The ant version that comes with CentOS 5.5 didn't work for me that's why I downloaded a new one. This should leave you with a hadoop-lzo-0.4.9.tar.gz file in the build directory which you can extract to get all the necessary files for your servers:

hadoop-lzo-0.4.9.jar needs to be copied into /usr/lib/hadoop/lib on each server
lib/native/Linux-amd64-64 needs to be copied into /usr/lib/hadoop/lib/native on each server

cron & log cleaning
We've had a problem with unintentional debug logs filling up our hard drives. The investigations that followed that incident resulted in a blog post by Lars George explaining all the log files Hadoop writes. It is a worthwhile read. Hadoop writes tons of logs in various processes and phases and you should make sure that these don't fill up your hard drives. There are two instances in the current CDH3b3 where you have to manually interfere:

Hadoop daemon logs
Job XML files on the JobTracker

Hadoop uses a DailyRollingFileAppender which unfortunately doesn't have a maxBackupIndex setting like the RollingFileAppender. So either change the appender or manually clean up logs after a few days. We chose the second path and added a very simple cron job to run daily:

find /var/log/hadoop/ -type f -mtime +14 -name "hadoop-hadoop-*" -delete

This jobs deletes old log files after 14 days. We'll take care of the Job XML files in a similar way at the JobTracker.

HDFS

One property needs to be set for both the NameNode and the DataNodes in the file /etc/hadoop/conf/core-site.xml: fs.default.name. So just add this and replace $namenode with the IP or name of your NameNode:

<property>
  <name>fs.default.name</name>
  <value>hdfs://$namenode:8020</value>
</property>

2.1. NameNode
Installing the NameNode is straightforward:

yum install -y hadoop-0.20-namenode

This installs the startup scripts for the NameNode. The core package was already installed in the previous step. Now we need to change the configuration, create some directories and format the NameNode.

In /etc/hadoop/conf/hdfs-site.xml add the dfs.name.dir property which "determines where on the local filesystem the DFS name node should store the name table(fsimage). If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy." We mentioned before that we're using a JBOD configuration. We do this even for our NameNode. So in our case the NameNode has two disks mounted at /mnt/disk1 and /mnt/disk2 but you might want to write to just one location if you use RAID. As it says in the documentation the NameNode will write to each of the locations. You can write to a third location: A NFS mount which serves as a backup. Our configuration looks like this:

<property>
  <name>dfs.name.dir</name>
  <value>/mnt/disk1/hadoop/dfs/name,/mnt/disk2/hadoop/dfs/name</value>
</property>

Make sure to create the dfs directories before starting the NameNode. They need to belong to hdfs:hadoop. Formatting the NameNode is all that's left:

su hdfs -c "/usr/bin/hadoop namenode -format"

Once you've done all that you can enable the service so it will be started upon system boot and start the NameNode:

chkconfig hadoop-0.20-namenode on
service hadoop-0.20-namenode start

You should be able to see the web interface on your namenode at port 50070 now. Ports that need to be opened to clients on the NameNode are 50070 (web interface, 50470 if you enabled SSL) and 8020 (for HDFS command line interaction). Only port 8020 needs to be enabled for all other servers in the cluster.

We also use a cron job to run the HDFS Balancer every evening:

/usr/lib/hadoop-0.20/bin/start-balancer.sh -threshold 5

2.2 DataNodes
The DataNodes handle all the data by storing it and serving it to clients. You can run a DataNode on your NameNode and especially for small- or test clusters this is often done but as soon as you have more than three to five machines or rely on your cluster for production use you should use a dedicated NameNode. Setting the DataNodes up is easy though after all our preparations. We need to set the property dfs.data.dir in the file /etc/hadoop/conf/hdfs-site.xml. It "determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. Directories that do not exist are ignored." These are the directories where the real data bytes of HDFS will be written to. If you specify multiple directories the DataNode will write to them in turn which gives good performance when reading the data.

This is an example of what we are using:

<property>
  <name>dfs.data.dir</name>
  <value>/mnt/disk1/hadoop/dfs/data,/mnt/disk2/hadoop/dfs/data</value>
</property>

Make sure to create the dfs directories before starting the DataNodes. They need to belong to hdfs:hadoop. When that's done you just need to install the DataNode, activate the startup scripts and start it:

yum install -y hadoop-0.20-datanode
chkconfig hadoop-0.20-datanode on
service hadoop-0.20-datanode start

Your DataNode should be up and running and if you have configured it correctly should also have connected to the NameNode and be visible in the web interface in the Live Nodes list and the configured capacity should go up. Ports that need to be opened to clients are 50075 (web interface, 50475 if you enabled SSL) and 50010 (for data transfer). For the cluster you need to open ports 50010 and 50020.

3. MapReduce

MapReduce is split in two parts as well: A JobTracker and multiple TaskTrackers. For small-ish clusters the NameNode and the JobTracker can run on the same server but depending on your usage and available memory you might need to run them on separate servers. We have 18 servers, 17 slaves and 1 master (with NameNode, JobTracker and other services) which isn't a problem so far. We need three properties set on all servers (in mapred-site.xml) to get started.

mapred.job.tracker: "The host and port that the MapReduce job tracker runs at. If 'local', then jobs are run in-process as a single map and reduce task."
- This just points to your JobTracker. There is no default port for this in Hadoop 0.20 but 8021 is often used.
- Our value (replace $jobtracker with the name or IP of your designated JobTracker): $jobtracker:8021
mapred.local.dir: "The local directory where MapReduce stores intermediate data files. May be a comma-separated list of directories on different devices in order to spread disk i/o. Directories that do not exist are ignored."
- As it says this is a local directory where MapReduce stores stuff an we spread it out over all our discs.
- Our value: /mnt/disk1/hadoop/mapreduce,/mnt/disk2/hadoop/mapreduce
- Create the directories on each server with the owner mapred:hadoop
mapred.system.dir: "The shared directory where MapReduce stores control files."
- This is a path in HDFS where MapReduce stores stuff
- Our value: /hadoop/mapreduce/system
- If dfs.permissions are on you need to create this directory in HDFS. Execute this command on any server in your cluster: su hdfs -c "/usr/bin/hadoop fs -mkdir /hadoop/mapreduce && /usr/bin/hadoop fs -chown mapred:hadoop /hadoop/mapreduce"

3.1 JobTracker
The JobTracker is very easy to setup and start:

yum install -y hadoop-0.20-jobtracker
chkconfig hadoop-0.20-jobtracker on
service hadoop-0.20-jobtracker start

The web interface should now be available at port 50030 on your JobTracker. Ports 50030 (web interface) and 8021 (not well defined in Hadoop 0.20 but if you followed my configuration this is correct) need to be opened to clients. Only 8021 is necessary for the TaskTrackers.

If the JobTracker is restarted some old files will not be cleaned up. That's why we added another small cronjob to run daily:

find /var/log/hadoop/ -type f -mtime +3 -name "job_*_conf.xml" -delete

3.2 TaskTracker
The TaskTracker are as easy to install as the JobTracker:

yum install -y hadoop-0.20-tasktracker
chkconfig hadoop-0.20-tasktracker on
service hadoop-0.20-tasktracker start

The TaskTracker should now be up and running and visible in the JobTracker's Nodes list. Only port 50060 needs to be opened to clients for a minimalistic web interface. Other than that no other ports are needed as TaskTrackers check in at the JobTracker regularly (heartbeat) and get assigned Tasks at the same time.

4. Configuration

I'll discuss a few configuration properties here that in a range of "necessary to change" to "nice to know about". I'll mention the following things for each property:

The default value,
the value we use for our cluster at GBIF if it differs from the default,
some of the defaults are quite old and have never been changed so I might mention a value I deem safe to use for everybody,
if we set the property to final so it can't be overridden by clients (we set a lot of the parameters to final for purely documentary reasons, even those that can't be overwritten in the first place),
if the property has been renamed or deprecated in Hadoop 0.21,
and if this property is required in a client configuration file or only on the cluster, if I don't mention it it's not needed.

Here are the default configuration files for Hadoop 0.20.2 and 0.21:

core-default.xml: 0.20.2, 0.21
hdfs-default.xml: 0.20.2, 0.21
mapred-default.xml: 0.20.2, 0.21

And I know that there are some duplications to the section above but I want to keep this Configuration section as a reference.

`core-site.xml`

fs.default.name

Default: file:///
We: hdfs://$namenode:8020
We set this to final
Renamed to fs.defaultFS in Hadoop 0.21
Needed on the clients

This is used to specify the default file system and defaults to your local file system that's why it needs be set to a HDFS address. This is important for client configuration as well so your local configuration file should include this element.

hadoop.tmp.dir

Default: /tmp/hadoop-${user.name}
CDH3 Default: /var/lib/hadoop-0.20/cache/${user.name}
We: Left it at the CDH3 default
We set this to final

As mentioned in the default file this is mainly a base for other temporary directories. If all other configuration options are set correctly there shouldn't be too much data in here.

fs.trash.interval

Default: 0
We: 10080
We set this to final

Hadoop has a Trash feature were removed files (using the command line tools) are moved to a .Trash folder in the users home folder. If set to 0 this feature is disabled but if set to a non-zero value this is the amount of minutes between Trash cleaner runs. As we have a lot of users in our system using Hadoop for the first time we chose a safe value here.

fs.checkpoint.dir

Default: ${hadoop.tmp.dir}/dfs/namesecondary
We: /mnt/disk1/hadoop/dfs/namesecondary,/mnt/disk2/hadoop/dfs/namesecondary
We set this to final

The secondary NameNode stores its images to merge here. If it is a comma separated list the data is replicated to all these locations on the local disks.

io.file.buffer.size

Default: 4096
Safe: 65536
We: 131072 (32 * 4096)
Can be overwritten by clients
We set this to final

This is used for buffers all over the place to copy, store and write data to. It should be a multiple of 4096 and it should be safe to use 65536 today but we use double that. The performance gain is not enormous but there have been blog posts in the past measuring the impact and it was positive. We've also done our own tests and saw a small performance gain. If you use HBase be careful not to set this too high.

io.compression.codecs

Default: org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec
We: org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec
We set this to final

This lists all installed compression codecs. If you followed my manual you've got to add two more to the default list of codecs: LzoCodec and LzopCodec.

io.compression.codec.lzo.class

We: com.hadoop.compression.lzo.LzopCodec

I have actually no idea why this setting is needed as I couldn't find any reference where it is actually used in the code but I didn't look very hard so I might be wrong. All I know is that the documentation mentions that this property needs to be set.

webinterface.private.actions

Default: false
We: true
We set this to final

By setting this to true the web interfaces for the JobTracker and NameNode gain some advanced options like killing a job. It makes life a lot easier while still in development or evaluation. But you probably should set this to false once you rely on your Hadoop cluster for production use.

`hdfs-site.xml`

dfs.name.dir

Default: ${hadoop.tmp.dir}/dfs/name
We: /mnt/disk1/hadoop/dfs/name,/mnt/disk2/hadoop/dfs/name
We set this to final

This is an important setting to set that's why I've already mentioned it above. The NameNode stores stuff in these directories by replicating all information to all these disks. One of them could be a mount on a remote disk (e.g. NFS) to have a backup.
dfs.data.dir

Default: ${hadoop.tmp.dir}/dfs/data
We: /mnt/disk1/hadoop/dfs/data,/mnt/disk2/hadoop/dfs/data
We set this to final

This is another important setting as explained above. Different to dfs.name.dir in that the data is not replicated to all disks but distributed among all those locations. The DataNodes save the actual data in these locations. So more space is better. The easiest thing is to use dedicated disks for this. If you save other stuff than Hadoop data on the disks make sure to set dfs.datanode.du.reserved (see below).

dfs.namenode.handler.count

Default: 10
We: 20
Safe: 10-20
We set this to final

The number of threads the NameNode uses to serve requests. This depends highly on your usage and size of your cluster. We've tried a bunch of different values and settled on 20 without seeing any notable differences. nnbench is probably a good tool to benchmark this. If you've got a large cluster or many file operations (create or delete) you can try upping this value.

dfs.datanode.handler.count

Default: 3
We: 5
Safe: 5-10
We set this to final

The number of threads DataNodes use. I can't tell what a good value is for large clusters but the TestDFSIO benchmark seems like a good test to run to find a good value here. Just play around. We've tried a bunch of different values up to 20 and didn't see a difference so we chose a value slightly larger than the default.

dfs.datanode.du.reserved

Default: 0
We: Left the default

This many bytes will be left free on the volumes used by the DataNodes (see dfs.data.dir). As our drives are dedicated to Hadoop we left this at 0 but if the drives host other stuff as well set this to an appropriate value.

dfs.permissions

Default: true
We: true
Renamed to dfs.permissions.enabled in Hadoop 0.21
We set this to final

This enables permission checking in HDFS. Unless you use Secure Hadoop (which we don't that's why I don't cover it here) it is still easy for anyone to read, write and delete anything on the cluster as there is no authentication of users done. So this is purely for safety reasons to avoid messing with the wrong data by accident.

dfs.replication

Default: 3
We: 3
Can be used in the client configuration

This is the default replication level used for new files in HDFS. if you change this value later on no existing files will be changed (that can be done on the command line though). Every file in HDFS can have a different replication level. This just sets the default.

dfs.block.size

Default: 67108864
We: 134217728
Renamed to dfs.blocksize in Hadoop 0.21
Can be used in the client configuration

This factor is on a per file basis and only used for new files. Files saved to HDFS are split in blocks at most this large (64 MB by default). This has multiple implications. The more blocks you have the more load there is on your NameNode. So if you have many files that are larger than the blocksize you might set this larger. If your files are mostly smaller than this you waste no space. All files only take as much space as they actually have data (this is unlike other file systems where a file takes up at least one block no matter how large it really is). So NameNode load (memory requirements as well) are one factor. The deciding factor for us to set this higher per default is that a lot of our calculations in MapReduce are very fast and Mappers finish quickly. As one Mapper usually processes one block and Mappers take a while to set up we chose a higher block size so that each Mapper has more data to process.

This can be set on a per file basis so you really have to find your own perfect value, perhaps even on a per dataset basis.

dfs.balance.bandwidthPerSec

Default: 1048576
We: 2097152
Renamed to dfs.datanode.balance.bandwidthPerSec in Hadoop 0.21
We set this to final

This property configures the amounts of bytes per second (default is 1 MB/s) that a DFS balancing operation can use per DataNode. The default is pretty low so we doubled it. We don't use a lot of bandwidth in our cluster at the moment so this is not a problem. Depends on your use case. The higher this number the faster balancing operations will complete. We run balancing every night on a cron job so we want it to be finished by morning.

dfs.hosts

Default: no default set
We: /etc/hadoop/conf/allowed_hosts
We set this to final

This file has to contain one name per line. Every name is the name of a DataNode that is allowed to connect to the NameNode. This prevents accidents like what happened to me: I test everything in Virtual Machines so I started a bunch of them, deployed the live config and forgot to change the NameNode so all of a sudden a bunch of Virtual Machines joined our HDFS cluster and blocks began replicating there.... So it is a good thing to explicitly list all allowed hosts in this file.

dfs.support.append

Default: false
We: true
As far as I know this option has been removed in Hadoop 0.21 and is enabled by default
We set this to final

This option has quite a history. To make it short: If you're using CDH3 set this to true, otherwise leave it false. You want/need this on true if you plan to use HBase.

dfs.datanode.max.xcievers

Default: 256
Safe: 1024
We: 2048
Yes, this is misspelt in Hadoop and it hasn't been fixed in Hadoop 0.21.
We set this to final

This is the maximum number of threads a DataNode may use (for example for file access to the local file system). There used to be bugs in Hadoop so that the default was a bit to low and needed to be set higher. Even today it's worth it to set it higher without a lot of risk. Especially if you're using HBase.

`mapred-site.xml`

HDFS is pretty straightforward to configure and benchmark. MapReduce is more of a black art unfortunately. I'll describe the MapReduce process here because it is important to understand where all the properties come in so you can safely change their values and tweak the performance. In my first draft of this post I wrote that I won't go into much detail on the internals of the MapReduce process. (Un-)fortunately this wasn't as easy as I thought and it has grown into a full blown explanation of everything I know. It is very possible that something's wrong here so please correct me if you see something that is off. And if you're not interested in how this works just skip to the descriptions of the properties itself.

All of this is valid for Hadoop 0.20.2+737 (the CDH version). I know that some things have changed in Hadoop 0.21 but that's left for another time.

The Map side

While a Map is running it is collecting output records in an in-memory buffer called MapOutputBuffer, if there are no reducers a DirectMapOutputCollector is used which makes most of the rest obsolete as it writes immediately to disk. The total size of this in memory buffer is set by the io.sort.mb property and defaults to 100 MB (which is converted to a byte value using a bit shift operation [100 << 20 = 104857600]). Out of these 100 MB io.sort.record.percent are reserved for tracking record boundaries. This property defaults to 0.05 (i.e. 5% which means 5 MB in the default case). Each record to track takes 16 bytes (4 integers of 4 bytes each) of memory which means the buffer can track 327680 map output records with the default settings. The rest of the memory (104857600 bytes - (16 bytes * 327680) = 99614720 bytes) is used to store the actual bytes to be collected (in the default case this will be 95 MB). While Map outputs are collected they are stored in the remaining memory and their location in the in-memory buffer is tracked as well. Once one of these two buffers reaches a threshold specified by io.sort.spill.percent, which defaults to 0.8 (i.e. 80%), the buffer is flushed to disk:

0.8 * 99614720 = 79691776
0.8 * 327680 = 262144

Look in the log output of your Maps and you'll see these three lines at the beginning of every log:

2010-12-05 01:33:04,912 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100
2010-12-05 01:33:04,996 INFO org.apache.hadoop.mapred.MapTask: data buffer = 79691776/99614720
2010-12-05 01:33:04,996 INFO org.apache.hadoop.mapred.MapTask: record buffer = 262144/327680

You should recognize these numbers!

Now while the Map is running you might see log lines like these:

2010-12-05 01:33:08,823 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: record full = true
2010-12-05 01:33:08,823 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 19361312; bufvoid = 99614720
2010-12-05 01:33:08,823 INFO org.apache.hadoop.mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
2010-12-05 01:33:09,558 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0

This means we've reached the maximum number of records we can track even though our buffer is still pretty empty (99614720 - 19361312 bytes still free). If however your buffer is the cause of your spill you'll see a line like this:

2010-12-05 01:33:08,823 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: buffer full = true

All of this spilling to disk is done in a separate thread so that the Map can continue running. That's also the reason why the spill begins early (when the buffer is only 80% full) so it doesn't fill up before a spill is finished. If one single Map output is too large to fit into the in memory buffer a single spill is done for this one value. A spill actually consists of one file per partition, meaning one file per Reducer.

After a Map task has finished there may be multiple spills on the TaskTracker. Those files have to be merged into one single sorted file per partition which is then fetched by the Reducers. The property io.sort.factor says how many of those spill files will be merged into one file at a time. The lower the number is the more passes will be required to arrive at the goal. The default is very low and it was considered to set the default to 100 (and in fact looking at the code it sometimes is set to 100 by default). This property can make a pretty huge difference if your Mappers output a lot of data. Not much memory is needed for this property but the larger it is the more open files there will be so make sure to set this to a reasonable value. To find such a value you should run a few MapReduce jobs that you'd expect to see in production use and carefully monitor the log files.

Watch out for log messages like these:

Merging <numSegments> sorted segments
Down to the last merge-pass, with <numSegments> segments left of total size: <totalBytes> bytes
Merging <segmentsToMerge.size()> intermediate segments out of a total of <totalSegments>

This is the process on the Map side where this factor is used. If your Mappers only have on spill file all of this doesn't matter. So if you try to benchmark this make sure to use a job with a lot of Map output data. If you only see a line like "Finished spill 0" but none of the above you're only producing one spill file which doesn't require any merging or further sorting. This is the ideal situation and you should try to get the number of spilled records/files as low as possible.

The Reduce side

The reduce phase has three different steps: Copy, Sort (which should really be called Merge) and Reduce.

During the Copy phase the Reducer tries to fetch the output of the Maps from the TaskTrackers and store it on the Reducer either in memory or on disk. The property mapred.reduce.parallel.copies (which defaults to 5) defines how many Threads are started per Reduce task to fetch Map output from the TaskTrackers.

Here's an example log from the beginning of a Reducer log:

2010-12-05 01:53:03,846 INFO org.apache.hadoop.mapred.ReduceTask: ShuffleRamManager: MemoryLimit=334063200, MaxSingleShuffleLimit=83515800
2010-12-05 01:53:03,879 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201012031527_0021_r_000103_0 Need another 1870 map output(s) where 0 is already in progress
2010-12-05 01:53:03,880 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201012031527_0021_r_000103_0 Thread started: Thread for merging on-disk files
2010-12-05 01:53:03,880 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201012031527_0021_r_000103_0 Thread waiting: Thread for merging on-disk files
2010-12-05 01:53:03,880 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201012031527_0021_r_000103_0 Thread started: Thread for merging in memory files
2010-12-05 01:53:03,881 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201012031527_0021_r_000103_0 Thread started: Thread for polling Map Completion Events

You can see two things in these log lines. First of all the ShuffleRamManager is started and afterwards you see that this Reducer needs to fetch 1870 map outputs (meaning we had 1870 Mappers). The map output is fetched and shuffled into memory (that's what the ShuffleRamManager is for). You can control its behavior using the mapred.job.shuffle.input.buffer.percent (default is 0.7). Runtime.getRuntime().maxMemory() is used to get the available memory which unfortunately returns slightly incorrect values so be careful when setting this. We'll get back to the last four lines later.

Our child tasks are running with -Xmx512m (536870912 bytes) so 70% of that should be 375809638 bytes but the ShuffleRamManager reports 334063200. No big deal, just be aware of it. There's a hardcoded limit of 25% of the buffer that a single map output may not surpass. If it is larger than that it will be written to disk (see the MaxSingleShuffleLimit value above: 334063200 * 0.25 = 83515800).

Now that everything's set up the copiers will start their work and fetch the output. You'll see a bunch of log lines like these:

2010-12-05 01:53:11,114 INFO org.apache.hadoop.mapred.ReduceTask: header: attempt_201012031527_0021_m_000011_0, compressed len: 454055, decompressed len: 454051
2010-12-05 01:53:11,114 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling 454051 bytes (454055 raw bytes) into RAM from attempt_201012031527_0021_m_000011_0
2010-12-05 01:53:11,133 INFO org.apache.hadoop.mapred.ReduceTask: Read 454051 bytes from map-output for attempt_201012031527_0021_m_000011_0
2010-12-05 01:53:11,133 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 from attempt_201012031527_0021_m_000011_0 -> (70, 6) from c1n7.gbif.org

In the first line you see that a map output was successfully copied and it could read the size of the data from the headers. The next line is actually what we've talked about earlier: The map output will now be decompressed (if it was compressed) and saved into memory using the ShuffleRamManager. The third line acknowledges that this succeeded. And the last line is information for a bug and should have been removed already according to a comment in the source code.

If for whatever reason the map output doesn't fit into memory you will see a similar log line to the second one above but "RAM" will be replaced by "Local-FS" and the fourth line will be missing. You obviously want as much data into memory as possible so shuffling on to the Local-FS is a warning sign or at least a sign for possible optimizations.

While all this goes on until all map outputs have been fetched there are two threads (Thread for merging on-disk files and Thread for merging in memory files) waiting for some conditions until they become active. The conditions are as follows:

The used memory in the in-memory buffer is above mapred.job.shuffle.merge.percent (default ist 66%, in our example that would mean 334063200 * 0.66 = 220481712 bytes) and there are at least two map outputs in the buffer
or there are more than mapred.inmem.merge.threshold (defaults to 1000) map outputs in the in-memory buffer, independent of the size
or if there are more than io.sort.factor * 2 -1 files on disk.

When one of the first two condition triggers you'll see something like this:

2010-12-05 01:53:42,106 INFO org.apache.hadoop.mapred.ReduceTask: Initiating in-memory merge with 501 segments...
2010-12-05 01:53:42,114 INFO org.apache.hadoop.mapred.Merger: Merging 501 sorted segments
...
2010-12-05 01:53:46,492 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201012031527_0021_r_000103_0 Merge of the 501 files in-memory complete. Local file is /mnt/disk1/hadoop/mapreduce/local/taskTracker/lfrancke/jobcache/job_201012031527_0021/attempt_201012031527_0021_r_000103_0/output/map_1.out of size 220545981
2

This could actually trigger the third condition as it writes a new file to disk. When that happens you'll see something like this:

2010-12-10 14:28:23,289 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201012101346_0001_r_000012_0We have  19 map outputs on disk. Triggering merge of 10 files

The io.sort.factor was set to the default of 10. 10 (out of the 19) files will be merged into one, leaving 10 on disk (i.e. io.sort.factor).

Both of these (the in-memory and the on-disk merge, the latter is also called Interleaved on-disk merge) will produce a new single output file and write it to disk. All of this is only going on as long as map outputs are still fetched. When that's finished we wait for running merges to finish but won't start any new ones in these threads:

2010-12-05 01:59:10,598 INFO org.apache.hadoop.mapred.ReduceTask: GetMapEventsThread exiting
2010-12-05 01:59:10,599 INFO org.apache.hadoop.mapred.ReduceTask: getMapsEventsThread joined.
2010-12-05 01:59:10,599 INFO org.apache.hadoop.mapred.ReduceTask: Closed ram manager
2010-12-05 01:59:10,599 INFO org.apache.hadoop.mapred.ReduceTask: Interleaved on-disk merge complete: 3 files left.
2010-12-05 01:59:10,599 INFO org.apache.hadoop.mapred.ReduceTask: In-memory merge complete: 314 files left.

As you can see by the timestamps no merges were running in our case so everything just shut down. During the copy phase we finished a total of three in-memory merges that's why we currently have three files on the disk. 314 more map outputs are still in the in-memory buffer. This concludes the Copy phase and the Sort phase begins:

2010-12-05 01:59:10,605 INFO org.apache.hadoop.mapred.Merger: Merging 314 sorted segments
2010-12-05 01:59:10,605 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 314 segments left of total size: 127512782 bytes
2010-12-05 01:59:13,903 INFO org.apache.hadoop.mapred.ReduceTask: Merged 314 segments, 127512782 bytes to disk to satisfy reduce memory limit
2010-12-05 01:59:13,904 INFO org.apache.hadoop.mapred.ReduceTask: Merging 4 files, 788519164 bytes from disk
2010-12-05 01:59:13,905 INFO org.apache.hadoop.mapred.ReduceTask: Merging 0 segments, 0 bytes from memory into reduce
2010-12-05 01:59:13,905 INFO org.apache.hadoop.mapred.Merger: Merging 4 sorted segments
2010-12-05 01:59:14,493 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 4 segments left of total size: 788519148 bytes

There are two things happening here. First of all the remaining 314 files that are still in memory are merged into one file on the disk (the first three lines). So now there are four files on the disk. These four files are merged into one.

There is an option mapred.job.reduce.input.buffer.percent which is set to 0 by default which allows the Reducer to keep some map output files in memory. The following is a snippet with this property set to 0.7:

2010-12-05 23:11:55,657 INFO org.apache.hadoop.mapred.ReduceTask: Merging 3 files, 661137901 bytes from disk
2010-12-05 23:11:55,660 INFO org.apache.hadoop.mapred.ReduceTask: Merging 312 segments, 127381881 bytes from memory into reduce
2010-12-05 23:11:55,661 INFO org.apache.hadoop.mapred.Merger: Merging 3 sorted segments
2010-12-05 23:11:55,688 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 661137889 bytes
2010-12-05 23:11:55,688 INFO org.apache.hadoop.mapred.Merger: Merging 313 sorted segments
2010-12-05 23:11:55,689 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 313 segments left of total size: 788519778 bytes

You can see that instead of merging the 312 segments from memory to disk they are kept in memory while the three files on disk are merged into one and all of the resulting 313 segments are streamed into the reducer.

There seems to be a bug in Hadoop though. I'm not 100% sure about this one so any insight would be appreciated. When the following conditions are true segments from the memory don't seem to be written to disk even if they should be according to the configuration:

There are segments in memory that should be written to disk before the reduce task begins according to mapred.job.reduce.input.buffer.percent
and there are more files on disk than io.sort.factor

If this happens you see this:

2010-12-10 16:39:40,671 INFO org.apache.hadoop.mapred.ReduceTask: Keeping 14 segments, 18888592 bytes in memory for intermediate, on-disk merge
2010-12-10 16:39:40,673 INFO org.apache.hadoop.mapred.ReduceTask: Merging 10 files, 4143441520 bytes from disk
2010-12-10 16:39:40,674 INFO org.apache.hadoop.mapred.ReduceTask: Merging 0 segments, 0 bytes from memory into reduce
2010-12-10 16:39:40,674 INFO org.apache.hadoop.mapred.Merger: Merging 24 sorted segments
2010-12-10 16:39:40,859 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 24 segments left of total size: 4143441480 bytes

So the steps being done in the Sort phase are the following:

Merge all segments (= map outputs) that are still in memory and don't fit into the memory specified by mapred.job.reduce.input.buffer.percent into one file on disk if there are less than io.sort.factor files on disk so we end up with at most io.sort.factor files on the disk after this step. If there are already io.sort.factor or more files on disk but there are map outputs that need to be written out of memory keep them in memory for now
1. In the first case you'll see a log message like this: Merged 314 segments, 127512782 bytes to disk to satisfy reduce memory limit
2. In the second case you'll see this: Keeping 14 segments, 18888592 bytes in memory for intermediate, on-disk merge
All files on disk and all remaining files in memory that need to be merged (case 1.b) are determined. You'll see a log message like this: "Merging 4 files, 788519164 bytes from disk".
All files that remain in memory during the Reduce phase are determined: "Merging 312 segments, 127381881 bytes from memory into reduce".
All files (on disk + in-memory) from step 2. are merged together using io.sort.factor as the merge factor. Which means that there might be intermediate merges to disk.
Merge all remaining in-memory (from step 3.) and on-disk files (from step 4.) into one stream to be read by the Reducer. This is done in a streaming fashion without writing new data to disk and just returning an Iterator to the Reduce phase.

This Iterator is given to the Reducer and so the Reduce phase starts.

Well, it turned out to be a rather detailed description of the process which is helpful to understand the configuration properties available to you. See below for a detailed list of all the relevant properties:
io.sort.factor

Default: 10
We: 100
Safe: 20-100
Renamed to mapreduce.task.io.sort.factor in Hadoop 0.21
Can be used in the client configuration

I've explained pretty thoroughly what this parameter does so I won't go into detail here. The whole situation with io.sort.factor and io.sort.mb is not ideal but as long as they are the options we have and the defaults are very low it is pretty safe to change them to a more reasonable value. It is worthwhile to take a look at your logs and search for the lines mentioned in the explanation above. This can be set on a per-job basis and for jobs that run frequently it's worth to find a good job specific value.

io.sort.mb

Default: 100
Renamed to mapreduce.task.io.sort.mb in Hadoop 0.21
Can be used in the client configuration

You can adjust the amount of memory used in the Mappers to collect Map outputs with this parameter. This parameter obviously depends heavily on the amount of memory you have available in total for your child VMs and on the memory requirements of your tasks. Your goal should be to minimize the amount of spilling that has to be performed as explained above and to utilize the available as best as possible. If your Map tasks don't need a lot of memory themselves you can use almost all available memory here. The default settings allocate 200 MB for child VMs and half of that is used for the output buffer so your Map tasks has about 100 MB available by default.

io.sort.record.percent

Default: 0.05
This has been removed in favor of automatic configuration in Hadoop 0.21
Can be used in the client configuration

The output buffer on the map side is split in two parts. One stores the actual bytes of the output data and the other one stores 16 bytes of metadata per output. This property specifies how much memory of the buffer (io.sort.mb) is used for tracking the metadata. The default is 5% and is often very low for jobs which output only small amounts of data in their map tasks. Look for lines indicating whether a spill to disk occurs because of record full = true. If this happens try to increase this value. This is another property which is very specific to the jobs you're running so it might need tuning for each and every job. Thankfully this mechanism has been replaced in Hadoop 0.21.

io.sort.spill.percent

Default: 0.8
Renamed to mapreduce.map.sort.spill.percent in Hadoop 0.21
Can be used in the client configuration

This property just configures when the data from the map output buffer will be written (spilled) to disk. The spilling process is running in a separate thread and output will be collected while it is running so it is important to start this process before the buffer is completely full as the map tasks will pause until there is space available.

mapred.job.tracker

Default: local
We: <jobtracker>:8021
We set this to final
Renamed to mapreduce.jobtracker.address in Hadoop 0.21
Needed on the clients

This lets the client know where to find the JobTracker and it lets the JobTracker know which port to bind to.

mapred.local.dir

Default: ${hadoop.tmp.dir}/mapred/local
We: /mnt/disk1/hadoop/mapreduce/local,/mnt/disk2/hadoop/mapreduce/local
We set this to final
Renamed to mapreduce.cluster.local.dir in Hadoop 0.21

This lets the MapReduce servers know where to store intermediate files. This may be a comma-separated list of directories to spread the load. Make sure there's enough space here for all your intermediate files. We share the same disks for MapReduce and HDFS.

mapred.system.dir

Default: ${hadoop.tmp.dir}/mapred/system
We: /hadoop/mapred/system
We set this to final
Renamed to mapreduce.jobtracker.system.dir in Hadoop 0.21

This is a folder in the defaultFS where MapReduce stores some control files. In our case that would be a directory in HDFS. If you have dfs.permissions (which it is by default) enabled make sure that this directory exists and is owned by mapred:hadoop.

mapred.temp.dir

Default: ${hadoop.tmp.dir}/mapred/temp
We: /tmp/mapreduce
We set this to final
Renamed to mapreduce.cluster.temp.dir in Hadoop 0.21

This is a folder to store temporary files in. It is hardly - if at all used. If I understand the description correctly this is supposed to be in HDFS but I'm not entirely sure by reading the source code. So we set this to a directory that exists on the local filesystem as well as in HDFS.

mapred.map.tasks

Default: 2
Renamed to mapreduce.job.maps in Hadoop 0.21

It is important to realize that this is just a hint for MapReduce as to the number of Maps it should use. In most cases this value is ignored and the actual number of Maps is dependent on the input data and generated automatically. For those rare cases where this value is used we set it to about 90% of our map slot capacity. This can be set client-side per job so if you have a job that relies on this property you better set it there to an appropriate value.

mapred.reduce.tasks

Default: 1
Renamed to mapreduce.job.reduces in Hadoop 0.21

This is different than the property for map tasks in that it is often not possible to calculate a "native" or optimal number of reduce tasks for a job. With this property you can specify the number of reduce tasks to start for a given job. The default is very low. The description suggests to set this to 99% of the cluster capacity so that all reduces finish in one wave. This is sensible when you use the default scheduler but as soon as multiple jobs run in parallel it's hard to guarantee that all reduces of one job finish in one wave. We're constantly playing around with this and currently have this at about 50% of our capacity.

This too can be specified on a per-job basis.

mapred.jobtracker.taskScheduler

Default: org.apache.hadoop.mapred.JobQueueTaskScheduler
We: org.apache.hadoop.mapred.FairScheduler
We set this to final
Renamed to mapreduce.jobtracker.taskscheduler in Hadoop 0.21

With the default configuration all jobs are placed in a priority FIFO queue and submitted one after the other. This is fine for testing but it doesn't utilize the available resources very well. This property allows you to change the scheduler used. These are the available schedulers in CDH3b3:

JobQueueTaskScheduler
FairScheduler
CapacityScheduler

Depending on the scheduler you decide to use there may be additional properties which I'm not going to mention here. Have a look at the dedicated documentation.

mapred.reduce.parallel.copies

Default: 5
We: ~20-50
We set this to final
Renamed to mapreduce.reduce.shuffle.parallelcopies in Hadoop 0.21

The reduce tasks have to fetch the map outputs from the remote servers. They have to fetch the output from each map of which there may be thousands. This option allows to parallelize the copy process. Tuning this to a value is very worthwhile. In our first tests this property gave us one of the best performance increases of all properties. We started to increase this property in steps of 5 and looked very carefully at the logs and our monitoring system to find a value that works for us. We've not yet finished this process but values between 20 and 50 seem to mostly work without problems.

mapred.tasktracker.map.tasks.maximum & mapred.tasktracker.reduce.tasks.maximum

Default: 2
We set this to final
Renamed to mapreduce.tasktracker.map.tasks.maximum & mapreduce.tasktracker.map.tasks.maximum in Hadoop 0.21

This setting is very important and we've yet to find values that we are comfortable with. This setting can be different on each TaskTracker and defines how many map or reduce task "slots" there are on a specific TaskTracker. You need to set these to values that don't overload your servers while still fully utilizing them. You also need to make sure that there's enough memory for all tasks and services running on a server (see mapred.child.java.opts). By setting this property to different values depending on your server configuration you can easily use heterogeneous hardware in your cluster. Each distinct hardware configuration will have these properties set to different values. A general rule from the Hadoop Definitive Guide book says that these properties can be set to number of cores - 1. We've tried various settings now but found the load on the servers to be very high with those settings so we'll have to do more benchmarking.

mapred.child.java.opts

Default: -Xmx200m
Can be used in the client configuration

These are the options given to each child JVM started (map- and reduce tasks). The default just sets the maximum memory to 200 MB. This can be set on the client to pass options needed for a specific job. GC logging for example can be enabled as well. This isn't configurable on a per TaskTracker basis so you have to make sure that every machine in your cluster fulfills the requirements. Available memory needs to be at least (mapred.tasktracker.map.tasks.maximum + mapred.tasktracker.reduce.tasks.maximum) * Xmx.

mapred.inmem.merge.threshold

Default: 1000
We: 0
Renamed to mapreduce.reduce.merge.inmem.threshold in Hadoop 0.21
Can be used in the client configuration

I've explained the effect of this property in the MapReduce description above but to reiterate: The reduce side fetches map outputs to memory. Once the memory is full or this many map outputs are in memory they are merged together to one file on the disk. This can be set on a per job basis but as a default we've disabled this behavior and just flush to disk when the memory is full. This seems to have been better for all our jobs so far but it's definitely a property to look out for.

mapred.job.shuffle.merge.percent

Default: 0.66
Renamed to mapreduce.reduce.shuffle.merge.percent in Hadoop 0.21
Can be used in the client configuration

Once the memory buffer in the copy (shuffle) phase of the reduce task is this full a background thread will start to merge all map outputs collected in memory so far and write them to a single file on disk. This is similar to what's happening on the map side. In the default configuration the mapred.inmem.merge.threshold parameter might actually trigger a merge before this value is hit. We haven't yet played around with this property but you'd have to be careful to turn it not too high so that the copy processes have to wait for the buffer to be empty again. That could be a huge performance hit.

An addition to Hadoop's logging would be nice that lets us know how full the buffer is the moment a merge finishes.

mapred.job.shuffle.input.buffer.percent

Default: 0.7
Renamed to mapreduce.reduce.shuffle.input.buffer.percent in Hadoop 0.21
Can be used in the client configuration

This is the amount of memory from the total available memory (specified by mapred.child.java.opts) that's allocated for collecting map outputs in memory on the reduce side. Another parameter we haven't played around with but my guess would be that this can be easily set a little bit higher.

mapred.job.reduce.input.buffer.percent

Default: 0.0
Renamed to mapreduce.reduce.input.buffer.percent in Hadoop 0.21
Can be used in the client configuration

Usually map outputs would be written to disk when the sort phase (on the reduce) ends. If you have reduce tasks that don't need a lot of memory themselves you can set this to a higher value so that map outputs up to this amount of memory (in percent of the total available memory) aren't written to disk but kept in memory. This is obviously faster than an intermediate spill to disk. Should be considered on a per-job basis.

mapred.map.tasks.speculative.execution & mapred.reduce.tasks.speculative.execution

Default: true
We: false
We will set this to final once we're in production
Renamed to mapreduce.map.speculative & mapreduce.reduce.speculative in Hadoop 0.21
Can be used in the client configuration

Speculative Execution starts multiple instances of certain map or reduce tasks when it detects certain circumstances (like an unusually slow task or node) to avoid waiting for stragglers too long. This sounds like a good idea and we've got it enabled at the moment but when we go to production this will probably be disabled as it uses valuable resources on the cluster that mostly goes to waste and while one job may finish faster all the others have to wait longer.

mapred.job.reuse.jvm.num.tasks

Default: 1
Renamed to mapreduce.job.jvm.numtasks in Hadoop 0.21
Can be used in the client configuration

Child JVMs are spawned for the map and reduce tasks. This parameter lets you reuse these VMs for multiple tasks. The default value creates a new JVM for each task which has some overhead (the book says about one second per JVM). We've played around with it a bit and it can make things faster but you've got to be careful with memory leaks and shared state. Basically you should be sure that your jobs can handle this. If you have a performance critical job you can play around with this but we've had some OutOfMemory errors when using this so we're conservative at the moment. If you set it to -1 a JVM will never be destroyed.

tasktracker.http.threads

Default: 40
We: 80
We set this to final
Renamed to mapreduce.tasktracker.http.threads in Hadoop 0.21

The map output is fetched by the reducers from the TaskTrackers via HTTP. This property lets you adjust the number of threads that server those requests. When we upped the parallel copies we had some errors about fetch-failures so we slowly increased this value. Those two parameters need to be carefully tuned. 80 seemed to cause no problems for us so we stuck to it for now. You have to restart your TaskTrackers after changing this value.

mapred.compress.map.output

Default: false
We: true
Renamed to mapreduce.map.output.compress in Hadoop 0.21
Can be used in the client configuration

Turning this on will compress the output of your Mappers using SequenceFile compression. Depending on the codec you chose this computation may be CPU intensive and result in varying degrees of compression. We've benchmarked jobs of different sizes with this intermediate compression enabled and disabled and while some of them took slightly longer than before it is still good to enable it. The cost isn't too high and there is a lot less intermediate data generated. Less I/O in general is good especially if multiple jobs are running.

mapred.map.output.compression.codec

Default: org.apache.hadoop.io.compress.DefaultCodec
We: com.hadoop.compression.lzo.LzoCodec
Renamed to mapreduce.map.output.compress.codec in Hadoop 0.21
Can be used in the client configuration

With this property you specify the specific compression codec to use for Map output compression. So far we've only tried LZO. This choice was based on the experience of others and the general properties of the algorithm being very fast but sacrificing a bit of compression efficiency for its speed. We plan to test the other algorithms as well.

mapred.hosts

Default: no default set
We: /etc/hadoop/conf/allowed_hosts
Renamed to mapreduce.jobtracker.hosts.filename in Hadoop 0.21
We set this to final

This is the same as dfs.hosts just specifies which TaskTrackers are allowed to get work from the JobTracker. They both have the same format so it's quite common for them to be the same file.

Conclusion

After setting up all these parameters the way you like them you should have a fully functional but basic Hadoop cluster running. You can submit jobs, use HDFS etc. But there are a few more things that we can do like installing Hive, Hue, Pig, Sqoop, etc. We've also yet to cover Puppet. All of this is hopefully forthcoming in more blog posts in the future.

We're also very interested in other users (or interested people and companies) of Hadoop, HBase & Co. in Scandinavia who would be interested in a Hadoop Meetup. We're located in Copenhagen. Contact us if you're interested.

If you have any questions or spot any problems or mistakes please let me know in the comments or by mail.

How To Configure GeoServer For The GBIF Data Portal

2010-05-12T22:33:00.006+02:00

We are Daniel Amariles and Hector Tobon, we work for the International Center for Tropical Agriculture, based in Cali, Colombia, and we are currently correcting some bugs on the GBIF Data Portal.

A few days ago we had to install Geoserver in order to deal with some map issues from the GBIF Data Portal. We want to take this opportunity to document our experience (installation and configuration) so that anybody inside the community can benefit from this knowledge. This post refers to a customised installation of GeoServer for the GBIF Data Portal.

Installing GeoServer

There are different ways to install GeoServer:

Install Geoserver as a WAR in Apache Tomcat
Install Geoserver as a binary file (OS independent)

Configuring system variables

Configure the following variables according to your system configuration.

JAVA_HOME=Your JAVA installation path
JAVA_OPTS="-XX:PermSize=512M -Xmx1g -Djava.awt.headless=true -Dcom.sun.management.jmxremote"

Install GeoServer as a WAR in Apache tomcat (recommended)

Get the Apache Tomcat from http://tomcat.apache.org/ (Binary Distributions - Core)
In this case we are going to use Apache Tomcat Zipped Binary Distribution 6.0.26

Get the GeoServer WAR file from http://geoserver.org/display/GEOS/Stable (Web Archive Format)
In This case we are going to use GeoServer 2.0.1

Unzip the tomcat file and place it in a location of your choice and put the GeoServer WAR file in the folder $TOMCAT/webapps/geoserver.war

Start Tomcat by running $TOMCAT/bin/startup.sh or $TOMCAT/bin/startup.bat according to your OS.

Go to URL http://localhost:8080/geoserver and sign in using the default username: admin password: geoserver

How to change the port

Shutdown Tomcat running $TOMCAT/bin/shutdown.sh or $TOMCAT/bin/shutdown.bat
Open file $TOMCAT/conf/server.xml
Search line <connector port="8080" protocol="HTTP/1.1" ... and change the default to your convenience port.
Start Tomcat

Install GeoServer as Binary (OS independent)

Note: If you install Geoserver as Binary it will be run by a jetty server rather than Tomcat.

Get the bin folder from http://geoserver.org/display/GEOS/Stable (Binary - OS independent)
In this case we are going to use GeoServer 2.0.1

Unzip the folder and place it in a location of your choice.

Run $GEOSERVER/bin/startup.bat or $GEOSERVER/bin/startup.sh according to your OS.

Go to URL http://localhost:8080/geoserver and sign in using the default username: admin password: geoserver

How to change the port

Shutdown Jetty server running
$GEOSERVER/bin/shutdown.sh or $GEOSERVER/bin/shutdown.bat
Open file $GEOSERVER/etc/jetty.xml
Search line <Set name="port">8081</Set>
Start Jet

Resources & libraries

Shapefile

Copy the files corresponding to the country shape file

country.dbf country.fix country.ORG.dbf country.prj country.qix country.shp country.shx

to the shapefiles directory of the GeoServer $GEOSERVER/data_dir/data/shapefiles

The gt-ala-tab library

Copy the library gt-ala-tab-1.0-SNAPSHOT.jar to the libraries directory of the GeoServer $GEOSERVER\geoserver\WEB-INF\lib

Workspace

Create the Workspace named gbif with URI http://www.gbif.org and set it as default workspace

Stores

country stores

Add the stores type Shapefile with names country_borders, country_fill and country_names. With the following parameters:

Workspace: gbif
URL: file:data/shapefiles/country.shp

tab_density store

Add the store tab_density of type Tab Url DataStore.With the following parameters:

Workspace: gbif
minx: -180
miny: -90
maxx: 180
maxy: 90

Styles

Add the following SLD styles. Click on the validate button to verify the style is a valid SLD document.

country_borders style

<?xml version="1.0" encoding="ISO-8859-1"?> 
    <StyledLayerDescriptor version="1.0.0" xmlns="http://www.opengis.net/sld" xmlns:ogc="http://www.opengis.net/ogc" 
      xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
      xsi:schemaLocation="http://www.opengis.net/sld http://schemas.opengis.net/sld/1.0.0/StyledLayerDescriptor.xsd"> 
      <NamedLayer> 
        <Name>Countries Borders</Name> 
        <UserStyle> 
          <Title>Countries Borders</Title> 
          <Abstract>A style that just draws a 1 pixel stroke around each countries borders</Abstract> 
          <FeatureTypeStyle> 
            <Rule> 
              <Title>Polygon</Title> 
              <PolygonSymbolizer> 
                <Stroke> 
                  <CssParameter name="stroke">#006600</CssParameter> 
                  <CssParameter name="stroke-width">1</CssParameter> 
                </Stroke> 
              </PolygonSymbolizer> 
            </Rule> 
          </FeatureTypeStyle> 
        </UserStyle> 
      </NamedLayer> 
    </StyledLayerDescriptor>

country_fill style

<?xml version="1.0" encoding="ISO-8859-1"?> 
    <StyledLayerDescriptor version="1.0.0" xmlns="http://www.opengis.net/sld" xmlns:ogc="http://www.opengis.net/ogc" 
      xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
      xsi:schemaLocation="http://www.opengis.net/sld http://schemas.opengis.net/sld/1.0.0/StyledLayerDescriptor.xsd"> 
      <NamedLayer> 
        <Name>Country Polygons Fill</Name> 
        <UserStyle> 
          <Title>Country Polygons style</Title> 
          <Abstract>A style that fills the countries polygons with a specific color</Abstract> 
          <FeatureTypeStyle> 
            <Rule> 
              <Title>Polygon</Title> 
              <PolygonSymbolizer> 
                <Fill> 
                  <CssParameter name="fill">#003333</CssParameter> 
                  <CssParameter name="fill-opacity"><ogc:Literal>1</ogc:Literal></CssParameter>               
                </Fill> 
              </PolygonSymbolizer> 
            </Rule> 
          </FeatureTypeStyle> 
        </UserStyle> 
      </NamedLayer> 
    </StyledLayerDescriptor>

country_names style

<?xml version="1.0" encoding="ISO-8859-1"?> 
<StyledLayerDescriptor version="1.0.0" xmlns="http://www.opengis.net/sld" xmlns:ogc="http://www.opengis.net/ogc" 
  xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
  xsi:schemaLocation="http://www.opengis.net/sld http://schemas.opengis.net/sld/1.0.0/StyledLayerDescriptor.xsd"> 
  <NamedLayer> 
    <Name>Country Names</Name> 
    <UserStyle> 
      <Title>Country Names</Title> 
      <Abstract>Style that renders the names of the countries on the map</Abstract> 
      <FeatureTypeStyle> 
  <Rule> 
  <TextSymbolizer> 
    <Label> 
      <ogc:PropertyName>CNTRY_NAME</ogc:PropertyName> 
    </Label>   
    <Font> 
      <CssParameter name="font-family"><ogc:Literal>Lucida Sans</ogc:Literal></CssParameter> 
      <CssParameter name="font-style"><ogc:Literal>normal</ogc:Literal></CssParameter> 
      <CssParameter name="font-size"><ogc:Literal>10.0</ogc:Literal></CssParameter> 
      <CssParameter name="font-weight"><ogc:Literal>bold</ogc:Literal></CssParameter> 
    </Font>   
<LabelPlacement> 
<PointPlacement> 
<AnchorPoint> 
<AnchorPointX>0.5</AnchorPointX> 
<AnchorPointY>0.5</AnchorPointY> 
</AnchorPoint> 
</PointPlacement> 
</LabelPlacement> 
    <Fill> 
      <CssParameter name="fill">#6e8686</CssParameter> 
    </Fill> 
  </TextSymbolizer> 
  </Rule> 
      </FeatureTypeStyle> 
    </UserStyle> 
  </NamedLayer> 
</StyledLayerDescriptor>

density_layer style

<?xml version="1.0" encoding="UTF-8"?> 
<StyledLayerDescriptor version="1.0.0" 
 xsi:schemaLocation="http://www.opengis.net/sld StyledLayerDescriptor.xsd" 
 xmlns="http://www.opengis.net/sld" xmlns:ogc="http://www.opengis.net/ogc" 
 xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> 
 <NamedLayer> 
   <Name>densityLayer</Name> 
   <UserStyle> 
     <FeatureTypeStyle> 
       <!-- If it is a point data, render as such --> 
       <Rule> 
         <ogc:Filter> 
           <ogc:PropertyIsEqualTo> 
             <ogc:Function name="geometryType"> 
               <ogc:PropertyName>geom</ogc:PropertyName> 
             </ogc:Function> 
             <ogc:Literal>Point</ogc:Literal> 
           </ogc:PropertyIsEqualTo> 
         </ogc:Filter> 
         <PointSymbolizer> 
           <Graphic> 
             <Mark> 
               <WellKnownName>circle</WellKnownName>
               <Fill> 
                 <CssParameter name="fill">#cc0000</CssParameter> 
                 <CssParameter name="fill-opacity">1.0</CssParameter> 
               </Fill> 
             </Mark> 
             <Size>6</Size> 
           </Graphic> 
         </PointSymbolizer> 
       </Rule> 
       <!-- 1-9 --> 
       <Rule> 
         <ogc:Filter> 
           <ogc:And> 
             <ogc:PropertyIsGreaterThanOrEqualTo> 
               <ogc:PropertyName>count</ogc:PropertyName> 
               <ogc:Literal>1</ogc:Literal> 
             </ogc:PropertyIsGreaterThanOrEqualTo> 
             <ogc:PropertyIsLessThan> 
               <ogc:PropertyName>count</ogc:PropertyName> 
               <ogc:Literal>10</ogc:Literal> 
             </ogc:PropertyIsLessThan> 
           </ogc:And> 
         </ogc:Filter> 
         <PolygonSymbolizer> 
           <Fill> 
             <CssParameter name="fill"> 
               <ogc:Literal>#ffff00</ogc:Literal> 
             </CssParameter> 
             <CssParameter name="fill-opacity"> 
               <ogc:Literal>1</ogc:Literal> 
             </CssParameter> 
           </Fill> 
          <Stroke> 
            <CssParameter name="stroke">#ffff00</CssParameter> 
            <CssParameter name="stroke-width">0</CssParameter> 
          </Stroke> 
         </PolygonSymbolizer> 
       </Rule> 
       <!-- 10-99 --> 
       <Rule> 
         <ogc:Filter> 
           <ogc:And> 
             <ogc:PropertyIsGreaterThanOrEqualTo> 
               <ogc:PropertyName>count</ogc:PropertyName> 
               <ogc:Literal>10</ogc:Literal> 
             </ogc:PropertyIsGreaterThanOrEqualTo> 
             <ogc:PropertyIsLessThan> 
               <ogc:PropertyName>count</ogc:PropertyName> 
               <ogc:Literal>100</ogc:Literal> 
             </ogc:PropertyIsLessThan> 
           </ogc:And> 
         </ogc:Filter> 
         <PolygonSymbolizer> 
           <Fill> 
             <CssParameter name="fill"> 
               <ogc:Literal>#ffcc00</ogc:Literal> 
             </CssParameter> 
             <CssParameter name="fill-opacity"> 
               <ogc:Literal>1</ogc:Literal> 
             </CssParameter> 
           </Fill> 
          <Stroke> 
            <CssParameter name="stroke">#ffcc00</CssParameter> 
            <CssParameter name="stroke-width">0</CssParameter> 
          </Stroke> 
         </PolygonSymbolizer> 
       </Rule> 
       <!-- 100-999 --> 
       <Rule> 
         <ogc:Filter> 
           <ogc:And> 
             <ogc:PropertyIsGreaterThanOrEqualTo> 
               <ogc:PropertyName>count</ogc:PropertyName> 
               <ogc:Literal>100</ogc:Literal> 
             </ogc:PropertyIsGreaterThanOrEqualTo> 
             <ogc:PropertyIsLessThan> 
               <ogc:PropertyName>count</ogc:PropertyName> 
               <ogc:Literal>1000</ogc:Literal> 
             </ogc:PropertyIsLessThan> 
           </ogc:And> 
         </ogc:Filter> 
         <PolygonSymbolizer> 
           <Fill> 
             <CssParameter name="fill"> 
               <ogc:Literal>#ff9900</ogc:Literal> 
             </CssParameter> 
             <CssParameter name="fill-opacity"> 
               <ogc:Literal>1</ogc:Literal> 
             </CssParameter> 
           </Fill> 
          <Stroke> 
            <CssParameter name="stroke">#ff9900</CssParameter> 
            <CssParameter name="stroke-width">0</CssParameter> 
          </Stroke> 
         </PolygonSymbolizer> 
       </Rule> 
       <!-- 1000-9999 --> 
       <Rule> 
         <ogc:Filter> 
           <ogc:And> 
             <ogc:PropertyIsGreaterThanOrEqualTo> 
               <ogc:PropertyName>count</ogc:PropertyName> 
               <ogc:Literal>1000</ogc:Literal> 
             </ogc:PropertyIsGreaterThanOrEqualTo> 
             <ogc:PropertyIsLessThan> 
               <ogc:PropertyName>count</ogc:PropertyName> 
               <ogc:Literal>10000</ogc:Literal> 
             </ogc:PropertyIsLessThan> 
           </ogc:And> 
         </ogc:Filter> 
         <PolygonSymbolizer> 
           <Fill> 
             <CssParameter name="fill"> 
               <ogc:Literal>#ff6600</ogc:Literal> 
             </CssParameter> 
             <CssParameter name="fill-opacity"> 
               <ogc:Literal>1</ogc:Literal> 
             </CssParameter> 
           </Fill> 
          <Stroke> 
            <CssParameter name="stroke">#ff6600</CssParameter> 
            <CssParameter name="stroke-width">0</CssParameter> 
          </Stroke> 
         </PolygonSymbolizer> 
       </Rule> 
       <!-- 10000-99999 --> 
       <Rule> 
         <ogc:Filter> 
           <ogc:And> 
             <ogc:PropertyIsGreaterThanOrEqualTo> 
               <ogc:PropertyName>count</ogc:PropertyName> 
               <ogc:Literal>10000</ogc:Literal> 
             </ogc:PropertyIsGreaterThanOrEqualTo> 
             <ogc:PropertyIsLessThan> 
               <ogc:PropertyName>count</ogc:PropertyName> 
               <ogc:Literal>100000</ogc:Literal> 
             </ogc:PropertyIsLessThan> 
           </ogc:And> 
         </ogc:Filter> 
         <PolygonSymbolizer> 
           <Fill> 
             <CssParameter name="fill"> 
               <ogc:Literal>#ff3300</ogc:Literal> 
             </CssParameter> 
             <CssParameter name="fill-opacity"> 
               <ogc:Literal>1</ogc:Literal> 
             </CssParameter> 
           </Fill> 
          <Stroke> 
            <CssParameter name="stroke">#ff3300</CssParameter> 
            <CssParameter name="stroke-width">0</CssParameter> 
          </Stroke> 
         </PolygonSymbolizer> 
       </Rule> 
       <!-- 100000+ --> 
       <Rule> 
         <ogc:Filter> 
           <ogc:PropertyIsGreaterThanOrEqualTo> 
             <ogc:PropertyName>count</ogc:PropertyName> 
             <ogc:Literal>100000</ogc:Literal> 
           </ogc:PropertyIsGreaterThanOrEqualTo> 
         </ogc:Filter> 
         <PolygonSymbolizer> 
           <Fill> 
             <CssParameter name="fill"> 
               <ogc:Literal>#cc0000</ogc:Literal> 
             </CssParameter> 
             <CssParameter name="fill-opacity"> 
               <ogc:Literal>1</ogc:Literal> 
             </CssParameter> 
           </Fill> 
          <Stroke> 
            <CssParameter name="stroke">#cc0000</CssParameter> 
            <CssParameter name="stroke-width">0</CssParameter> 
          </Stroke> 
         </PolygonSymbolizer> 
       </Rule> 
     </FeatureTypeStyle> 
   </UserStyle> 
 </NamedLayer> 
</StyledLayerDescriptor>

country_borders_black style

<?xml version="1.0" encoding="ISO-8859-1"?>
<StyledLayerDescriptor version="1.0.0" xmlns="http://www.opengis.net/sld" xmlns:ogc="http://www.opengis.net/ogc"
 xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://www.opengis.net/sld http://schemas.opengis.net/sld/1.0.0/StyledLayerDescriptor.xsd">
 <NamedLayer>
   <Name>Countries Borders</Name>
   <UserStyle>
     <Title>Countries Borders</Title>
     <Abstract>A style that just draws a 1 pixel stroke around each countries borders</Abstract>
     <FeatureTypeStyle>
       <Rule>
         <Title>Polygon</Title>
         <PolygonSymbolizer>
           <Stroke>
             <CssParameter name="stroke">#000000</CssParameter>
             <CssParameter name="stroke-width">1</CssParameter>
           </Stroke>
         </PolygonSymbolizer>
       </Rule>
     </FeatureTypeStyle>
   </UserStyle>
 </NamedLayer>
</StyledLayerDescriptor>

country_names_black style

<?xml version="1.0" encoding="ISO-8859-1"?>
<StyledLayerDescriptor version="1.0.0" xmlns="http://www.opengis.net/sld" xmlns:ogc="http://www.opengis.net/ogc"
 xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://www.opengis.net/sld http://schemas.opengis.net/sld/1.0.0/StyledLayerDescriptor.xsd">
 <NamedLayer>
   <Name>Country Names</Name>
   <UserStyle>
     <Title>Country Names</Title>
     <Abstract>Style that renders the names of the countries on the map</Abstract>
     <FeatureTypeStyle>
 <Rule>
 <TextSymbolizer>
   <Label>
     <ogc:PropertyName>CNTRY_NAME</ogc:PropertyName>
   </Label>
   <Font>
     <CssParameter name="font-family"><ogc:Literal>Lucida Sans</ogc:Literal></CssParameter>
     <CssParameter name="font-style"><ogc:Literal>normal</ogc:Literal></CssParameter>
     <CssParameter name="font-size"><ogc:Literal>13.0</ogc:Literal></CssParameter>
     <CssParameter name="font-weight"><ogc:Literal>bold</ogc:Literal></CssParameter>
   </Font>
 <LabelPlacement>
   <PointPlacement>
     <AnchorPoint>
       <AnchorPointX>0.5</AnchorPointX>
       <AnchorPointY>0.5</AnchorPointY>
     </AnchorPoint>
   </PointPlacement>
 </LabelPlacement>
   <Fill>
     <CssParameter name="fill">#6666FF </CssParameter>
   </Fill>
 </TextSymbolizer>
 </Rule>
     </FeatureTypeStyle>
   </UserStyle>
 </NamedLayer>
</StyledLayerDescriptor>

Layers

Add the following layers in the layers menu

Adding the country layers

Add the layers from the stores gbif:country_borders, gbif:country_fill and gbif:country_names, respectively with the names country_borders, country_fill y country_names. With the following parameters:

Data tab

Coordinate Reference Systems: The Native SRS is UNKNOWN and the Declared SRS is EPSG:4326.
Bounding Boxes: In both Native Bounding Box and Lat/Lon Bounding Box, set the following:

Min X	Min Y	Max X	Max Y
-180	-90	180	83,623

Publishing tab

Default Title: Set the Default Style as the respective layer name.

Adding the tabDensityLayer layer

Add the layer tabDensityLayer from the store gbif:tab_density. With the following parameters.

Data tab

Coordinate Reference Systems: The Declared SRS is EPSG:4326.
Bounding Boxes: In both Native Bounding Box and Lat/Lon Bounding Box, set the following:

Min X	Min Y	Max X	Max Y
-180	-90	180	90

Publishing tab

Default Title: Set the Default Style as the respective layer name.

Test the layers

To test if the country layers are working well, enter to the GWC GeoWebCache, then to "A list of all the layers and automatic demos" and check the behavior of the country layers, they should work according to their names.

Struts 2.1.6 and GUICE

2009-10-05T16:23:00.004+02:00

Google GUICE is a great lightweight dependency injection framework that comes with a plugin for struts2. Using guice 2.0 and struts2.1.6 on tomcat6 I run into a dependency problem with different xwork and ognl jars provided by struts and the struts2 plugin from guice:

[INFO] +- org.apache.struts:struts2-core:jar:2.1.6:compile
[INFO] |  +- com.opensymphony:xwork:jar:2.1.2:compile
[INFO] |  |  \- org.springframework:spring-test:jar:2.5.6:test (scope managed from compile)
[INFO] |  +- opensymphony:ognl:jar:2.6.11:compile
[INFO] |  \- commons-fileupload:commons-fileupload:jar:1.2.1:compile
...
[INFO] +- com.google.inject.extensions:guice-struts2-plugin:jar:2.0:compile
[INFO] |  \- opensymphony:xwork:jar:2.0.0:compile
[INFO] |     \- ognl:ognl:jar:2.6.9:compile

This can easily be resolved excluding the older OGNL library from guice like this in your POM:

    <dependency>
      <groupid>com.google.inject.extensions</groupid>
      <artifactid>guice-struts2-plugin</artifactid>
      <version>2.0</version>
        <exclusions>
        <exclusion>
          <groupid>opensymphony</groupid>
          <artifactid>xwork</artifactid>
        </exclusion>
        <exclusion>
          <groupid>ognl</groupid>
          <artifactid>ognl</artifactid>
        </exclusion>
      </exclusions>
    </dependency>

Darwin Core Archive Reader part1

2009-07-17T23:37:00.006+02:00

After writing software that produces Darwin Core Archives (DwC-A), I thought it is time to introduce a little client library that can read DwC-A and makes it very simple to consume it. To create DwC-A there are a couple of resources available:

The Darwin Core Text Guidelines, pretty much the specification
The ECAT Checklist Format gives some best practices and focus on how to encode taxonomic data as DwC-A
The IPT produces DwC-A and has a rich web interface but is limited to ~1 million records
The IPT-lite produces large DwC-A datasets very fast, but lacks any visualisation of the data itself. Its a great way of quickly creating DwC-A and hosting it online

The DwC Archive reader library is under active development and currently being used for indexing checklists. The project is hosted as part of the GBIF Indexing and Harvesting Toolkit, where a jar with all dependencies is hosted for download too. You can use this jar in the terminal to inspect archives like this:

$ java -jar DwcA-reader-1.0-SNAPSHOT.jar hershkovitz.txt

Opening archive: /Users/markus/Desktop/hershkovitz/hershkovitz.txt
Core file(s) found: [/Users/markus/Desktop/hershkovitz/hershkovitz.txt]
Core row type:
Core identifier column: 0
Cannot locate term dwc:kingdom
Cannot locate term dwc:family
Number of extensions 0
Archive contains 3249 core records.

The reader can handle just a simple CSV or tab file, or you can point it to a dwc archive folder with a meta.xml descriptor and several data files. The iterator(s) allow you to walk the core records of the archive while conveniently retrieving all extension records at the same time. You can also tell it to show some core record, passing a limit and offset like this from the commandline:

java -jar DwcA-reader-1.0-SNAPSHOT.jar hershkovitz 10 25

A simple dummy source code example on how to use this reader in your code is shown here:

package org.gbif.dwc.text;

import java.io.File;
import java.io.IOException;
import java.util.Iterator;

import org.gbif.dwc.model.DarwinCoreRecord;
import org.gbif.dwc.model.ExtensionRecord;
import org.gbif.dwc.model.StarRecord;
import org.gbif.dwc.terms.DwcTerm;

public class UsageExample {

 public static void main(String[] args) throws IOException, UnsupportedArchiveException {
  // opens csv files with headers or dwc-a direcotries with a meta.xml descriptor
  Archive arch = ArchiveFactory.openArchive(new File("pontaurus.txt"));

  // does scientific name exist?
  if (!arch.getCore().hasTerm(DwcTerm.scientificName)){
   System.out.println("This application requires dwc-a with scientific names");
   System.exit(1);
  }

  // loop over core darwin core records
  Iterator iter = arch.iteratorDwc();
  DarwinCoreRecord dwc;
  while(iter.hasNext()){
   dwc = iter.next();
   System.out.println(dwc);
  }

  // loop over star records. i.e. core with all linked extension records
  for (StarRecord rec : arch){
   // print core ID + scientific name
   System.out.println(rec.id()+" - "+rec.value(DwcTerm.scientificName));
   for (ExtensionRecord erec : rec){
    // does this extension have Long/lat?
    if (rec.dataFile().hasTerm(DwcTerm.decimalLongitude) && rec.dataFile().hasTerm(DwcTerm.decimalLatitude)){
     System.out.println("Georeferenced: " + rec.value(DwcTerm.decimalLongitude)+","+rec.value(DwcTerm.decimalLatitude));;
    }
    
   }
  }
 }
}

Profiling memory usage of various String collections

2009-06-23T15:15:00.017+02:00

Wanting to know the memory footprint and performance of different java options to keep simple string lookups in memory, I profiled different java.util collection classes that are filled with the same list of strings to see how much the memory usage differs. I then loaded the same data into 2 lucene in memory indices using lucenes RAMDirectory. I finally also evaluated an embedded file based & in memory H2 database. The data that has been loaded are 1.573.345 scientific name strings, the longest being about 150 characters. The original uncompressed text file is 31.6MB (zipped 8.1MB). To also test ID lookup in case of java.util.map or the KVP lucene index, the row number of each name has been used. The machine I used for testing was a MacPro 8-core 3GHz, 5GB RAM using Java6 with 2GB of memory (-Xmx2g) on Mac OSX 64bit. Here are the shortened results using System.currentTimeMillis() and JProfiler inspecting deep object copies in heap dumps (a seriously memory intensive thing too in some cases like lucene and H2 which even crashed):


Text file: 31.6 MB
Zipped: 8.1 MB

java.util.HashSet
# 264MB
# contains test of 10.000 x 12 terms took 28 msecs
# uses HashMap internally...

java.util.TreeSet
# 256MB
# contains test of 10.000 x 12 terms took 98 msecs
# uses TreeMap internally...

java.util.HashMap
# 300MB
# contains (key) test of 10.000 x 12 terms took 21 msecs
# includes Integer values as opposed to above Set

java.util.TreeMap
# 292MB
# contains (key) test of 10.000 x 12 terms took 93 msecs
# includes Integer values as opposed to above Set

java.util.ArrayList
# 172MB
# contains test of 100 x 12 terms took 43257 msecs

java.util.LinkedList
# 220MB
# contains test of 100 x 12 terms took 50564 msecs

String[] array
# 172MB


org.apache.commons.collections.map.HashedMap
# 384 MB
# contains (key) test of 10000 x 12 terms took 23 msecs
# no generics support !


javolution.util.FastList
# 220 MB
# contains test of 100 x 12 terms took 58886 msecs

javolution.util.FastMap
# 396 MB
# contains (key) test of 10000 x 12 terms took 18 msecs

javolution.util.FastSet
# 276MB
# contains test of 10000 x 12 terms took 10 msecs

javolution.util.FastTable
# 172MB
# contains test of 100 x 12 terms took 50961 msecs


gnu.trove.THashMap
# 331MB
# contains (key) test of 10000 x 12 terms took 65 msecs

gnu.trove.THashSet
# 185 MB
# contains test of 10000 x 12 terms took 47 msecs


com.google.common.collect.ImmutableSet
# 331MB
# contains test of 10000 x 12 terms took 19 msecs

com.google.common.collect.ImmutableMap
# 404MB
# contains (key) test of 10000 x 12 terms took 32 msecs


Lucene KVP index
# 94MB
# lucene 100 x 12 TermQueries took 58 msecs
# lucene 10000 x 12 TermQueries took 590 msecs
# in memory Index building a key value index with each term being a document and storing the value as a document field
# overhead of using an IndexSearcher to just the RAMDirectory is minimal
# 1573345 records loaded into Lucene KVP index in 13453 msecs

Lucene term index
# 44MB
# lucene 100 x 12 TermQueries took 32 msecs
# lucene 10000 x 12 TermQueries took 369 msecs
# in memory Index storing only the pure term index
# 1573345 records loaded into simple Lucene term index in 10222 msecs

H2 file based
# 33MB (connection object, not during querying)
# sql equals test of 10000 x 12 terms took 634 msecs
# file based db with 1 table, 1 indexed varchar(255) column
# 1573345 records loaded into H2 file database in 245025 msecs

H2 in memory
# ???MB (JProfiler requires >3gig memory)
# sql equals test of 10000 x 12 terms took 1024 msecs
# in memory db with 1 table, 1 indexed varchar(255) column
# 1573345 records loaded into H2 in memory database in 18760 msecs

Lucene came out with the least memory footprint of only 44MB, gnu.trove does best in terms of memory for classic sets, while javolution outperformes anyone else on speed. But their footprint is even slightly larger than the java.utilHashSet. Compared to KVP lucene the performance of H2 is pretty similar. Building the H2 file db took more than 10 times longer than the in memory one, so for regular updated and writes in memory offers lot more performance, but reads are a surprise. The file based version was nearly twice as fast as the in memory one! gnu.trove contains a lot of specialised classes to hold primitives as keys in sets or maps for example. In case one is using int or long this should be a much better footprint, but I haven't tested it as I am interested in Strings currently. For those interested, here is the source code that created the objects and did the time measurments (most tests are commented out in this particular revision).