Blacklight 3.2

As of releasing Blacklight 3.1, they won’t be supporting Rails 3.0. We all agreed that Rails 3.1 provides a very compelling set of improvements. Anyone who is on Rails 3.0 should be eager to upgrade to 3.1. In order to take advantage of those improvements, hydra-head should switch to assuming a Rails 3.1 approach by default while supporting Rails 3.0 users until the release of hydra-head 4.0.

PBCore

There are a couple of spots in PBCore that are difficult to support in OM. Rather than fixing OM, does it make sense to switch to using RDF to express PBCore information? Already, everyone who’s using PBCore with Hydra is breaking up PBCore documents and instances into a graph of objects related by (RELS-EXT) RDF, so actual PBCore XML documents can only be generated by assembling information from an RDF graph. Since that’s the case already, why not fully embrace a linked data approach by expressing the PBCore information as RDF all the way through?

A linked-data approach to PBCore “3.0″ would entail:

• Continue to use the useful parts of the PBCore data model
• Continue to use PBCore controlled vocabularies
• Provide PBCore 2.0 XML serialization from PBCore 3.0 RDF
• Use existing RDF ontologies where possible (ie. EXIF, Dublin Core, etc.) instead of using one-off PBCore predicates for things like “identifier”, “description” and “frame rate”

In essence, this means mixing DC RDF with a minimal set of preferred ontologies and vocabularies that have mostly been identified already by the PBCore maintainers.  The conceptual model of PBCore is useful and the recommended vocabularies are very useful while the XML serialization is just another XML schema to support.

OM

OM is the least mature of the Hydra components. It has some rough edges and the syntax for declaring Terminologies leaves many things to be desired. We want it to improve but don’t have a clear idea of what a better, smarter OM would look like.

At the DLF Fall Forum, Naomi pointed out that OM tries to do 3 distinct things:

1. Let you build a DSL for accessing values from XML, (XPath registry + DSL)
2. Provide instructions for indexing XML content into solr (DSL-driven solr indexer)
3. Construct new XML nodes & ancestors based on DSL (Template Registry with DSL)

If we managed to do all of these things together gracefully, that would be spectacular. Right now, OM isn’t so graceful.

If we dropped the ability to construct XML nodes and their ancestors, we could make the syntax for defining OM terminologies much more straightforward. Is that the solution?

In the absence of a vision of what the better, faster, stronger OM would look like, we formed a loose resolve around focusing on a couple useful functional additions while addressing a couple of long-standing annoyances.

Focus on

• Make it easy to declare xpath-only terms. This simplifies the most common path of adoption (“I just want to read and update XML node values”) It’s ok if these raise an error when you try to use them to insert nodes and/or ancestors.
• Support mixing together terminologies so that you can do things like reuse the “Base” PBCore Terminology and then mix your own custom Terminology into it.
• This requires that you be able to override/add indexing instructions on Terms without editing the Terminology file directly

Standing itches to address

• Switch solrizer indexing strategy to only index terms with indexing instructions
• HYDRA-358: OM should allow you to create Terminologies with nil namespaces (Fixed this during the call!)

In short, we treat the CatalogController as a SolrSearchesController and only use Blacklight helpers, etc. in that context. Meanwhile, we implement model-specific Controllers (ie. ModsArticlesController ) using regular Rails style, and pass ActiveModel objects (ie. @article) into their views.

Conceptual clarification:

In Blacklight, the CatalogController loads a SolrResult (@result) and a list of Solr Documents (@document_list) and passes those into the views. All views are written to use the @document_list and most of the view partials expect a solr document (document). This constrains you to inspecting solr documents (hashes) for values that they may or may not have set.

In conventional Rails apps, your controller loads AcitveModel objects (ie. Article objects loaded in variables named @article or @articles) and passes those into the views. Those objects have all of the behaviors of the model, allowing you to call model-specific methods like @article.title or @article.files.each { |file| puts file.file_size }

Recommendation: Only use (Blacklight style) views when rendering a Search interface with a “search” bar. In contrast, when rendering a simple list of objects (index), use Rails style ActiveModel objects.

Examples:

Model-specific views expect ActiveModel objects

views/recordings/show.html.erb : expects a Recording object called @recording
views/recordings/edit.html.erb : expects a Recording object called @recording
views/recordings/index.html.erb : expects an array of Recording objects in @recordings
views/recordings/_index.html.erb : expects a Recording object passed in as a local called ‘recording’

Search Result views (including model-specific search result view partials) expect Solr document lists and/or Solr documents and live within views/catalog

views/catalog/_index_partials/_recording.html.erb : expects a solr document (a hit from a solr result)

Further Recommendation: model-specific search result view partials live in model-specific view directories within views/catalog rather than action-specific view directories. This is a replacement for the existing implementation of the render_document_partial view helper.

ex. views/catalog/recordings/_index.html.erb

instead of: views/catalog/_index_partials/_recording.html.erb

Recommendation: When listing children of an object, put the corresponding partials in the views directory of the parent.

Example: When rendering a list of the lectures associated with a Course, you would define these views:

views/courses/_lectures.html.erb:

views/courses/_lecture.html.erb

The _lectures.html.erb partial expects a local called ‘course’, sets up the list/table HTML and passes the collection of course.lectures into the ‘courses/lecture’ partial.

A quick search turned up two useful posts on the topic:

Bedazzle your bash prompt with git info
Current Git Branch in Bash Prompt

I also ended up digging around in the documentation about color hashes in bash.  Now that’s some seriously ugly configuration code.

I set up my prompt to look like this:

~/Develop/projects/hydra-head (rails3) :

Here’s how I did it:

All I had to do was put these lines in ~/.profile

RED="[33[0;31m]"
YELLOW="[33[0;33m]"
GREEN="[33[0;32m]"
BLUE="[33[01;34m]"
GREY="[33[37m]"
BLACK="[33[38m]"
CYAN='[33[0;36m]'
PURPLE='[33[0;35m]'

export PS1="$BLACKw $CYAN`ruby -e "print (%x{git branch 2> /dev/null}.grep(/^*/).first || '').gsub(/^* (.+)$/, '(1) ')"`$GREY:[33[00m] "

I’ve included the color-hash variables for everyone’s sanity.

Caravan crossing the Silk Road. (detail of the map of Asia) The Catalan Atlas, Spain, Majorca 14th century.*

According to Al Gore, there’s an old African proverb “If you want to go fast, go alone.  If you want to go far, go together.”  This idea caught our attention early in the Hydra collaboration and has become the driving principle for how/why we work as a community.  I think that Tom Cramer of Stanford gets credit for being the person who recited this principle the most over the past few years,  but now we all find ourselves saying it.  At first, it just sounded like a nice ideal that we could all nod our heads to but over time the truth of the adage has rung true.

Digital repositories address a distinctive set of use cases ranging from end-user interactions to technical preservation scenarios.  As a result, developing solutions for digital repositories requires you to grapple with many complex technical challenges.  Very few people in the world have deep familiarity with those issues. The Hydra community has already managed to attract an impressive number of those people.  Among them are some of the top Fedora experts and some of the most experienced creators of next generation search interfaces.  It’s an amazing advantage to have all of these deeply knowledgeable people contributing to the same set of designs, code, and solutions.

I’m pretty sure that no single institution could muster a team of this size and caliber to work on digital repository solutions.

Off the top of my head, here’s a list of areas that we have addressed in ways that explicitly drew on knowledge, skills and resources spread across multiple institutions:

• Data Modeling/Content Modeling
• Repository Security & Gated Discovery
• Search Interfaces (designing & implementing)
• Search Indexes (building & managing)
• Metadata Schemas (choosing & using)
• Workflow & Reusable Services
• Accessibility & Usability
• Testing & Documenting Code
• Deployment & Systems Administration

In addition to addressing these technical areas, the Hydra community has become an intense testbed of best practices for developers and managers.  Even the work of building and sustaining the open source project itself has been achieved through open, iterative, collaborative process.

This only scratches the surface of the benefits that I personally see in Hydra’s collaborative model.  Taking a step back, I’d say that working together as a community has allowed us to achieve a potent balance of pragmatism and long-term vision.  The next two Hydra principles address these things directly.

Next Week: Making a solution that works for us.

*Image caption from CUNY course materials

Antique depiction of a hydra.

One Body, Many Heads references our project’s namesake, Hydra (greek Ὕδρα), the fearsome water beast with more heads than an ancient greek painter could fit on a vase.  Anyone who’s watched Percy Jackson and the Lighting Thief knows that it’s really hard to hill a Hydra because every time you cut off one of its heads, two more spring up in its place.

We want to build digital repository solutions that are like the Hydra. We want to support many users interacting with diverse content in diverse ways while content accumulates in one repository. By creating many Hydra Heads on top of a repository, each specifically addressing a particular type of user interacting with a certain set of content, we make our repository and its content more durable. This is how our solutions live up to the name Hydra: One Repository exposed through many Hydra Heads.

In 2009/2010, we created three Hydra Heads as a starting point:

• Hydrangea: a reference implementation focused on faculty self-submission of open access articles and datasets
• SALT: am EAD-aware toolkit for Luminaries to browse through & annotate digitized versions of content they’ve submitted to university archives
• Stanford’s ETD Head: a solution for submission, review, and accessioning of students’ Electronic Theses and Dissertations (ETDs)

Based on our experiences building these three heads, we published a beta version of the Hydra Framework in July 2010. Since then, teams of developers have begun using the Hydra Framework to build heads suited to the users and content they need to support.

The 2011 generation of heads is slated to include solutions for:

• Archival arrangement & description of Born-digital materials
• Creation & dissemination of Image Collections
• Creation & dissemination of Audio & Video collections
• Management of Content Ingest Workflows
• Dissemination of audio/video recordings from courses & lectures
• Laboratory Dataset Submission & Management

For an up to date list of know Hydra heads, see the Known Implementations page on the Hydra wiki.

Work on these solutions is being carried out by a distributed community of developers spread across multiple continents. Where there’s a need, people are invited to pick up the code and start creating their own solutions. The Hydra Framework and the Hydra Community are designed to support this distributed approach to innovation.

One Hydra Framework … Many Heads … Many Teams … Many Types of Content … Many Users

The Hydra Framework is the base of code that makes it possible for all of these Hydra Heads to crop up in response to specific needs. It’s the basic DNA for repository solutions that combine the patterns of rapid application development with the sensibilities of ensuring re-use of both code and content.

Much of our effort over the past three years has focused on designing the Hydra Framework to support distributed development while also encouraging teams to share innovations and patterns across Hydra Heads. What we’ve learned is that good code is not enough to achieve this. We needed to encourage an entire culture of identifying, exchanging and applying best practices throughout the design, development, deployment, maintenance and usage of repository solutions. The principle of One Body, Many Heads is equally applicable here: A single community with a shared goal has spawned many teams with their own eyes, ears, and fangs primed to attack the repository needs before them.

Next week: If you want to go Fast, go alone. If you want to go Far, go together.

As a community, we are constantly refining our understanding of the work that we do. Our community principles, just like our collaborative process and development techniques, are a work in progress — you might say that they’re in permanent Beta — with many people drawing on their experiences to achieve the best possible result.  As it stands right now, we have settled on four principles:

• One Body, Many Heads
• If you want to go Fast, go alone. If you want to go Far, go together
• Making a solution that works for us
• Sustainability/Longevity: Eyes on the Horizon, Feet on the Ground

Soon we will be launching projecthydra.org with the final version of these principles. In the meantime, I’m going to publish a series of weekly posts presenting each principle as I currently understand it.

Next week: One Body, Many Heads.

Using Little Rails Apps to Solve a Hefty Problem

An overview of the Vision and Implementation of the Hydra Framework — from HydraCamp 2010 (slides, Octover 2010)

Using Little Rails Apps to Solve a Hefty Problem

An overview of the Vision and Implementation of the Hydra Framework — from HydraCamp 2010 (slides, Octover 2010)

RUM Sept 2010 – MediaShelf & Hydra Framework

Designing, Building, and Releasing the HydraFramework: laying a foundation for the deep web and the golden age of digital libraries

Matt’s alternative presentation on Hydra for Ruby Users of Minnesota September 2010.  (slides, Sept 2010)

Solrizer – Interplay between Searching Managing and Indexing

Matt’s Open Repositories 2010 presentation on Solrizer (slides, July 2010)

Open Repositories 2010
Madrid, Spain
July 2010

Hydra – One body Many heads

An introduction to the Hydra Framework and MediaShelf (slides, Summer 2010)