To my eyes, this seems like a wonderful new meaning of “long tail”.  It’s something that we’ve seen pending on the horizon but it’s now finally beginning to manifest.  Our civilization documents itself so thoroughly that we can grab a detailed background on nearly any topic.  This has been true in a cursory way for a while, manifesting in sites like wikipedia, but the internet is quickly reaching an information saturation point and architectural maturity that allows us to view the entire web as a living, self-documenting wiki.

Up until now, the long tail has mainly referred to an economic construct: by reducing barriers to entry into markets, by radically reducing distribution costs, and by increasing the opportunities for direct engagement between producer and consumer, the internet has made it profitable (or at least financially tenable) to cater to the countless minority niche interests in any given market.

I see a different kind of long tail coming to prevalence now.  Where the economic long tail is far reaching, the long tail of information runs deep.  As with the NPR podcast, we can look back in time and find a wealth of source material.  Armed with 20/20 hindsight, we can view and review the many ways that our civilization has chosen to express itself.  Published materials no longer die a day or a week after their creation; instead they stay alive for us to find them, or find new meaning in them, in the future.  Even better, we have begun to resurrect the materials that might have been presumed dead, destined to spend eternity on a dark dusty shelf.

When I look at that NPR podcast, I see context.  I see one thread in a complex history that I get to explore and rearrange at my own leisure.  Each of us sees this ocean of information differently, and each time we dip our hands into its depths we return with our own fresh version of the story, woven from the many disparate threads (and the gems upon them) that lie beneath the surface.

Here are sample Model declarations for Audio Records and Oral Histories that we are using in a current project.  Keep in mind that this is just a teaser.  The syntax is likely to change over the next few months.

require 'active-fedora'

class AudioRecord

include ActiveFedora::Model

relationship "parents", :is_part_of, [nil, :oral_history]
# Also considering...
#has n, :parents, {:predicate => :is_part_of, :likely_types => [nil, :oral_history]}
# OR
# is_part_of [:oral_history]

property "date_recorded",   :date
property "file_name", :string
property "duration",  :string
property "notes", :text

datastream "compressed", ["audio/mpeg"], :multiple => true
datastream "uncompressed", ["audio/wav", "audio/aiff"], :multiple => true

end

Note that we are making it possible to inject custom methods into a class that search against RDF predicates.  This way, thanks to line 9 below, calling oral_history.parts will return everything pointing at the oral history object with info:fedora/isPartOf.  We are also thinking of supporting constraint paramaters like oral_history.parts(:type => AudioRecord), which would only return the parts that are of type AudioRecord.

require 'active-fedora'

class OralHistory 

    # Imitating DataMapper ...

    include ActiveFedora::Model

    relationship "parts", :is_part_of, [:audio_record], :inbound => true

    # These are all the properties that don't quite fit into Qualified DC

    # Put them on the object itself (in the properties datastream) for now.

    property "alt_title", :string

    property "narrator",  :string

    property "interviewer", :integer

    property "transcript_editor", :text

    property "bio", :string

    property "notes", :text

    property "hard_copy_availability", :text

    property "hard_copy_location", :text

 

    has_metadata "dublin_core", :type => ActiveFedora::MetadataDatastream::QualifiedDublinCore do |m|

      # Default :multiple => true, :refinements => :none

      #

      # on retrieval, these will be pluralized and returned as arrays

      # ie. subject_entries = my_oral_history.dublin_core.subjects

      #

      # aiming to use method_missing to support calling methods like

      # my_oral_history.subjects  OR   my_oral_history.titles  OR EVEN my_oral_history.title whenever possible

      m.field "identifier", :string, :refinements => ["info:fedora", "info:doi"]

      m.field "title", :text, {:multiple => false, :required => true}

      m.field "subject", :text, :refinements => ["dcterms:LCSH", :none]

      m.field "date", :date

      m.field "language", :text

      m.field "location", :text

      m.field "coverage", :text, :refinements => ["dcterms:TGN"]

      m.field "temporal", :text, :refinements => ["dcterms:Period"]

      m.field "abstract", :text

      m.field "rights", :text

      m.field "type", :text

      m.field "SizeOrDuration", :text

      m.field "format", :text

      m.field "medium", :text

    end

    has_metadata "significant_passages" do |m|

      m.field "significant_passage", :text

    end

    has_metadata "sensitive_passages" do |m|

      m.field "sensitive_passage", :text

    end

end

In the midst of his presentation about the RepoMMan project, Richard posed an interesting pair of questions regarding the prospect of giving users a private “My Repository” space for managing their stuff.  He asked us:

1. What might a user want to get from “My Repository”?
2. What might a user want to put into “My Repository”?

He allowed the room to ponder these questions for a while.  I must admit that I was left doubting my knee-jerk responses and in turn thinking a bit further about what users really want from systems like this.  Richard then reported that a survey of his users at University of Hull provided a resounding response.  His users wanted:  Storage (safe, backed up), Access (easy and from anywhere), Management (full version control), and Preservation (to know stuff is there when they want it, short and long term).  I found this to be much more straight forward than the responses I expected.

Richard then gave us a tour of the RepoMMan interface.  Some key characteristics of the systems are the fact that the web interface, which is implemented in Flex, mimics an FTP client (to provide familiarity) and the metadata editor uses Data Fountains to pre-populate objects with automatically generated metadata so that users can then review and revise existing metadata rather than starting from a blank form.

The presentation will continue this afternoon.  By the end of the week, Richard’s full slide deck for the presentation will be up in the RIRI repository.

Mark Leggott has put together a great event. There are people here from all over North America, and even one visitor from Australia.  Everyone has been enjoying the beautiful environs of Prince Edward Island and the quality of information being exchanged is top notch.  I particularly like the fact that Mark is “drinking his own kool-aid” by setting up a Drupal/Fedora site for the institute.

This should be a great week.

This morning I added The Register to my RSS subscriptions.  I’m a bit intimidated by the volume of content that the feed puts out, but the info is just so darn tasty.

 As a first stab, I’m putting effort into three things:  

1. bring together the streams of communication (ie. blogs, irc, etc) 
2. help projects find and connect with others who are doing similar work
3. identify the major themes: problem areas, innovations, exciting solutions, etc.

Ultimately, I hope this will allow us to shed light on the various avenues of exploration in Fedora-centric application development.  So many people are doing such interesting and exciting work.  It’s time for us to talk more openly and enthusiastically about it.

The other Fedora Solutions Councils are organized around themes like eScience, Museums, and Education.   In contrast, the Integration Council is aimed at addressing the cross-cutting concerns of application development.  We all have to deal with things like access controls, scalability, and workflow.  The best solutions to these types of challenges are often applicable in many contexts, regardless of whether you are an eScience project or a small humanities archive.  Our aim is to get as much information flowing between developers as possible.  I want to let developers decide for themselves which ideas apply to their work.

Watch this space. 

Since then, the topic of synchronization has come up repeatedly in my life.

On my laptop, I’m finally looking into synching Apple iCal with Google Calendar.

At home, I’ve started using Dopplr to figure out travel plans with my family and friends.

In my work life, I’ve started recognizing the fact that I actually play a sync role in the Fedora Commons community. I’m passionate about helping people use Fedora, so I’m constantly asking developers “How did you do that?” or “What went wrong? How did you fix it?”. This has naturally lead me to conversations where I find myself saying “Oh! You should talk to XXX project about the work that they’re doing. It’s right down your alley.” or “I think that someone has already solved that problem. Let’s ping the fedora-users list before we reinvent a wheel.”

I like this new theme. It fits with the way I want to operate in the world.

Fedora Commons is a community-driven project. The team in Ithaca has taken great strides to stabilize and facilitate community process. [In fact, the footwork and brainwork that Sandy Payette has done behind the scenes this year is facinating, but that's a topic for another post.] They now have a Chief Architect (Daniel Davis), a Director of Communications (Carol Minton Morris), and a Director of Community Strategy (Thornton Staples). When these three talented people joined Fedora Commons, I thought “Phew! Problem solved.” What I didn’t realize was that there is still a missing link.

I’ve learned that there is only so much that a centralized organization can do to synchronize community efforts. Ultimately, you still need people who slosh around in the morass of innovations, workarounds and hacks in order to find those gems of best practices and well designed solutions. More importantly, you need those people to put momentum behind the good ideas and ensure that they filter back into the common pool.

Until this month, I had not realized how important this is to community-driven open source software development. There are tons of projects out there who are more than happy to collaborate, to share ideas and solutions, and even to contribute code. However, one thing is consistently true about these projects: their hands are full. They rarely have time to look over each others’ shoulders and trade notes, let alone figuring out how to share their code.

There are, of course, notable exceptions to this rule. For example, Gert Pedersen has done an admirable job of maintaining GSearch and making it generally useful for everyone. Every time a new use case or problem crops up, he usually has a solution on SourceForge within a few weeks.

What about all of the other work that people are doing?

As of late, projects have started inviting me to play an advisory role in their Fedora work, to be their missing sync tool. I’m really excited about this because ultimately it means that I have an opportunity to help more people play to their strengths. I hope that by playing this role, I can help ensure that more great solutions find their way directly into Fedora itself while other solutions join the constellation of tools, services, and documentation that populate the Fedora Commons galaxy.

As we accumulate and preserve massive volumes of rich, complex information in content repositories, we have to find new ways to represent and interact with that content.  Our culture has barely begun to crack the surface of the possibilities here.  We are engaging in a new and exciting field of inquiry, one which focuses on putting more information in the hands of end-users in increasingly powerful ways.

The Fedora commons team is working hard to set up stable channels of information to help people answer questions like this.  In the meantime, here is my quick rundown.  This is not meant to be a definitive reference.  I’m just rattling off the most salient points from the top of my head in the hope that someone will find it useful.

Java Solutions

Fedora’s APIs are entirely webservice based, so Java is not actually a necessity.  Nonetheless many projects choose to implement their client applications in Java since they already have a Java stack in play on their servers.  

The Fedora Client (distro)

Fedora itself is distributed with a Java Swing GUI client called fedora-admin.  This is a good place to get raw components for a client app.  If you download the source code for the fedora distribution, you can re-use the SOAP stubs from the fedora-admin client to connect your applications to Fedora.  This pretty much gives you the same code that you would get by generating Java code from Fedora’s WSDL, but I hear that it has been tweaked by the Fedora dev team to work more smoothly.

Muradora

Muradora is a Java application created by the DRAMA team in Australia.  Muradora started out as a proof of concept for the DRAMA Authentication and Authorization middleware.  Since then, it has become a prominent end-user client for Fedora.  Muradora does a good job of using Fedora’s existing features as much as possible.  It creates really clean Fedora objects, and will also recognize new Fedora objects automatically as long as you use some very simple, re-usable RELS-EXT relationships to arrange your objects into collections.

If you want to create your own client application, Muradora is a good place to look for samples of best practices for creating and using Fedora objects. 

Struts & Spring

As the Fedora Wiki reflects, there are a number of projects using Struts and Spring to create Fedora client applications.  Muradora is one of those projects.

Grails, Wicket, etc.

I’ve been hearing a lot about the new RAD-inspired Java frameworks like Grails and Apache Wicket.  At the moment, I’m not aware of any Fedora projects that are using these frameworks.  I’m sure that will change soon. 

PHP Solutions

Fez

Fez is a prominent PHP-driven frontend solution for Fedora.  It is maintained by developers at the University of Queensland in Australia.  I haven’t looked at the Fez code in over a year, but that might be a good place to start if you want to create your own PHP client for Fedora. 

Drupal

Numerous projects are currently implementing Drupal modules for Fedora.  I’ve heard about most of them informally, so I can’t list the organizations, but I can tell you that the University of Prince Edward Island is one of them.  UPEI is hosting the Red Island Repository Institute this August.  I anticipate that there will be quite a lot of Drupal-centric skill sharing going on there. 

Python Solutions

Ben O’Steen’s work at Oxford

Ben is a rockstar.  I’ve already praised him in prior posts and, honestly, his work speaks for itself.  If you intend to do any work with Python and Fedora, definitely start by looking at his code.  I also recommend planning for the fact that you will almost definitely want to pull his innovations into your code on an ongoing basis. The best place to get up to date information about Ben’s work is on his blog Less Talk, More Code. 

Plone

I have heard about at least one substantial project doing work with Plone and Fedora.  They are currently in early stages of development. 

Django

I’m surprised that I still haven’t heard about anyone using Django with Fedora.  This nifty framework by Adrian Holovaty hails from the world of online publishing.  It was originally developed in-house by the Washington Post.  Django seems like a really great framework for doing rapid application development.  Due to its history in publishing, it seems like a perfect fit for quite a few Fedora use cases.  

Though I still have not heard about any active projects using Django with Fedora, I have inspired some local developers here in Minneapolis to push for it in their own organization.  Watch this space.

Ruby Solutions

RubyFedora

I’m pleased to let you know that the RubyFedora gem is available for download from RubyForge.  This library allows any Ruby application or script to use the full REST API of Fedora 3.0.  We here at MediaShelf wrote both Fedora’s REST API and the RubyFedora gem that consumes it, so the two work together really well.

ActiveFedora

ActiveFedora is a work in progress.  When it’s ready, we will post it on RubyForge alongside RubyFedora.  The intention of ActiveFedora is to allow Ruby developers to interact with Fedora repositories in the same ways that they currently interact with relational databases.  We want developers to think of Fedora as “just another component” in their user-driven applications.  We are achieving this by imitating Ruby on Rail’s ORM and database management features. 

The ActiveFedora gem will lay the foundation for true, sustainable, rapid development of Fedora client applications.    It will also provide a structure for figuring out and re-using best practices around Fedora’s Content Model Architecture.

acts_as_fedora

acts_as_fedora is a Rails plugin that ties Fedora into an existing Rails application.  Rather than replacing the database connection completely, it adds hooks into Fedora within your database objects’ lifecycle.  I’m not sure of the status of this gem.  It may not be open source, and may not be actively supported.  If anyone has more information, let me know.

ColdFusion Solution

At the recent JA-SIG conference, I learned about a project at Cornell University that is using ColdFusion to create a Fedora client application.   For more info, check out the page on the JA-SIG website.