MediaShelf Blog

I’ve posted my last videos to the Code4Lib & Dev8D 2010 channel on Vimeo. (NB: the Dev8D responses are all on YouTube, not Vimeo).  We only had a half day on Thursday but I managed to run around and gather enough video for one montage of hello messages and three mini-interviews.

I convinced Thursday’s emcee to let me conclude the morning announcements by showing Dev8D’s video saying hello to us and recording a quick video of the whole conference saying “Hi Dev8D. How’s the weather in London?”.  Lesson learned: when recording 200 people saying hello, keep the microphone away from your mouth.

A snowstorm was threatening to disrupt everyone’s travel plans so we spread to the winds pretty swiftly once the figurative gong rang at noon, but I managed to grab lunch at Asheville’s infamous 12 Bones Smokehouse with some friends before departing.

Code4Lib 2010 Last Day from Matt Zumwalt on Vimeo.

Mark Matienzo at Code4Lib 2010 Recommends “Ask Anything” from Matt Zumwalt on Vimeo.

Matt Cordial at Code4Lib 2010 - Lightweight Tools and Robust Testing from Matt Zumwalt on Vimeo.

A Little Blacklight Lovefest at Code4Lib 2010 from Matt Zumwalt on Vimeo.

It’s extremely gratifying to attend a conference where super smart presenters don’t shy away from showing real code.  It’s even better to be at a conference where the presenters manage to show complex code while making it understandable even to people who don’t use the related systems.  Every presenter today has managed to do that really well.

This is what happens when smart people people come together, communicate often (year round), and respect each other.

Building a Better Advanced Search

Naomi Dushay & Jessie Keck’s presentation on building a better advanced search for SearchWorks is a key example of this unabashedly technical yet clear and concise communication.  They opened with a clear presentation of the real-world situation, including usability goals, etc.  They followed this with a breakdown of the desirable features and the possible ways to implement them.  Naomi then plunged into seriously complex solr config details that she managed to make intelligible step-by-step.  Hint: she used screenshots of the code with the salient items circled.

Drupal 7: A more powerful platform for building library applications

Cary Gordon from the Cherry Hill Company showed us the new features in Drupal 7.  I expected to be bored, since I don’t use Drupal and can’t stand the sight of PHP code. Instead, I was pleasantly impressed and intrigued by the features Cary walked through.  I keep hearing that Drupal has matured substantially in recent years, but I didn’t believe it until now.

Enhancing Discoverability With Virtual Shelf Browse

Andreas Orphanides, Cory Lown, and Emily Lynema from NCSU showed us their snazzy virtual bookshelf with an “infinite shelf” allowing you to scroll through book covers five at a time ad infinitum.  One wonderful idea is the use of “faux covers”, where they generated something that looks like a real book cover whenever a real cover image couldn’t be retrieved for a search result.

As with almost any Code4Lib presentation, this would have been intriguing if they only gave a demo of the software but they didn’t stop there.  80% of the time was spent explaining how they did it, what technologies they used, how they thought about the problem, what worked well and what went wrong.  Gosh I love hackers who share.

How to Implement A Virtual Bookshelf With Solr

In their third presentation of the day (remember, presentations were chosen by open popular vote), Naomi Dushay and Jessie Keck showed us their work around implementing the much-desired support for browsing through digital collections using a virtual shelf organized by call number.  This feature sounds simple, but the implementation proved more challenging.  To start with, the librarians couldn’t come to a consensus about what constitutes “correct” ordering.  Solution: ignore the arguing librarians and go straight to the users!

The real challenge lay in sorting out 8 million call numbers from numerous libraries using a variety of call number systems (ie. LC, Dewey, SUDOC, Thesis, etc.)  Who knew that call numbers were so complex? Not I, until now.

I knew these guys were good - I’ve collaborated with them, I’ve seen their code - but wow.  I didn’t realize how remarkably good they were until this third presentation.

Lightning Talks & Breakout Sessions

The day closed with 13 Lightning Talks followed by 6 Breakout Sessions.

Lightning Talks:

• LibX Update - Godmar Back
• How to build a Virtual Bookshelf Without Solr (or MySQL) - Maccabee Levine
• VIVO, an interdisciplinary national network - Paul Albert
• WolfWalk, two ways - Jason Casden
• Custom metasearch widgets - Alex Smith
• Node.js development - Gabriel Farrell
• Catalog Auto-suggest using SOLR - Jill Sexton
• EmeraldView, a PHP frontend for Greenstone - Yitzchak Schaffer
• Faceted browse on the cheap - Tom Keays
• EAD, APIs, and Cooliris: providing access to digitized archival materials. - Tim Shearer
• Kill the Search Button - Michael Nielsen, Jørn Thøgersen [facilitated by Roy Tennant]
• You Heard It Here First… - Roy Tennant
• File Information Tool Set (FITS) - Spencer McEwen

Breakout Sessions:

• Mobile Dev
• Blacklight - Bess Sadler
• Tools4Lib - the best tools used in your job - Devon Smith/decasm
• Let’s Link Our Data - If you have data, metadata, vocabularies, or just about anything else you want to link, show up and we’ll figure out what to do with it! - Ryan Scherle/ryscher
• Homebrew - building your own instead of using Fedora/DSpace/Blacklight/whatever - Esmé Cowles/escowles
• API queries to vocabularies - Ya’aqov Ziso

Media, Blacklight, and Viewers Like You

Chris Beer from WGBH (Boston’s Public Television; producers of Nova, Julia Child, etc.) walked us through the inner workings of the new WGBH website that runs on Blacklight, Fedora, and LightHTTPD.  Without getting bogged down in the details, he touched on many of the technical and organizational challenges they faced and how they addressed them.  Some of the technologies in the mix

• Blacklight
• Fedora
• PBCore: Metadata Standard for Media based on Dublin Core
• ODRL: Open Digital Rights Language
• JQuery

Chris did some amazing things with JQuery and FlowPlayer in his customized Blacklight views.  One example: You can browse to a point in a video by clicking in the transcript. If you then play the video, the transcript automatically scrolls with it — accurately.  This was made possible by the fact that WGBH already has detailed timecodes embedded in their transcripts.

The data model for a single episode in the WGBH archive fills an entire page — raw video, edited video, music, images, post-processing, transcripts, etc. with rights and permissions varying for each part.  This highlights the grace of Blacklight, which allowed him to simply leave that system in place and merely expose the public parts with a slick, faceted search & discovery interface.

One detail that Chris called out was the fact that Fedora has trouble with large (5GB+) datastreams.  He pulled up the corresponding (unresolved) tickets in Fedora’s Jira to establish the fact that this is a long-known unresolved problem.  He then added that WGBH’s own internal (proprietary) DAM system has the same problems.  At least with open source software, addressing a bug like this is a matter of public process and community initiative rather than being subject to the whims of a single organization.

WGBH will soon be launching a number of public websites based on this Fedora+Blacklight combination.

Becoming Truly Innovative: Migrating from Millennium to Koha

Ian Walls from NYU gave a humorous account of finally ditching proprietary Integrated Library Systems, which he represented as battling kittens, in favor of the open source ILS called Koha.  He went through the details of how they handled the challenge of migrating their data from iii (the proprietary ILS) to Koha and closed by telling the room that the process he used should work for anyone with a similar proprietary install.  The NYU migration took a total of 3-4 months without any all-nighters.

All of the code he used is available at contribs.koha.org. I don’t personally work with ILS systems, so I can’t say I have a solid read on the impact of this information, but I got the distinct impression that this was very encouraging news for a many of the people in the room.  When someone from the audience asked for a show of hands “How many people would like to migrate away from iii?” more than a third of the room raised their hands.

Ask Anything! (aka. Human Search Engine)

I know the Dev8D people will dig this one.

Dan Chudnov from Library of Congress facilitated an open-ended session where anyone with a question/request/missed connection could grab a mic and announce their desire.  In short, the “If there’s anything that you want to ask everybody, now is your chance to speak up.” The core idea was to give people an opportunity to tap the collective knowledge of the room.

The process was to hear a question, identify people who want to respond, and then move on to the next question.  The full-room discussion was kept to a minimum.  Here are most of the questions asked (I missed a few).

• Switching from Perl: What language should I switch to?
• How do I make an API accessible only from specific domains
• Who is using my library (pymarc) and how are you using it?  Please tell me so I can make it better.
• How do I extract files from Internet Archive .arc format?
• Is anyone interested in helping my organization port a custom staff app from Millenium ILS to Aleph ILS?
• Is there any interest in a 1-2 day Blacklight meeting?
• Has anyone used the California Digital Library digital curation microservices?
• Please tell us: What do Librarians (non-techies) need to know about software development?
• What’s the best way to do complex log analysis? (answer: splunk)
• I want to allow people to add data to my site without maintaining user info for them?  I want to use something like OAuth but it has to work with something like curl as the client? (recommendation: RPX)
• How do I model dates & date ranges in RDF?
• Proposal for a breakout session on building homebrew digital repositories (instead of Fedora, DSpace, etc)
• Request for 10 minute tutorial on 3D graphics programming
• Anyone interested in modeling archival description in RDF? If so, join http://groups.google.com/group/semantic-archives
• Is anyone using OpenCalais module for Drupal?
• Is anyone here going to be working on the OLE project?
• I’ve created software for hiding borrower information in library systems, but how can I share the code without making it vulnerable to “the man”?
• Is anyone doing work with HTML5? Especially geodata services, etc?
• Are there any libraries out there using Plone for their website?
• Who else is developing Facebook apps for their libraries?  Any ideas how to make a useful Facebook app for a library?
• Is anyone using MARCLogic or eXist in production systems?
• Since this seems to be really working, Should we start something like a Stackoverflow for library hackers? (answer: Yes.  Awesome idea.)
• We’re working on search & discovery with non-roman scripts in a solr/lucene context.  Please provide suggestions.
• Is anyone working on natural language processing implementations for machine learning?
• We’re moving all of our production systems to cloud environments.  Has anyone else done that?
• Is there a JSON library for MARC? (answer: it’s built into evergreen, OCLC is also looking into it.)

The verdict on Ask Anything: resounding applause, many smiles.

Surprise Trivia

Alex O’Neil from University of Prince Edward Island emcee’d a very fun round of trivia before we adjourned for lunch.  It was wicked fun.

Hello to Dev8D from Code4Lib from Matt Zumwalt on Vimeo.

The first day of Code4Lib has drawn to a close.  The irc channel was flooded with raucous banter and twitter was very active as well.  In the evening, one person who had only followed the tweets commented to me that “people at this conference are very polite”.  I jokingly assured her that all of the sideways comments must have been reserved for the irc channel.

The conference is off to a great start.  Meanwhile, many people expressed intrigue at the prospect of Dev8D attendees remotely adding their noise to the mix tomorrow.

Keynote: Cathy Marshall on Personal Digital Information

The conference opened with an insightful and humorous keynote presentation by Cathy Marshall from Microsoft Research. The presentation delved into the technical, psychological, and sociological factors that impact management and retention of personal digital information.

Morning Sessions: Linked Data, Cloud Computing

After an impressive and RDF-laden linked data pep talk by Ross Singer, the morning’s presentations had a strong cloud computing bent, which swiftly gave rise to groans on the irc channel about cloud computing being over-hyped. The complaints were quickly challenged by others who pointed out that Code4Lib’s entire schedule was decided by popular vote.  ”Where were your complaints when it was up for vote?”

Personally, in addition to watching the presentations and following the rapid-fire commentary on irc, I spent time exploring the commands that zoia the irc bot can respond to.

Afternoon Sessions: It all comes back to MARC Metadata.

The bread and butter for hackers in libraries inevitably revolves around MARC metadata and Integrated Library Systems (ILS).  MARC records are notoriously troublesome and unreliable.  It’s no surprise, then, that the afternoon’s presentations were almost exclusively about technologies and techniques for controlling, cleaning up, aggregating, de-duplicating, consolidating, and editing MARC metadata.

Lightning Talks & Breakout Sessions

The day closed with 14 Lightning Talks followed by 7 Breakout Sessions.

The Lightning Talks were:

• UW Forward - Steve Meyer
• MODS4Ruby & Opinionated XML - Matt Zumwalt
• The Digital Archaeological Record - Matt Cordial
• Hydra: Blacklight + ActiveFedora + Rails - Willy Mene
• Why CouchDB? - Benjamin Young
• Data integrity (cheap, fast, and easy) - Gwen Exner
• HathiTrust Large Scale Search update - Tom Burton-West
• EAD and MARC Sitting in a Tree: D-R-U-P-A-L - anarchivist
• EZproxy Wondertool - Paul Joseph
• HathiTrust APIs - Albert Bertram
• Repository of MARC Abominations - Simon Spero and J-Rock
• Mystery Meat - Joe Atzberger
• Fuwatto Search - Masao Takaku

The Breakout Sessions were:

• Code4Lib Journal open meeting/discussion
• Cloud4lib - next steps
• xC - Extensible Catalog
• VuFind
• Solr
• CouchDB
• MODS for Ruby & Opinionated XML

The lightning talks were, as always, diverse and energizing.  The breakouts drew a fairly even spread of interest across all 7 topics, allowing everyone to dive deep into their chosen subject area with 8 to 20 people.

In order to take advantage of the fact that Code4Lib and Dev8D are happening simultaneously this year, we’re going to try to facilitate a bit of Transatlantic crosstalk. After all, how could thousands of miles, five time zones and a couple trillion gallons of water keep developers apart? Nerdy enthusiasm knows no boundaries.

With these two crowds of innovative and skillful people, there’s no way to fully anticipate how collaboration will occur. As a baseline, we will be arranging the following: daily video updates, daily live chats, regular updates on the MediaShelf blog, and links to other blogs that are reporting on the events.

Dates & Schedules

The two events will only overlap for 2 days - the 24th and 25th of February.

Code4Lib dates: 22nd-25th February
Dev8D dates: 24th-27th February

For more info, see the Code4Lib schedule and the Dev8D programme.

Hashtags & irc

There are no official twitter hashtags (nor blog/delicious tags) designated for these events. As the excitement ramps up, the tags to watch at first will be

• code4lib, c4l10, (or both)
• dev8d

If you read really fast, you could watch all three together.

We will try to keep you posted as new memes and hashtags come to the forefront.

Of course, the real place to witness code4lib chatter is in the #code4lib irc channel.

UStream Channels

We’ve created two UStream channels where we will be hosting occasional live feeds and uploading the daily video updates. Check them out and watch for changes: Code4Lib 2010 on UStream & Dev8D 2010 on UStream.
When we decide on the times for hosting the live chats and posting the daily video updates, we will announce the times here and on the UStream channels.

If you would like to contribute your own videos to these channels, create a UStream account, add the videos to your account, and tag the videos with either code4lib2010, dev8d2010, or both. Once you’ve done that, let us know your UStream username and we will add you to the list of contributors. Your videos will then show up on the channels according to which tag you used.

In early 2009, I chanced upon becoming the only person to attend both Dev8D in London and Code4Lib 2009 in Providence, Rhode Island. While at Code4Lib, I had every intention of posting a comparison of the two conferences, but eventually decided that nobody would care to hear about it.  The blog post sat unfinished for a year until the news came out that both conferences would be happening simultaneously this year.  Suddenly a side-by-side comparison seemed much more meaningful.  I dug into my notebooks and found the table you see here.

Further down the page, for the uninitiated, I’ve included an off-the-cuff description of each conference.

Code4Lib & Dev8D 2009 - Quick Comparison

Code4Lib: the big annual download for coders in libraries

Code4Lib has a large, established community based mostly in North America.  As the name implies, the community consists of “computer programmers and library technologists who largely work for and with libraries”[from Code4Lib wiki]. The Code4Lib Journal and the #code4lib irc channel are hubs of intense discourse year round.  The Code4Lib conferences are run as a single track where every presenter delivers his/her work before the entire group of conference attendees. The lineup of presenters, along with the location and dates for the conference are chosen each year by means of an open, public voting process.

The annual Code4Lib conferences are the primary place for this community of extremely smart, innovative people to exchange notes about the work they’re doing.  It’s very intense, very technical, and extremely informative.  There is very little hand-holding; everyone is expected to keep up with the flow of information or get out of the way.  For the past two years (possibly longer), registration has reached capacity within 48 hours of opening up.

Taken as a group, the Code4Lib crowd can seem intimidating.  They have a very strong meritocracy, which is one of their biggest strengths.  The only sure-fire way to get attention in this community is to make great software and to share it.  If you do that, you will be acknowledged and socially rewarded.  I heard complaints about “cliques” forming, mainly because everyone was left to fend for themselves in the evenings - people who already knew each other clumped together and swiftly disappeared.  Arriving as an outsider, this was unnerving for me at first but in the end everyone was actually very friendly and eager to trade ideas with anyone who could keep up with the intellectual pace.

My most memorable moment at Code4Lib was the Linked Data preconference hosted by Ed Summers, Dan Chudnov and Michael Giarlo from the Library of Congress.  As part of the day-long workshop, they set up one of the best hands-on lessons I’ve ever been part of.  The premise was simple: use FOAF profiles as the means for entering into a raffle.  This relatively simple exercise forced everyone in the room to, at the very least, publish a FOAF document (anywhere on the web) with a couple of required assertions included.  Those with more experience helped those with less.  Everyone gained something and one key concept nailed its mark: the linked data movement of 2009 has many, many things in common with the early world wide web, and everyone is invited along for the ride.

Code4Lib 2010 will be held in Asheville, NC February 22-26.  It has been sold out since December.

Dev8D: making developer happiness in the UK

Dev8D is the brainchild of David Flanders at JISC.  The name, Developer Happiness Days, harkens to the meme of developer happiness that became one of the core mantras for “lightweight application framework” evangelists around 2005/2006.  As I understand it, David looked around at all of the developers working on JISC-funded projects across the UK and saw a massive pool of technical talent hampered by the fact it lacked an  ongoing community for exchange of ideas, skills, criticism and praise.  He convinced JISC to fund a small conference specifically for software developers, and specifically aimed at fostering a developer community.  They made him promise to provide proof of “innovation” happening and — tada — Dev8D was born.

I’m tempted to describe Dev8D as zany.  David threw in as many activities, challenges, rewards, and discussion topics as he could possibly muster.  His goal was simple — get people talking, encourage them to dream up new technical possibilities, reward them for sharing those ideas, reward them even more for executing those ideas, and have fun in the process.

It would have been hard for an enthusiastic software developer not to have fun at Dev8D.  We were all basically along for the ride.  Every time you turned a corner, someone had cooked up a new nifty little web app or come up with some novel way to use an existing service to augment our Dev8D free-for-all.  David used wordle to generate tag cloud “name tags” from everyone’s blogs.  By lunch on the first day, someone started projecting a twitter aggregation of #dev8d on the wall.  By tea time, we had two or three custom apps floating around that did wacky stuff based on codes in your tweets.  (In February 2010, this all may sound a bit tired, but remember that a mere year ago all things twitter were brand new to nearly everyone.)  The name of the game was “dream it up and do it”, and we certainly did that.

My most memorable moment at Dev8D was The Dragon’s Den.  This was David’s solution to JISC’s requirement that he provide proof of innovation as outputs from the event.   JISC set up a Developers Challenge, providing real-world users from UK academia and soliciting the Dev8D attendees to submit their best ideas for new software to solve those users’ technological needs.  The best team would win a wad of cash in exchange for documenting their idea.  The way you submitted your idea was by presenting it in the Dragon’s Den, where a panel of judges from JISC and local industry would hear your idea, ask questions, and provide feedback.  Though my submission didn’t win the challenge, I learned a lot from those judges and would participate again in a heartbeat simply for the feedback.

Dev8D 2010 will be held in London 24-27 February.  They may still be accepting new registrations, though all of the free accommodations have been slurped up.

Crosstalk in 2010

Later this month, Code4Lib 2010 and Dev8D 2010 will have three days of overlap.  I will be in Asheville, NC for Code4Lib and Eddie Shin will be in London for Dev8D, so MediaShelf will be plugged in on both sides of the pond.  We have promised to encourage crosstalk between the events, though we haven’t sorted out the details.  When we settle on a plan, we will post an update on this blog.

Leading up to this year’s Open Repositories, it became clear that there was demand for a BOF (Birds of Feather) session focused on agile languages and Fedora.  I pitched the idea in an email to a couple colleagues beforehand and then announced the BOF at my presentation on Monday morning.  Rather than constricting it to Fedora projects, I billed it as Agile Languages and Repositories.  About 30 people showed up.  The split was pretty even between Ruby, Python, and PHP developers.  About a third seemed to be Java developers in the process of defecting.  In addition to people doing stuff with Fedora, there were a handful of DSpace developers and possibly a couple who maintain ePrints repositories.  

For the first half of the BOF we sat in mixed groups, eating our lunches and each talking about the work we do.  We then split up by language (Ruby, Python, PHP) and discussed language-specific topics.  For that second half I sat at the Ruby table where we talked about ActiveFedora, JRuby, RDF support for Ruby, MODS support for Ruby, Solr (solr-ruby and RSolr), and how Blacklight fits into the mix. 

I closed the conversation by asking if we should set up email lists for collaboration.  It seemed reasonable to set up a general mailing list for the solutions community as well as a list specifically for people doing stuff with Ruby, Fedora repositories, and (most likely) ActiveFedora.  I also resolved to encourage the creation of Python-oriented and PHP-oriented equivalents.  For now I have created two lists on Google Groups.  The first one, Fedora Commons Create, is for general discourse about creating client applications for Fedora.  The second, ActiveFedora / Ruby + Fedora Commons, is for Ruby-specific collaboration.

In the end, I was really pleased to realize that for the first time we had a substantial group of people interested in each of the main interpreted languages (Ruby, Python, PHP) and each group had at least one open source Fedora-based project to use as a starting point for their conversations.  The Ruby group had ActiveFedora, the Python group had Ben O’Steen’s work and Peter Herndon’s Django integration, and the PHP/Drupal people had Islandora & Fez to start from. 

This was a comfortable step forward from the scenario as it was a year ago.

Fedora Commons Create

Visit this group

ActiveFedora / Ruby + Fedora Commons

Visit this group

The Red Island Repository Institute (RIRI), hosted by the University of Prince Edward Island (UPEI) has started with a bang.  Sandy Payette spent an entire day feeding the room with a wonderful mix of vision, software architecture, social context, and technical details.

Mark Leggott has put together a great event. There are people here from all over North America, and even one visitor from Australia.  Everyone has been enjoying the beautiful environs of Prince Edward Island and the quality of information being exchanged is top notch.  I particularly like the fact that Mark is “drinking his own kool-aid” by setting up a Drupal/Fedora site for the institute.

This should be a great week.

Picking up from the ideas in The Missing Sync for Fedora Commons, I’ve been talking with Thorny and Sandy at Fedora Commons about creating a Fedora Solutions Integration Council.  We haven’t quite figured out the structure of it, but the ideas are coming together pretty quickly. Bottom line, the council’s responsibility is to help everyone make informed decisions and support each other’s work.  

 As a first stab, I’m putting effort into three things:  

1. bring together the streams of communication (ie. blogs, irc, etc) 
2. help projects find and connect with others who are doing similar work
3. identify the major themes: problem areas, innovations, exciting solutions, etc.

Ultimately, I hope this will allow us to shed light on the various avenues of exploration in Fedora-centric application development.  So many people are doing such interesting and exciting work.  It’s time for us to talk more openly and enthusiastically about it.

The other Fedora Solutions Councils are organized around themes like eScience, Museums, and Education.   In contrast, the Integration Council is aimed at addressing the cross-cutting concerns of application development.  We all have to deal with things like access controls, scalability, and workflow.  The best solutions to these types of challenges are often applicable in many contexts, regardless of whether you are an eScience project or a small humanities archive.  Our aim is to get as much information flowing between developers as possible.  I want to let developers decide for themselves which ideas apply to their work.

Watch this space.