MoJo posts

How to turn the big idea of yesterday into the focused project of today

Julien — Mon, 18 Jul 2011 10:54:14 +0200

A few years ago, I had a big idea for a new art project involving video interviews.

Ideas can stay ideas for years, and then, they become real project in a few days. I should know, I built events that do exactly that :-)

This summer, I’m participating in the Knight Mozilla Learning Lab. My goal has grown: end the video interviews data obscurity – by building tools to expose (meta)data from video interviews, allowing automated analysis and data-based remix.

You can do it for a series of tweets, for recipes, for Creative Commons licences. But not for a video interview.

I knew I had to iterate, identify the real goal, and build a community. So what’s new? I have a unique opportunity to act, and there is great people that tell great stories giving flesh and blood to those principles.

First I tried to list several problems I though I wanted to solve:

Find a semantics to use for facial expressions, hand gestures, non-verbal signals
Decide how data should be linked to specific area on the video
Balance the level of integration of the toolkit. Is this a popcorn.js plugin? A javascript library? A w3c standard?
Design a front tool that would use the data to parse, create links, arrange, cluster and compare the video interviews in ways not possible before.
Integrate series of (maybe long) video interviews as a semantic data sources in apps like Storify or Flipboard.

Once the laundry list of problems is written, it is not helping much.

So I asked a twitter question to both Hadrien Gardeur and Dan Mills: what comes first in a new standard.

Hadrien is cofounder at Feedbooks and was instrumental in building the OPDS standard. Dan is building the new BrowserID standard and tools at Mozilla.

It depends how it all started. For many standards (OAuth or OPDS for example), the use case is already clearly defined

You just have different people solving the same problem differently. In this case, sketching means criticism of existing stuff

The best prototype is to have a popular service adopt the technology (Twitter adopted OAuth early, Fedbooks adopted OPDS)

Hadrien July 12, 2011

We’ve been developing use-cases and experimenting for a while, actually. I think we’re on to something this time!

thunder July 14, 2011

In quest of a nice use case beyond Le Sac à Main, I found that the powerful project It Gets Better could greatly benefit of exposing its video data.

Their website is a collection of (auto)-interviews, but there is no way for a 13 years old to link to stories about bullying at 13, in Arizona, by a shy person – just like him. «What about people like me» is an example of putting things in context.

PS: Focusing on existing use cases is key, but sketching pie in the sky interface like this one :

is fun :-)

Why HTML5 alone can't unify videos and the web

Julien — Tue, 26 Jul 2011 04:58:17 +0200

If you were to draw a map of videos on the web, you would find that your map is composed mostly of dead-ends.

HTML 5 started a big change for video, making in it a «first class citizen of the web». We can manipulate videos via Javascript and CSS. Media files are exposed as web resources, referenced natively via URLs.

But there is other dead-ends for web videos. Data dead-ends. Findability dead-ends.

When you hit a video on the web – or an audio media, for that matter – its subtitles are segregated in file silos, meta-data is not exposed.

Even licenses are rarely linked from the video – mostly when they are from the Creative Commons family, a funny paradox in a copyright obsessed world.

For example, this kind of geo-block :

on a video is not exposed in the HTML in any data format at all. It should.

Even the useful interactive transcripts, adopted by innovative video platforms like TED and YouTube, turn into dead-ends, using incompatible tools and formats.

They mostly use Flash players, at the moment, but it’s not really the point: there is working interactive transcript in pure HTML 5 and Javascript, thanks to the Popcorn.js library and clever hackers like Henrik and Mark:

Even using HTML 5, the transcripts are still mostly stored like subtitles files, as external, non hypertext, static files. They only become useful thanks to the code that use them.

They represent another dead-end on the web, a data document out of band, non-browser friendly and that links nowhere without the proper code.

For documents, the web is the API. We need to bring media meta-data back in the HTML document. And that meta-data, transcripts, license, location, people, emotions or objects must link to the resource concerned, using URIs, in particular Media Fragments:

With exemplar use cases like itgetsbetter.org it’s easier to evaluate existing tools and standards. We want to turn a video-rich but data-poor project into a data-rich resource for journalists, activists, artists and scientists.

With that design goal in mind, we ask ourselves, «should we use this technology?». Do we feel enthusiasm, do we find the technology simple, open and rich enough to help us reach our goal? If not, it’s probably not the right technology.

This week I started to talk with the Web Made Movies / Popcorn.js community. It’s a small but dedicated open source community building a great javascript video and audio manipulation library, and I wanted to know what they were using today, and what they could use tomorrow to store and use media meta-data in their projects.

In the spirit of working in the open, here are the result of a week of reviewing the state of web video (meta) data. It’s NOT a blog post, but a work document that sketches a quick panorama of where we are in term of turning web videos into citizens of the ‘web of data’ (hint: not very far)

I know have a much better understanding of the technical needs to reach my goal:

a simple, common-sense standard that store video and audio timed meta-data alongside the media itself in HTML,
a proof-of-concept tool to create and edit this timed data,
and a separate Popcorn.js plugin would be the end user player first brick

For this to work, I’ll work closely with the Web Made Movies community, where already 3 persons have manifested their interest – Mark Boas, Samuel Huron and Nick Doiron, all of them also Learning Lab fellows!

Teaching journalists how to fish?

Julien — Sat, 06 Aug 2011 01:14:13 +0200

You know the saying: teach people how to fish… instead of giving them a fish. Because of the web, because of the iPad, and because of Twitter, Facebook and YouTube, newsrooms have a cruel choice to make.

They can hope that someone else will built tools and formats for the digital age, and that the tools and formats will be simple enough for anyone (including a classically trained journalist) to use and apply in their newsroom. Or they can learn to fish, and try to turn the newsroom into a full speed tech company.

Eating vendors’s fishes

It’s a cruel dilemma, because we can’t be sure any of them is a winning choice. Many of the internally built tools are nothing more than cool experiments – not game changers.

Of course you can work with great agencies like Information Architects, Inc., but if you want to use them to grow your internal capacity to innovate, you are back to square one.

It may not be a choice at all, but much more a question of the internal culture of a given news organization. Eating the fishes you buy is a comfortable way to think of technology: we’ll just have to adopt new things at some point. For many organizations, this is the only way to deal with technology, because they don’t have any digital skills and culture internally.

(They use to have print mastery and have broadcast skills and culture internally, so why not digital mastery?)

We have seen that a relatively young news channel like Aljazeera is much more adept to experiment with the web than older channels. That’s because Aljazeera is still building its identity, going private this year.

Making sense of the flow

For example, it is becoming a truism that good stories are weaved with external materials more and more.

But it’s difficult to makes sense of this external material, because it comes in small pieces, loosely join – «micro-chunks» as Mohamed Nanabhay called them during its lecture.

As any news organization would have, Aljazeera at first wondered during the Egyptian revolution about the news value of those micro chunks, for example «3 minutes videos of people running in the street» without context to makes sense of them. The online newsroom quickly acknowledged them as an element of the news flow that people watch in between deeper reporting, and integrated them in their live blogs.

It can go much farther: a news team with at least some technical awareness and an active community can find «facts hidden in plain sight» in the daily flow of data says Evan Hansen.

Wired pushed its readers to use WikiScanner and report wiki-washing cases – covert self-serving edits by brands or politicians on Wikipedia.

Designing the news

News organizations are now not merely competing with blogs and other web news pure players, but they are now competing with web apps and mobile apps developers like Flipboard or Instapaper. These apps beats news websites in term of user experience design – or more simply said, in term of readability, exemplified by Oliver Reichenstein in his premium reading designs.

The more technically advanced news organizations seems to be on the way to transform themselves in tool building company.

We have seen this before with book publishers: O’reilly, because of its root as a technical books publisher, not only explored ebooks way before the others, but built several unique platforms, tools and workflows for publishing in the digital era - and open sourced several of them along the path.

The tipping point for news organizations – and other originally non-digital organizations – might be best summarized by Shazna Nessa when she says: «it was not long ago that I was presented as the IT person – it doesn’t happen anymore».

Introducing MetaFragments, a common format for timed metadata in HTML

Julien — Mon, 08 Aug 2011 14:37:10 +0200

MetaFragments gives to web tools, mobile apps, browser and search engines a simple way to explore, connect and share the inside content of web videos.

Even when liberated from Flash, web videos are data dead-ends. Meta-data for video or audio is mostly non-existent in exposed, usable form.

When it exists, it’s segregated in various silos: external files in non-browser friendly formats, APIs, JavaScript code…

That leads to bad findability inside the website, and bad search engine optimization. It makes it hard to connect similar stories, to cite a specific sequence in a longer video, or simply to link back to a precise phrase in the video

A standard way to expose timed meta-data in web pages can benefit a large range of web creators and users.

Video bloggers, mega-platforms like YouTube, news organizations betting on web videos as a rising media will get better findability and engagement.

Viewers will get new ways to find, share and connect to videos across the web.

How it works

The idea behind MetaFragments is to use media fragments URIs and in-HTML meta-data to store timed data relating to audio and videos sequences

Everything happens in the DOM, so web clients, crawlers and javascript libraries can easily find and use this data without prior knowledge of the website or its API. They just need to know existing standards.

MetaFragments is only one week old, so obviously it is a work in progress. We encourage you to read more about its proposed specifications here: MetaFragments Draft.

But the great news is that we are already rallying supports from Knight Mozilla Lab participants.

The Hyperaudio Pad tool from Mark Boas adopted it as its working format in its basic form, without RDFa.

Samuel Huron’s data visualization video player and Nicholas Doiron's FollowFrost linked video interview player are the next candidates – if MetaFragments works well with their data-enriched viewers, it will give them a free HTML API for search engine crawlers and other apps. It could help make their apps work together. I’m working on mapping their respective (and different!) internal JSON formats into MetaFragments, and we will iterate from here.

Shaminder Dulai, with its ambitious workflow project for tagging videos in the newsroom is also bringing feedback from a media producer, non-coder, point of view.

Here is the HTML code for the common use case of a word in transcription timed word by word:

In addition to being simple, MetaFragments is also based on existing web standards. We are designing simple and common patterns to apply those standards, sometimes hard to grasp, to real world problem data problems:

Finally, it's designed with the goal that the meta-data is always at most a click away from the resource being described, for humans and for apps alike:

I publish video, what do I gain?

Are you a video or audio creator? Or maybe you own a big video website?

When you implement MetaFragments, your HTML pages effectively become the data API of your video site.

What does that mean? It means that any web client –like a browser, but also an iPhone or Android app, or the crawlers from Google or Yahoo– can make sense of the data in your video, without prior knowledge of your site.

This allow your viewers to spread further your video stories, automatically accompanying the usual link with citations, names, and more infos that they previously would have had to type manually.

You also can use MetaFragments to index your own videos, even across multiple different platforms and website. That way, it will allows you, or your web community, to cross references and find hidden stories in your own footage.

Who will benefit from it?

Journalists, both professional and citizen, can easily reference and find specific sequences in audio and video medias, assembling stories with simpler online tools.

Media publishers have a standard way to make their video and audio content visible in the HTML, driving traffic and engagement.

developers get a straightforward cross browser method of exposing media metadata and the ability to link, assemble and integrate media fragments into their applications.

Artists can find inspiring video and audio recordings on the web and integrate creative commons specific sequences into their works, both manually and in new, automated, ways not possible today.

Scientists and academics can analyze, find correlations and more easily references medias with word-level precision when a MetaFragments transcript is present.

What’s next and what do we need?

MetaFragments is in the early draft stage. We need to refine it – for example add simple, clean and useful RDFa patterns

The right process for refining this standard is to continue to involve the community, and put it to the test in real world applications: popcorn.js web apps, video platforms, news websites and CMS plugins.

Working on MetaFragments also revealed the lack of easily usable vocabularies for timed and non-verbal informations in videos. We need to specify simple and quick to implement vocabularies for emotions, gestures, when they don't exist.

In term of adoption into CMS and frameworks, we need to foster the development of a Worpress plugin, a Drupal module and a Popcorn.js plugin.

With the support of the open source community and of video platforms and video website owners, MetaFragments can leverage the millions of video and audio reports by citizens, bloggers and journalists and turn them into data-rich stories.

Update: I added the MetaFragments equivalent of the YouTube subtitles to this post. That means that there is a machine readable version of the subtitles available directly in the page. You can take a look at them by viewing the source of this page