Video variation and metadata for content
adaptation
The annotation framework automatically produces
metadata for video variation as a means for
adaptation. In video variation, the hope is to
generate new videos (variation videos) from the
source video, with a reduced data size and quality
by applying a variation or reduction method.
The supported variations include
temporal variation, which reduces the visual
data’s frame rate through frame dropping;
spatial variation, which encodes fewer pixels
(pixel subsampling) and thereby reduces the
level of detail in the frame; and
color variation, which reduces each pixel’s
color depth.
The CODAC project has defined and implemented
two new methods of video variations:5
Object-based variation extracts the foreground
and background objects in a source video and
re-encodes the video with two visual objects.
This facilitates object-based adaptation, which
otherwise would be impossible for a video
encoded without objects. For instance, in our
use case scenario, object-based variation
would be useful for segments with a dynamic
singer foreground and static image background.
This lets an adaptation engine discard,
in case of resource bottlenecks, the static
background image.
❚ Segment-based variation lets us apply variation
methods selectively for video segments based
on the physical characteristics of motion,
texture, and color, thereby minimizing the
quality loss and/or maximizing the reduction
in data size. We segment the source video into
its component shots and automatically select
and apply a set of appropriate methods by
analyzing the degree of physical characteristics
within the segment.
We accomplish variation creation by implementing
a server module called the Variation Factory,
which generates the variations.5 All the
variation methods—including the newly defined
object and segment-based variation methods—are
implemented in the Variation Factory. A source
video can be subjected to a single method or a
combination of them as deemed necessary—for
instance, temporal variation followed by spatial
variation, and so on. At each step, the Variation
Factory produces a variation video, thereby creating
a tree of variations. The Variation Factory is
an application programming interface (API)
including user interfaces to guide the metadata
generation process. The user can control the
adaptation results and rearrange parameters—for
instance, if the perceived quality is too low.
The Variation Factory’s architecture is developed
in Java under Java Media Framework (JMF).
The API supports the integration of audio and
video playback into Java applications and applets.6
The input is video, and the outputs are one or
more adapted variation videos and an MPEG-7
metadata document that describes the source and
the variation videos using descriptors of the VariationSet
description scheme. The MPEG-7 Document
Processor produces metadata. It uses Java
API for XML processing (JAXP), which is generally
used to parse and transform XML documents.
First, a Document Object Model tree is constructed.
Then, the DOM tree is parsed and an
XML text file is produced. The descriptors
include information on the variations’ fidelity,
data size, and priority. We store the created variation
videos in the media server along with the
source video and the MPEG-7 document in the
metadatabase. During delivery, the MPEG-7
metadata document is streamed together with
the requested video.
