Recognizing activities in real-world videos is a difficult
problem exacerbated by background clutter, changes in
camera angle & zoom, and rapid camera movements.
Large corpora of labeled videos can be used to train au-
tomated activity recognition systems, but this requires
expensive human labor and time. This paper explores
how closed captions that naturally accompany many
videos can act as weak supervision that allows automat-
ically collecting ‘labeled’ data for activity recognition.
We show that such an approach can improve activity
retrieval in soccer videos. Our system requires no man-
ual labeling of video clips and needs minimal human
supervision. We also present a novel caption classifier
that uses additional linguistic information to determine
whether a specific comment refers to an ongoing activ-
ity. We demonstrate that combining linguistic analysis
and automatically trained activity recognizers can sig-
nificantly improve the precision of video retrieval.