My HYP - Honours Year Project - is to do a news video summarization system. A typical scenario is a user searching for a news topic from a mobile device. Intelligently, the system will:
- use NLP to refine the search terms
- crawls the Internet (eg. google snippets, blogs) for the latest interesting related topics
- rank and retrieve the most well matched news video from our news corpus
- perform summarization on the selected news video story, ensuring that the summary is concise, informative, interesting and coherent
- perform device adaptation such that the video file is playable and fits the screen correctly
The research value comes from the summarization portion, in which a news story of about 1 to 5 minutes has to be shortened even further. But this is a very well needed system as our time is limited. I have always wished that I can have the full 30 min news shortened to just 5 min, but still keeping as much important information that is needed.
It turns out there are so many factors to consider when doing summarization. How do I rank and choose the most informative audio segments? How do I mix video shots from other temporal? How do I ensure that there will be coherence in the audio, video and content? Even at the end of producing a summary, I have to ask - How do I evaluate/grade my summarization techniques?
Video & Audio ProcessingTo develop the system, I need a lot of video & audio processing. As Java is my preferred language, I have no choice but to use the forsaken JMF (Java Media Framework).
I also used ffmpeg for any other preprocessing work (not within the system).
Some guides:
Some random development notes:
- On using frame or time as unit? Translate to time finally. Getting frame might not be possible for all.
- There are so much problems and unknown bugs in JMF when I am just doing transcode and cut & merge videos.
- One of the problem is the present of white noise between segments of video after I did a cut & merge. I spent 1 whole day on Christmas analyzing when will white noise be present and then I tried on a silly solution which I knew would not be solving the root of the problem. It did not work perfectly. After I came out from a shower, I tried with different audio encoding. Turns out setting the sample rate to 8000 solve the problem!
Libraries Used:
- DownloadServlet - It is one of the most common seek after servlet. I used it to serve video files outside my container webapps.
Location Based Service could be next big thing for mobile application.
