Project Presentation : Real time distributed video transcoding using twitter storm and ffmpeg

Distributed Video Transcoding Rischan, Sunni December, 18, 13

What we’re doing? Comparing output time when we transcode the video files in different mode: ● standalone mode ● distributed but using program that we’re made ● distributed using twitter storm

Apologize We didn’t use OpenStack because ….. 1. Server just has 2 GB RAM 2. Real Machine more faster than Virtual Machine.

Standalone mode 1. Transcode video files using ffmpeg library in one PC. 2. No Split and concat. 3. Calculate the time processes.

Distributed using program that we made 1. 2. 3. 4. 5. 6. 7.

The program developed by python. Video Splitter. Send the video to the worker. Call the Ffmpeg to transcode. (problem?) Send back video from worker to server. Video Concatter. Show time processes.

distributed using twitter storm Something that must we know for understanding storm: 1. Setting storm environment for cluster. 2. Components of a storm cluster.

Setting Storm env 1. We need more than one PC (we can use VM).

PC Server(x.x.x.160) :Ubuntu, Intel(R) Core(TM)2 Quad CPU

Q9400 @ 2.66GHz, RAM 2 GB

PC Worker 1(x.x.x.160):Ubuntu, Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, RAM 8 GiB PC Worker 2(X.X.40.44) : Macbook, Intel Core 2 Duo 2.4 GHz, RAM 2 GiB

Setting Storm env (2) 2. Every PC must have Ffmpeg & storm 3. Storm Server, we need : ● ZeroMQ ● JZMQ ● Java ● Python ● Zookeeper

Storm Arch

Nimbus = JobTracker

Installing Storm for Cluster Dont forget to add id_rsa.pub workers in the server .ssh directory wget https://raw.github.com/maltoe/storm-install/master/storm_install.sh chmod +x storm_install.sh storm_install.sh all

Running Storm

We must create topology and then submit the topology for running storm video transcoding & kill the topology for stop it.

Result & Code Explanation

File Informations file : storm-video.mp4 duration : 00:07:57.84 resolution : 576x360 size : 9.728 MB

Standalone mode ffmpeg -i storm-video.mp4 storm-video.mkv Transcode from mp4 to mkv in PC Server :

0m36.936s

distributed using program that we made

Code Explanation:

Transcode Code

Transcode Code (1)

Transcode Code (2)

Transcode Code (3)

Transcode Code (4)

Code for Distribution

Code for Distribution1)

Result: file size before

split in server

9.728 MB

0m12.473s

send to worker

transcode in worker

send to server

concat in server

file size after

0m0.456s

4.152 MB

note PC Server

0:00:43

0m6.484s

0:00:00

PC worker 1 (Lab)

0:00:44

1m4.308s

0:00:04

PC Worker 2 (Dorm) Macbook

Total : 124 s / 2m4s

Using Storm

SplitVideo

Transcode and Concat Video

Main Topology

VideoSpout

Transcode Video Bolt [python]

Split Video Bolt [python]

Result: ● ● ●

Error, because we use subprocess in python for calling ffmpeg. If we use python libraries for ffmpeg, it’s deprecated, for concat function. The libraries is too old compared to latest version of ffmpeg We have tried to use java library (xuggler) for calling ffmpeg, but this library is discontinued. Most java libraries for ffmpeg is discontinued or too old.

Conclusion ● Storm is not suitable for video transcoding ○ Storm is for stream data, so we can’t use static video ○ If we use stream video, it can’t be distributed ○ It’s better to use Hadoop, although based on a paper, it doesn’t give a good performance

● Until now, we haven’t found reference for the same research

Twitter Topology

Twitter Spout

Twitter Result