readme.md.html

<h2 id="webrtcbench">WebRTCBench</h2>
<h2 id="overview">Overview</h2>
<p>WebRTC is an industry and standards effort to provide real-time communication capabilities into all browsers and make these capabilities accessible to software developers via standard HTML5 and Javascript APIs [1].
WebRTC fills a critical gap in web technologies by allowing (a) the browser to access native devices (e.g., microphone, webcam) through a Javascript API and (b) to share the captured streams through using browser-to-browser Real-Time Communication. WebRTC also provides data sharing.</p>
<h3 id="webrtcbench-in-a-nutshell">WebRTCBench in a Nutshell</h3>
<p>The goal of WebRTCBench is to provide a quantitative comparison of WebRTC implementations across browsers and devices (i.e., hardware platforms). WebRTC accomplishes three main tasks: Acquiring audio and video; Communicating Audio and Video; Communicating Arbitrary Data. The
se tasks are mapped one to one to three main Javascript APIs. These are as follows: MeadiaStream (i.e., getUserMedia); RTCPeerConnection; RTCDataChannel. Hence, a quantitative assessment of WebRTC implementations across browser and devices is performed via collecting performance of MediaStream, RTCPeerConnection, and RTCDataChannel.
Because a MediaStream contains one or more media stream tracks (e.g., Webcam and Microphone), WebRTCBenc allows to define MediaStreams composed of Video, Audio, Data and any combination thereof. Likewise single peer connection with web server and multiple peer connections between browsers are supported in a WebRTC triangle [1].</p>
<h3 id="what-is-included-">What Is Included ?</h3>
<p>This release is composed of two components:</p>
<ol>
<li>a web server based on socket.io [2]</li>
<li>a WebRTCBench application on peers side.</li>
</ol>
<p>Webserver: Benchmark comes along with a Webserver which when run, allow clients to download the WebRTCBench application and to do the    signalling. Signaling trough STUN or TURN servers is currently not supported. Furthermore, it is used to integrate a database.
WebRTCBench application: A WebRTC application that allows to acquire streams, open connections, and both streams and data sharing.do testing and performance measuring of different functionalists of WebRTC implementation across different browsers.</p>
<h3 id="webrtcbench-0-2">WebRTCBench 0.2</h3>
<p>This version of WebRTCBench adds an automated mechanism to collect experiment information from peers' browsers within text format and also in rational database. Although benchmark supports having up to 10 peers connecting to each other in mesh form, collecting information is currently limited to connection between only two peers. Also, web server now provide a secure connection.</p>
<h2 id="getting-started">Getting started</h2>
<p>This section illustrates how to download WebRTCBench, install and execute the web server; and how to set up the database. Web server and database can be installed both on Linux and Windows based machines.</p>
<h3 id="setting-up-and-starting-the-web-server">Setting Up and Starting the Web Server</h3>
<p>In the first step download and extract the benchmark.</p>
<p>The web server is based on Node.js; After obtaining Node.js binary from <a href="http://nodejs.org/download/">http://nodejs.org/download/</a>, enter into the directory webrtcbench-code and execute the following command.</p>
<pre class="editor-colors"><code class="lang-bash"><div><span class="source shell">    <span class="variable other normal shell"><span class="punctuation definition variable shell">$</span>npm</span> install express async mysql socket.io</span></div></code></pre>
<p>To start the web sever use the following command (On Unix/Linux machines the user issuing the following command must be root or a sudoer):</p>
<pre class="editor-colors"><code class="lang-bash"><div><span class="source shell">    <span class="variable other normal shell"><span class="punctuation definition variable shell">$</span>node</span> socket-server.js <span class="keyword operator redirect shell">&gt;</span> socket-server.log <span class="keyword operator list shell">&amp;</span></span></div></code></pre>
<p>The web server will be listening to the socket indicated during its configuration. To stop the signaling server, use the following command on Unix/Linux machines:</p>
<pre class="editor-colors"><code class="lang-bash"><div><span class="source shell">    <span class="variable other normal shell"><span class="punctuation definition variable shell">$</span>kill</span> -9 <span class="string interpolated backtick shell"><span class="punctuation definition string begin shell">`</span>pidof node<span class="punctuation definition string end shell">`</span></span></span></div></code></pre>
<p>or press "Cntrl+C" on Windows machines.</p>
<h3 id="setting-up-the-database">Setting Up The Database</h3>
<p>At this step, you should have a version of MySQL database installed on your machine. In order to create the database, a MySQL database user with sufficient privileges should execute the "createdb.sql" script provided by the benchmark using the following command in MySQL console.</p>
<pre class="editor-colors"><code class="lang-bash"><div><span class="source shell">    <span class="variable other normal shell"><span class="punctuation definition variable shell">$</span>source</span> createdb.sql</span></div></code></pre>
<p>By executing this command a database called "webrtcdb" --if does not already exist-- will be created. After creating the database, the following modifications should be made to "config.js" configuration file.</p>
<ol>
<li>Make sure that "db.present" variable is set to true</li>
<li>parameters regarding database address and authentication is set properly. Note that regardless of database presence, timing information will always be saved into text files on the web server.<pre class="editor-colors"><code class="lang-js"><div><span class="source js">  db<span class="meta delimiter method period js">.</span>present <span class="keyword operator js">=</span> <span class="constant language boolean true js">true</span><span class="punctuation terminator statement js">;</span>  <span class="comment line double-slash js"><span class="punctuation definition comment js">//</span> DB is present</span></span></div><div><span class="source js">  db<span class="meta delimiter method period js">.</span><span class="support constant js">host</span> <span class="keyword operator js">=</span> <span class="string quoted single js"><span class="punctuation definition string begin js">'</span>localhost<span class="punctuation definition string end js">'</span></span><span class="punctuation terminator statement js">;</span></span></div><div><span class="source js">  db<span class="meta delimiter method period js">.</span>user     <span class="keyword operator js">=</span> <span class="string quoted single js"><span class="punctuation definition string begin js">'</span>user<span class="punctuation definition string end js">'</span></span> <span class="punctuation terminator statement js">;</span></span></div><div><span class="source js">  db<span class="meta delimiter method period js">.</span>password <span class="keyword operator js">=</span> <span class="string quoted single js"><span class="punctuation definition string begin js">'</span>pass<span class="punctuation definition string end js">'</span></span> <span class="punctuation terminator statement js">;</span></span></div><div><span class="source js">  db<span class="meta delimiter method period js">.</span>database <span class="keyword operator js">=</span> <span class="string quoted single js"><span class="punctuation definition string begin js">'</span>webrtcdb<span class="punctuation definition string end js">'</span></span> <span class="punctuation terminator statement js">;</span></span></div></code></pre>
</li>
</ol>
<h3 id="user-guide">User Guide</h3>
<p>To use WebRTCBench, a user has to open multiple instances of a browser and navigate to the benchmark's hosted location. For example, if the benchmarks hosted location is core7.ics.uci.edu, open a browser and access the benchmark home page at the URL <a href="http://core7.ics.uci.edu">http://core7.ics.uci.edu</a>.
User should provide its own machine specification at "User Agent Info" section menu. These specification include: device type, processor, connection type. By pressing "Remember Information" button the provided specification will be stored in browser cache and will be restored in later uses. Once at the benchmark home page, a user can select any combination of audio, video, or data in the connection type box, choose a channel id or number to open, and then select "Open Channel". To add a peer connection, input the opened channel id or number into the channel name of a new browser; then input the connection type into the appropriate box. A connection of the selected type will then be opened between the two browsers.
If an audio/video connection is selected, a prompt will appear to ask the user permission for accessing the corresponding devices; the user must allow access for the application to work properly. If the connection between browsers and web server is secure, browsers will remember user permission when accessing that website.
Multiple audio and video connections can be created by navigating the home page and joining channels from different browsers. If a data connection is opened, any user can send a text message or send a file to other users by using the appropriate web forms.
Information about completion time for operations related to MediaStream, RCTPeerConnection and RTCDataChannel is displayed in the console below the application. If "Sending timings to Server" is enabled in GUI, information will be sent to server and will be stored in text files named in the format "$channel_no.txt" where $channel_no is the channel number for that connection. Additionally if configured, those information can be stored in the database for further analysis. Currently, only experiments involving two peers can be inserted in the database.</p>
<h3 id="supported-browsers">Supported Browsers</h3>
<ul>
<li>Chrome 25.0+ (28.0+ for best performance)</li>
<li>Firefox 22.0+</li>
<li>Chrome 29.0+ on Android</li>
</ul>
<h2 id="developer-guide">Developer Guide</h2>
<p>Socket.io [2] and Node.js [3] are used to implement and execute the web server. Additionally, following Node.js modules are used:
express
node-mysql
async
We have used community edition of MySQL to set up the database. If using a different database is amenable, a different adapter should be provided in server code.</p>
<h3 id="webrtc-events">WebRTC Events</h3>
<p>A list of WebRTC events are defined in js/events.js file. WebRTC application will use them for logging.</p>
<pre class="editor-colors"><code class="lang-javascript"><div><span class="source js">        <span class="support variable js">exports</span><span class="meta delimiter method period js">.</span>Events <span class="keyword operator js">=</span> <span class="meta brace curly js">{</span></span></div><div><span class="source js">        CREATING_PC <span class="keyword operator js">:</span> <span class="string quoted double js"><span class="punctuation definition string begin js">"</span>Creating RTCPeerConnection object<span class="punctuation definition string end js">"</span></span><span class="meta delimiter object comma js">,</span></span></div><div><span class="source js">        PC_CREATED <span class="keyword operator js">:</span> <span class="string quoted double js"><span class="punctuation definition string begin js">"</span>RTCPeerConnection object created<span class="punctuation definition string end js">"</span></span><span class="meta delimiter object comma js">,</span></span></div><div><span class="source js">        GETTING_MEDIA<span class="keyword operator js">:</span> <span class="string quoted double js"><span class="punctuation definition string begin js">"</span>Getting user media<span class="punctuation definition string end js">"</span></span><span class="meta delimiter object comma js">,</span></span></div><div><span class="source js">        LOCAL_MEDIA_CAPTURED<span class="keyword operator js">:</span> <span class="string quoted double js"><span class="punctuation definition string begin js">"</span>Local media captured<span class="punctuation definition string end js">"</span></span><span class="meta delimiter object comma js">,</span></span></div><div><span class="source js">        LOCAL_PLAYBACK_STARTED<span class="keyword operator js">:</span><span class="string quoted double js"><span class="punctuation definition string begin js">"</span>Local media playback started<span class="punctuation definition string end js">"</span></span><span class="meta delimiter object comma js">,</span></span></div><div><span class="source js">        REMOTE_STREAM_ARRIVED<span class="keyword operator js">:</span> <span class="string quoted double js"><span class="punctuation definition string begin js">"</span>Remote stream arrived<span class="punctuation definition string end js">"</span></span><span class="meta delimiter object comma js">,</span></span></div><div><span class="source js">        REMOTE_PLAYBACK_STARTED<span class="keyword operator js">:</span> <span class="string quoted double js"><span class="punctuation definition string begin js">"</span>Remote playback started<span class="punctuation definition string end js">"</span></span><span class="meta delimiter object comma js">,</span></span></div><div><span class="source js">        CREATING_DATA_CHANNEL<span class="keyword operator js">:</span> <span class="string quoted double js"><span class="punctuation definition string begin js">"</span>Creating Datachannel object<span class="punctuation definition string end js">"</span></span><span class="meta delimiter object comma js">,</span></span></div><div><span class="source js">        DATA_CHANNEL_OPENED<span class="keyword operator js">:</span> <span class="string quoted double js"><span class="punctuation definition string begin js">"</span>Datachannel opened<span class="punctuation definition string end js">"</span></span><span class="meta delimiter object comma js">,</span></span></div><div><span class="source js">        CALLEE_ARRIVED <span class="keyword operator js">:</span> <span class="string quoted double js"><span class="punctuation definition string begin js">"</span>Callee arrived<span class="punctuation definition string end js">"</span></span><span class="meta delimiter object comma js">,</span></span></div><div><span class="source js">        CREATING_OFFER <span class="keyword operator js">:</span> <span class="string quoted double js"><span class="punctuation definition string begin js">"</span>Creating offer<span class="punctuation definition string end js">"</span></span><span class="meta delimiter object comma js">,</span></span></div><div><span class="source js">        OFFER_CREATED <span class="keyword operator js">:</span> <span class="string quoted double js"><span class="punctuation definition string begin js">"</span>Offer created<span class="punctuation definition string end js">"</span></span> <span class="meta delimiter object comma js">,</span></span></div><div><span class="source js">        CREATING_ANSWER<span class="keyword operator js">:</span> <span class="string quoted double js"><span class="punctuation definition string begin js">"</span>Creating answer<span class="punctuation definition string end js">"</span></span><span class="meta delimiter object comma js">,</span></span></div><div><span class="source js">        ANSWER_CREATED<span class="keyword operator js">:</span> <span class="string quoted double js"><span class="punctuation definition string begin js">"</span>Answer created<span class="punctuation definition string end js">"</span></span><span class="meta delimiter object comma js">,</span></span></div><div><span class="source js">        SETTING_REMOTE_DESCRIPTION<span class="keyword operator js">:</span> <span class="string quoted double js"><span class="punctuation definition string begin js">"</span>Setting remote description<span class="punctuation definition string end js">"</span></span><span class="meta delimiter object comma js">,</span></span></div><div><span class="source js">        REMOTE_DESCRIPTION_SET <span class="keyword operator js">:</span> <span class="string quoted double js"><span class="punctuation definition string begin js">"</span>Remote description set<span class="punctuation definition string end js">"</span></span><span class="meta delimiter object comma js">,</span></span></div><div><span class="source js">        NEW_ICE_CANDIDATE<span class="keyword operator js">:</span> <span class="string quoted double js"><span class="punctuation definition string begin js">"</span>On new ICE Candidate<span class="punctuation definition string end js">"</span></span><span class="meta delimiter object comma js">,</span></span></div><div><span class="source js">        CONNECTING_TO_SIGNALLING_SERVER<span class="keyword operator js">:</span> <span class="string quoted double js"><span class="punctuation definition string begin js">"</span>Connecting to signaling server<span class="punctuation definition string end js">"</span></span><span class="meta delimiter object comma js">,</span></span></div><div><span class="source js">        CONNECTED_TO_SIGNALLING_SERVER <span class="keyword operator js">:</span> <span class="string quoted double js"><span class="punctuation definition string begin js">"</span>Connected to signalling server<span class="punctuation definition string end js">"</span></span><span class="meta delimiter object comma js">,</span></span></div><div><span class="source js">        ICE_CONNECTED <span class="keyword operator js">:</span> <span class="string quoted double js"><span class="punctuation definition string begin js">"</span>ICE Connected<span class="punctuation definition string end js">"</span></span></span></div><div><span class="source js">    <span class="meta brace curly js">}</span><span class="punctuation terminator statement js">;</span></span></div></code></pre>
<h5 id="logging-events">Logging events</h5>
<pre class="editor-colors"><code class="lang-javascript"><div><span class="source js">    <span class="storage modifier js">var</span> eventLogger <span class="keyword operator js">=</span> <span class="meta brace curly js">{</span></span></div><div><span class="source js">      <span class="meta function json js"><span class="entity name function js">info</span> :  <span class="storage type function js">function</span><span class="punctuation definition parameters begin js">(</span><span class="variable parameter function js">event</span>, <span class="variable parameter function js">timestamp</span><span class="punctuation definition parameters end js">)</span></span><span class="meta brace curly js">{</span></span></div><div><span class="source js">      <span class="meta brace curly js">}</span> <span class="meta delimiter object comma js">,</span></span></div><div><span class="source js">      <span class="meta function json js"><span class="entity name function js">verbose</span> :  <span class="storage type function js">function</span><span class="punctuation definition parameters begin js">(</span><span class="variable parameter function js">event</span>, <span class="variable parameter function js">timestamp</span><span class="punctuation definition parameters end js">)</span></span> <span class="meta brace curly js">{</span></span></div><div><span class="source js">      <span class="meta brace curly js">}</span></span></div><div><span class="source js">    <span class="meta brace curly js">}</span><span class="punctuation terminator statement js">;</span></span></div></code></pre>
<ul>
<li>This is a client-side object which provides two functions that insert an event into a events dictionary. Both methods require two parameters: "$timestamp" is the measured time and will be stored in dictionary location indexed by "$event" type.</li>
<li>Events inserted using verbose method will be printed on console only in verbose mode.</li>
<li>Events dictionary will be sent to server later at the end of experiment.</li>
<li>Server will measure WebRTC performance by interpreting these events according to WebRTC connection establishment flow.</li>
</ul>
<p><img src="c:\Users\Saji\Dropbox\WebRTCBench-1.0-Clean\webrtchbench-1.0\flow.png" alt="alt tag"></p>
<h3 id="webrtcbench-using-https-connection">WebRTCBench Using HTTPS Connection</h3>
<p>When connected through HTTPS, media access permission granted to WebRTC application will be persistant for that particular web site (Firefox doesn't support persistant permission yet). Hence, the time measured to "Get user media" would exclude human interaction and would be more accurate. To enable web server to accept HTTPS connections, a certificate and key file should be provided. These files can be generated by tools such as OpenSSL[4]. The following java script code shows how to create a node server with HTTPS support given certificate and key files.</p>
<pre class="editor-colors"><code class="lang-javascript"><div><span class="source js">    <span class="storage modifier js">var</span> https <span class="keyword operator js">=</span> <span class="support function js">require</span><span class="meta brace round js">(</span><span class="string quoted single js"><span class="punctuation definition string begin js">'</span>https<span class="punctuation definition string end js">'</span></span><span class="meta brace round js">)</span><span class="punctuation terminator statement js">;</span></span></div><div><span class="source js">    <span class="storage modifier js">var</span> options <span class="keyword operator js">=</span> <span class="meta brace curly js">{</span></span></div><div><span class="source js">        key<span class="keyword operator js">:</span> fs<span class="meta delimiter method period js">.</span>readFileSync<span class="meta brace round js">(</span><span class="string quoted single js"><span class="punctuation definition string begin js">'</span>./server.key<span class="punctuation definition string end js">'</span></span><span class="meta brace round js">)</span><span class="meta delimiter method period js">.</span><span class="support function js">toString</span><span class="meta brace round js">()</span><span class="meta delimiter object comma js">,</span></span></div><div><span class="source js">        cert<span class="keyword operator js">:</span> fs<span class="meta delimiter method period js">.</span>readFileSync<span class="meta brace round js">(</span><span class="string quoted single js"><span class="punctuation definition string begin js">'</span>./server.crt<span class="punctuation definition string end js">'</span></span><span class="meta brace round js">)</span><span class="meta delimiter method period js">.</span><span class="support function js">toString</span><span class="meta brace round js">()</span></span></div><div><span class="source js">    <span class="meta brace curly js">}</span><span class="punctuation terminator statement js">;</span></span></div><div><span class="source js">    <span class="storage modifier js">var</span> securePort <span class="keyword operator js">=</span> config<span class="meta delimiter method period js">.</span>ws<span class="meta delimiter method period js">.</span>securePort<span class="punctuation terminator statement js">;</span></span></div><div><span class="source js">    server <span class="keyword operator js">=</span> https<span class="meta delimiter method period js">.</span>createServer<span class="meta brace round js">(</span>options<span class="meta delimiter object comma js">,</span>app<span class="meta brace round js">)</span><span class="meta delimiter method period js">.</span>listen<span class="meta brace round js">(</span>securePort<span class="meta brace round js">)</span><span class="punctuation terminator statement js">;</span></span></div></code></pre>
<h2 id="webrtcbench-database">WebRTCBench Database</h2>
<p>This section describes the database that is used to store experiments information.</p>
<h3 id="database-structure">Database Structure</h3>
<p>WebRTCBench database as depicted as follows is composed of four tables. Fields with yellow key sign (e.g. experiment_id and peer_id in peer table) are uniquely assigned by the database engine. Database tables are briefly explained here:
<img src="c:\Users\Saji\Dropbox\WebRTCBench-1.0-Clean\webrtchbench-1.0\Scheme.png" alt="alt tag"></p>
<ol>
<li><p>Peer: An entry of this table represents a client(peer) involved in experiment. Each entry has four fields:
device: Client Device Type( e.g. Mobile/PC)
os: Client Operating System
browser: Client Browser
Network: Client Network type ( e.g. LAN,WIFI)</p>
</li>
<li><p>Experiment: Each entry of this table represents an experiment conducted between two specific peers specified by sender_id and receiver_id.
stream_type: Type of stream
3 characters 'A', 'V' and 'D' represents Audio, Video and Data respectively. For connection with multiple streams a combination of letters will be used. e.g. 'AV' for 'Audio+Video'.
task_setup: Task setup( e.g. PC)</p>
</li>
<li><p>sender_timings and receiver_timings:</p>
</li>
</ol>
<h3 id="database-adapter">Database adapter</h3>
<p>Database integration is composed of the following function:</p>
<pre class="editor-colors"><code class="lang-javascript"><div><span class="source js">    <span class="meta function js"><span class="storage type function js">function</span> <span class="entity name function js">addExperiment</span><span class="punctuation definition parameters begin js">(</span><span class="variable parameter function js">connection</span>,<span class="variable parameter function js">sender</span>,<span class="variable parameter function js">receiver</span>,<span class="variable parameter function js">streamType</span>,<span class="variable parameter function js">taskSetup</span><span class="punctuation definition parameters end js">)</span></span></span></div></code></pre>
<p>a.This is a server-side function which inserts experiment information into database. It should be called when all experiment information from two peers are collected.
b. First, sender and receiver peers information will be inserted into peer tables (if not already in there. Peers are identified by their agent information: processor, browser, ...)
c. Inserts an experiment involving two peers from previous step (only if such experiment does not exist already).
d. sender and receiver timings table will be filled using data fields from sender and receiver objects accordingly.  </p>
<h3 id="adding-measurements">Adding Measurements</h3>
<p>The following steps describe how to add another measurement to WebRTC benchmark:</p>
<ol>
<li><p>Add the new event to "EVENTS" variable in "events.js" file if does not already exists. Entries in "Events" have the following structure:</p>
<pre class="editor-colors"><code class="lang-js"><div><span class="source js">       $Id <span class="keyword operator js">:</span> <span class="string quoted double js"><span class="punctuation definition string begin js">"</span>text<span class="punctuation definition string end js">"</span></span></span></div></code></pre>
<p>when $Id is a unique identifier for that entry and $text is textual representation of event.</p>
</li>
<li><p>Use a reserved column in the sender_timings and receiver_timings. Execute the following command in MySQL console to rename the column name.</p>
<pre class="editor-colors"><code class="lang-mysql">        ALTER TABLE sender_timings CHANGE timing_type_i new_name INT;
        ALTER TABLE receiver_timings CHANGE timing_type_i new_name INT;
</code></pre>
</li>
<li>Modify "addExperiment" function to reflect database changes.</li>
<li>After performance measuring of your area of interest with WebRTCBench, call "eventLog" function with event and time parameters.</li>
</ol>
<h3 id="references">References</h3>
<p>[1] Alan B. Johnston and Daniel C. Burnett. 2012. WebRTC: APIs and RTCWEB Protocols of the Html5 Real-Time Web. Digital Codex LLC, USA.
[2] Socket.io <a href="http://socket.io/">http://socket.io/</a></p>