Web DataView - Introduction

The goal of this project will be to have a simple, easily adaptable web-based view of different kinds

of (environmental) data.
A simple but attractive webpage must show both actual and historical values using different kinds of charts. The backend server must be easy to deploy on Linux and Windows and must provide a straightforward interface with both a database and text based data files.
This blog is all about the building of the infrastructure, creating the webservice and the pages.

• The Server
•  Getting the Data
• Showing the data

The Server

The server will be standard Apache. Combined with the WSGI framework this will give a fairly standard python based webserver configuration. I actually already wrote a blog post on how to set this up. This post however stops at serving the 'Hello World' and some additional steps will be required to serve both the web pages as to process the data requests.
First I added the following to the httpd.conf file to configure Apache:

<VirtualHost *:8081>

    ServerAlias example.com
    ServerAdmin cees@nortek-bv.nl

    DocumentRoot c:/DATA/xampp/apache/htdocs

    <Directory c:/DATA/xampp/apache/htdocs>
     AuthType Basic
     AuthName "Scour Server Login"
     AuthUserFile c:/DATA/xampp/apache/passwords/.htpasswd
     AllowOverride all
     Require valid-user
    </Directory>

    WSGIScriptAlias /get_data c:/DATA/xampp/wsgi/get_data.wsgi

    <Directory c:/DATA/xampp/wsgi>
     Require all granted
    </Directory>

</VirtualHost>

# Python WSGI interface module
<IfModule wsgi_module>
  WSGIPythonHome "C:/Python27"
</IfModule>

Putting the rules in a 'VirtualHost' section  allows us to configure differnent setups on other ports if required. The 'AuthType', 'AuthUserFile' and  'Require valid-user' enable a user-name / password security for the site. User-name and Password have to be entered in the '.htpasswd' file using the htpasswd utility. (Check the Authentication page in the Apache docs for details)
By setting the WSGIScriptAlias "/get_data" to the actual program path I can call the program by simply going to the URL www.my_site.com/get_data.
A very nice basic instruction on how to use the WSGI module and to process the requests is given on the WSGI Tutorial pages . What is particularly nice is the example where the program shows all values in the 'environ' dictionary of the WSGI application:

COMSPEC: C:\WINDOWS\system32\cmd.exe
CONTEXT_DOCUMENT_ROOT: C:/DATA/xampp/apache/htdocs
CONTEXT_PREFIX: 
DOCUMENT_ROOT: C:/DATA/xampp/apache/htdocs
GATEWAY_INTERFACE: CGI/1.1
HTTP_ACCEPT: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
HTTP_ACCEPT_ENCODING: gzip, deflate
HTTP_ACCEPT_LANGUAGE: nl,en-US;q=0.7,en;q=0.3
HTTP_CONNECTION: keep-alive
HTTP_HOST: localhost:8081
HTTP_USER_AGENT: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0
PATHEXT: .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY
PATH_INFO: /
PATH_TRANSLATED: C:\DATA\xampp\apache\htdocs\index.html
QUERY_STRING: age=10&hobbies=software&hobbies=tunning
REMOTE_ADDR: ::1
REMOTE_PORT: 52609
REQUEST_METHOD: GET
REQUEST_SCHEME: http
REQUEST_URI: /myapp/?age=10&hobbies=software&hobbies=tunning
SCRIPT_FILENAME: C:/DATA/xampp/wsgi/hello_world.wsgi
SCRIPT_NAME: /myapp
SERVER_ADDR: ::1
SERVER_ADMIN: cees@nortek-bv.nl
SERVER_NAME: localhost
SERVER_PORT: 8081
SERVER_PROTOCOL: HTTP/1.1
SERVER_SIGNATURE: 
SERVER_SOFTWARE: Apache/2.4.12 (Win64) mod_wsgi/3.4 Python/2.7.9
SystemRoot: C:\WINDOWS
WINDIR: C:\WINDOWS
mod_wsgi.application_group: Lenovo-Cees.nortek.local:8081|/myapp
mod_wsgi.callable_object: application
mod_wsgi.enable_sendfile: 0
mod_wsgi.handler_script: 
mod_wsgi.input_chunked: 0
mod_wsgi.process_group: 
mod_wsgi.request_handler: wsgi-script
mod_wsgi.script_reloading: 1
mod_wsgi.version: (3, 4)
wsgi.errors: <mod_wsgi.Log object at 0x0000000004054DF0>
wsgi.file_wrapper: <built-in method file_wrapper of mod_wsgi.Adapter object at 0x0000000002A4BC60>
wsgi.input: <mod_wsgi.Input object at 0x0000000002A52470>
wsgi.multiprocess: False
wsgi.multithread: True
wsgi.run_once: False
wsgi.url_scheme: http
wsgi.version: (1, 0)

That's a wealth of information ! When you look at this you immediately get a grip on the power of the WSGI framework.

Debugging the wsgi app and Apache

Debugging is harder than normal. Since the wsgi script only runs when called by the server there is no 'real  time' debugging information. Fortunately apache writes all error messages to a file: error.log so it is at least possible to see what goes wrong when you get a server error. What's annoying is that you have to re-open this file to see the changes. Looking for a solution to this I found the Document Monitor plugin for Notepad++. Just go to the 'Plugins->Plugin Manager' and select 'Document Monitor'. When activated on the 'error.log' file it will now update continuously when new lines are added to the file.

Showing the data - Charts

The options to display charts on a web-page are numerous. Since I want this to be a client side action I will limit myself to using a Javascript library. There is an overwhelming amount of Javascript based frameworks that will do just that so it's hard to make a decision on which one to use.


The ones that I considered so far:

jqPlot  . Fully open source library by Chris Leonello .Very nice looking charts. Small and lightweight. Not very actively developed right now, but the core seems pretty complete and bug-free.

Flot.  JavaScript plotting library for jQuery, with a focus on simple usage, attractive looks and interactive features. Although at the time of writing (February 2015) still the library has not hit the 1.0 release, it seems very complete and there is steady development going on. It is used in the Start Bootstrap template. Flot is completely free to use and commercial support is provided on special request to the creator.

morris.js. Another library that is used in the Bootstrap template. Although it looks equally attractive as Flot, it looks like this is one man project and it's not clear where it's heading. Morris.js is a very simple API for drawing line, bar, area and doughnut charts. For full usage instructions and documentation for Morris.js charts, visit http://www.oesmith.co.uk/morris.js/.


Chart.js Chart.js uses the HTML5 canvas element. It's supported in all modern browsers, and polyfills support for IE7/8. It is dependency free and super lightweight. All six core chart types are only 11.01kb when minified, concatenated and served gzipped. It will resize your chart if the browser changes size, along with providing the perfect scale granularity for that size. Development seems pretty active and it's got good documentation.

dygraphs. Handles huge data sets: dygraphs plots millions of points without getting bogged down.
Interactive out of the box: zoom, pan and mouseover are on by default. Strong support for error bars / confidence intervals. Highly customizable: using options and custom callbacks, you can make dygraphs do almost anything.
There's an active community developing and supporting dygraphs, but it's large and can be overwhelming.

Google Charts. Very complete charting library with the added benefit that it is backed by Google. Google Chart requires internet connection and also has some limits on the number of client requests allowed.

D3.js is a JavaScript library for manipulating documents based on data. D3 helps you bring data to life using HTML, SVG and CSS. D3’s emphasis on web standards gives you the full capabilities of modern browsers without tying yourself to a proprietary framework, combining powerful visualization components and a data-driven approach to DOM manipulation. It's very comprehensive and probably overkill if you just want some line-charts.

RGraph.  Javascript library with over 20 chart types that uses the Canvas. Free for non-commercial use, but 99 UKP for a commercial license is not expensive either. It does have the slightly more advanced looks that you find in commercial products.
 

gRaphael. Very slick charts. Based on Raphaël graphics library, which is currently part of the Sencha Labs suite. (commercial). gRaphaël currently supports Firefox 3.0+, Safari 3.0+, Opera 9.5+ and Internet Explorer 6.0+. There has not been much development on it for the past 2 years, but the existing version is pretty good.

Ember Charts. Open source repository built with D3.js and Ember.js. It provides time series, bar, pie, and scatter charts that are easily customisable. It uses SVG to render charts.

EJS Chart The charts look clean and readable than most of the older charting libraries. It is also compatible with IE6+ and other older browsers. Here is a list of examples.
EJS Chart comes in free and paid versions. The free version limits you to use maximum of 1 chart per page and 2 series per chart. Single developer license is only US$100,- though.

uvCharts. Based on d3.js and SVG/HTML5 to build charts. 12 chart types, 100+ options. Well documented.

JSCharts. Good looking charts, and from looking at the examples, easy to use. There is a free version that has a watermark on it, and the (single domain) developer version is only US$79,- so that's not to bad.
 
Highcharts. Very comprehensive library. Fully HTML5 based Javascript, and explicit permission to modify and edit the source if needed. Impressive users list (Including Yahoo, Facebook and Visa) . Most charts are interactive and support pan and zoom. Free version available, and single website license for US$90,-

Getting the data

Before anything can be displayed the data has to come to the user. The most universal way to do so is probably by using JSON. Every front end library supports the JSON format so it will be easy to convert this to any type of chart or display.
Since we want to be able to deploy the server part on both Windows and Linux machines it is best to use a programming language / framework that is natively available on both.
For Windows machines the currently popular route would be to use ASP.NET WebAPI.  It requires the Microsoft stack (.NET, IIS) and is easy to build from Visual Studio. And though it is probably possible to get it running on Linux using Mono that would be much harder.
For a more operating-system agnostic solution it is worth looking at Python and it's myriad of libraries and built-in functions.  Just some Googling for the 'fastest solution to serve JSON from database using Python' brought up several solutions.

The DataSet library does exactly what is needed here. It translates simple calls to the appropriate SQL queries and returns data as JSON objects. A simple abstraction layer removes most direct SQL statements without the necessity for a full ORM model - essentially, databases can be used like a JSON file or NoSQL store.






After setting up the server and my WSGI environment as described, the initial framework of the application is almost ready. The next step would be to retrieve data as generated by the wsgi program from a webpage. The following html page shows the minimal code to do exactly that:

<!DOCTYPE html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.0/jquery.min.js"></script>
<script type="text/javascript">
   function loadData(){
      Data = $.ajax({
      url: "get_data",
      dataType: "text",
      success: function(Data) { $("#status_text").text(Data);}
      }
     );
    }
  
  </script>
</head>
<body>
      <p>
         <button onclick="loadData()"> Get data </button>
      </p>
    <p>
     <div id="status_text">WAITING...</div>
   </p>
      
</body></html>




An Ajax function is defined that just gets the data at url 'get_data'. This function is triggered from the onClick handler of the button. The returned values (just interpreted as plain text in this case) are then shown on the page by replacing the WAITING... text.