Mikael Nilssons Development Blog

My first Android application (actually my second, the first was a commercial one) is released and published. The application is already downloaded more than 1000 times and that's OK for a national application. The name of the application is inherited from the excellent web site Temperatur.nu and is TemperaturNu. Well it's hard to find a god name when you want to implement a excellent idea.

Architecture.
The TemperaturNu application is built with several of the Android OS cornerstones that make up a rigid application. There are the MapView activity, together with a location view, because that's what give this application a twist. Then there are the service, handling data downloading from the web API. Nobody want to handle communication exceptions in the UI, right? I have bad experience in that area from my previous Android application and will not go down that road again!
To join those parts together we need a content provider for data abstraction and a broadcast handler for notification. The data provider also make it easy to handle the asynchronous data download. And...  that's it!

Details.
The TemperaturNu application uses the Android location API to determine your current position. The position is sent in a request to the Temperatur.nu API, and position and temperature for the nearest weather stations is returned together with some other meta data. This result is coded as xml and are parsed by the data download service. When the data is parsed it's sent to the content provider for storage and a broadcast intent is sent. All of this is done in a 'fire and forget' thread in the service that is started in the start server handler. It's essential to run this on a separate thread because if you dot do that the request and parsing will run on the main activity UI thread and that's not what we want.

In the main activity the received broadcast trigger a reload of data from the content provider and update  the items on the map overlay. Several overlays are used in the main activity to handle the display of location and items

The content provider just keep the data in memory because I judged the temperature data to be to  transient to store in a database. The most important role of the content provider is to decouple and abstract the data from the data down-loader.

Temperatur.nu web API
Erik at temperatur.nu have been kind to set up a public API (description here) which the application use for data collection. The temperature data can be requested either by geo position or by a station id. Up to five readings can be requested in each request and the response is coded in xml and structured lika a rss feed. The API can also provide a graph image of the last 24h readings from a station. This feature will be exposed by the next release of temperaturNu.

Passwords everywhere

I’m getting to old to remember all those passwords sprinkled all over the web!  Most embarrassing is that I can’t remember the password to my own private applications, but there is a couple though: local mail server, homeserver user and admin, local wiki and blog, several mail accounts to name a few.

So to remedy this plethora of passwords I have joined the OpenId movement and try to add OpenId login wherever I can. My DasBlog already have it but not my ScrewTurn Wiki wiki. I really like the architecture of ScrewTurn Wiki with dynamically loaded providers and it was fairly simple to write a new user storage provider for OpenId users and adding the DotNetOpenId control to the login page. Unfortunately I have not found a way to avoid changing the application when adding the OpenId login. The alternatives are either to add a new sign-on page with the open id login control or modify the existing login page.  If the login page is modified then I have to apply the patch on every update but I get a direct login link from the main page. Any way the application has to be re-compiled and I lose some of the benefit of loading a provider dynamically. But I have not investigated other solutions so there might be a simple fix for this.

Login page modifications:

Download the DotNetOpenId control from http://code.google.com/p/dotnetopenid/ and set up a reference to it from the web application.
Add the assembly import in the login.aspx page:
<%@ Register Assembly="DotNetOpenId" Namespace="DotNetOpenId.RelyingParty" TagPrefix="cc1" %>

Then the actual login control can be added anywhere on the page.
<hr />
<cc1:OpenIdLogin ID="OpenIdLogin1" runat="server" CssClass="openidLogin"
      RequestEmail="Require"
      RequestNickname="Require"
      RequestFullName="Request"
      RegisterVisible="false"
      RememberMeVisible="True"
      PolicyUrl="~/PrivacyPolicy.aspx" TabIndex="1"
      onfailed="OpenIdLogin1_Failed"
      onloggedin="OpenIdLogin1_LoggedIn"
      onsetuprequired="OpenIdLogin1_SetupRequired"
/>

When using auto registration at least email and nick name shall be required and the full name can be requested. Either the nick name or the full name can be used as display name. The ScrewTurn Wiki username will be derived from the identifier as described below.

Provider

The hardest thing was to adapt the way authorization works when using OpenId. The user shall NOT be authorized when logging in to the application. The user is already authorized; the application shall just set the access rights and eventually register the user.  User data like user name and email is requested in the call to the authorization service and are used to register the user. I used a trick to get the claimed identifier (http://myuser.name.myopenid.com/) down to the OpenId provider. First attempt was to use the identifier as user name but then the user name regular expression complained. Technically the user should not be edited but the name is verified when the user is brought up for group assignment. Anyway I needed a unique identifier for the user and want to use the claimed identifier.

Well the way I resolved it was to use the password parameter instead. When an OpenId user is created the identifier is modified to construct a new username: the http:// and .com part is removed and the dots replaced by spaces. This will give a user name like ‘myuser name myopenid’ and that is unique enough. I chose to automatically register the user if the login fails. The OpenId provider defines two default groups: OpenId_Users and OpenId_Administrators and can be configured to automatically add a new user to one of those groups. Users will be added to the anonymous group if they are not added to any of the default groups. 

Provider configuration:

My primary goal was to enable me to logon as an administrator to my wiki so I added a feature for automatically adding a user to a user group. The provider maintains two predefined groups’ wher users can be added with the following configuration:
addtoadmin=http://myuser.name.myopenid.com
addtouser=http://myuser.name.myopenid.com

Installation:

When the login page have been modified, or a new sign-on page added to the web application it’s time to deploy. Start up the wiki and log on as an administrator to configure the wiki for OpenId access.  Upload the OpenIdProvider.dll to add it to the framework and configure the provider if you want the automatic logon feature (see above).  Then go to the group administration page and set access rights for the two default OpenId groups. That’s it! Logout and go to the login / sign on page and login via OpenId.

Login problems

Sometimes during the tests I have problem to get the sign in to work and gets Login_Failed. The same thing happens on my DasBlog. Usually it works after a login/logout on myopenid. 

Disclaimer

This was not intended to be an industrial grade solution for OpenId implementation on ScrewTurn Wiki but it can be used as a starting point for them who want to add it to their installation.  The code is delivered ‘as is’ without any guaranties regarding security etc.
I got rid of another password to forget and can move on with next project
The OpenId source code and dll can (soon) be downloaded from http://minidev/download/Openidprovider.zip

Well this is a thing that I have been thinking about for a while: I want to use UTC times in the rrdb to be able to handle when time change from summer time (day light saving) to winter time. It must be better to always use UTC time internally and convert on data insert and fetch. That will take care of the exception I get when the clock is set back one hour and I try to insert 'old' data. I will also remove the conversion to and from epoch time and just store seconds from the DateTime class.

The problem is that I need to restructure the code in several places. I have never liked the connection between the classes where one class store a reference to a 'parent' instance like the Archive have a parent RrdDb and the ArcState have a reference to a parent Archive. Then the ArcState access members in the parent.parent instance like Header header = parentArc.getParentDb().getHeader(); I don't think that the ArcState should have any knowledge about the Archive and absolutely not about the relation between Archive and RrdDb!  Why not? Because I plan to use local time outside the RrdDb and UTC time inside and I must be sure that all access to time variables are converted correctly.

Example:

      public long getLastUpdateTime()
      {
         lock (sync)
         {
            return header.getLastUpdateTime();
         }
      }

Here I want to do a conversion to local time but in Archive there are code like this:

      long lastUpdateTime = parentDb.getHeader().getLastUpdateTime();

What happens if someone refactor it to:

      long lastUpdateTime = parentDb.getLastUpdateTime();

Well if I put the local time conversion in RrdDb.getLastUpdateTime() it will generate some nasty bugs that are hard to track!

But according to the text above I don't want the Archive class to have knowledge about the RrdDb to Header relation so the refactoring is correct! Thats why I want to remove all relations to the 'parent' object and supply all needed data as parameters in the method call. Anyway it looks strange to me that a RrdDb instance call into an Archive instance that call back, via it's parent reference, into the RrdDb instance to get some parameter that could have been sent into the call!

I'm not sure how to handle the changes in the Google code repository. I still keep the code in my private SVN repository and I will do the changes on trunk there. I will try to update the Google repository when I'm done.

Rrd4n(et) are public at last. After a long and painful time of ambivalence and un-determination I decided to publish my rrd4n lib at Google code. I hope I made the right thing and selected the right license type. Well time will tell. The project name on Google is rrd4net and not rrd4n because I manage to register rrd4n on my professional Google account.

And now the disclaimers: The software isn’t finished!

Especially the configuration tool need a lot of work until it is usable, but I needed a tool to set up the rrd databases and view the graphs during development. There are several menus that don’t work anymore and it’s not obvious or intuitive to use. There is one nice feature though: you can drag an rrd database from the left tree wiew and drop it in the graph definition area to make a quick test of the data in the register.

The separation of the data storage from the graphing is a bit over complicated. It’s nice to be able configure the data source in the app.config file but I’m not sure if it’s worth the effort. But it was really nice when I wrote a MySql accessor  (not in the Google code repository right now) to feed data from my MySql database into the graph. That experiment also made me realize why RRD is such a good idea: the MySql database contain temperature readings from 4 sensors with a resolution of 1 minute for about 3 years i.e. a huge amount of useless data with high resolution. It takes for ever to get the data out of the database. So now I have to write a program that feed the data from the MySql database into a RRD database.

Well right now I’m planning for an Android project so I will not spend that much time on rrd4n for a while.

Happy Round Robin

I have some plans to make the source code public in the near feature. The original java package was published under GNU Lesser General Public License and as I understand it it's OK if I do the same as long as the files contain the license text. I have to do some more testing and there are some 'technical debt' collected during the development. I think I will put the source code in a Google code repository when I'm ready.

At last, after a lot of refactoring and polishing the configuration tool is finished, or good enough, for now. I added some nice docking features (thanks to WeifenLuo DockPanel Suite) to enable a bit of plug-in behavior (more on that later). The docking view made it possible to show both database properties and graphs in a natural way. And as a frosting on the cake I added a drag and drop feature: a database definition can be dragged into the graph definition text box. The database definition then converted to a graph definition with a DEF declaration and a line definition for each data source in the database, a convenient way to do a quick view of the data in the database.

Database view:

Graph view:

I have spent a some time on my RRD code package. Porting the graphics and formatting, specially the date time formatting, was pretty straight forward but I have spent a lot of time on the configuration tool. I didn't realized that there was a fairly good implementation in Java until I was half way through! But my tool is fully visual (and still a bit buggy) and fully functional for setting up the database, import cvs data and test the graphing.

I have also made some serious refactoring on the core rrdb code, separating the storage part from the actual data access. I assumed that if an asp.net application should be able to access the rrdb files on another computer there could be some security issues! So I created a rrdb file server application where the graph processor could fetch the data by .net remoting (yes I know it's slow and over-complicated but it works). There are also a local file provider so I don't have to run on the server during testing. And naturally the right access provider is injected into the graph processor depending on the configuration file! The service has also a data store interface for data collecting that i plan to use in the data logger.

Talking about the logger it's up and running now. For the moment I collect data from my geo-thermal-heat-exchanger, or heater for short, where I measure compressor run time, hot water production and return water temperature. I need to get some more DS1820 sensors to measure brine temperature as well

Tonight I added the graphing to my rrd4n configuration tool and generated the first graph from the logged data. The logger have been running for almost 48 hours but the photosensor get out of position on the heat exchanger so there are some hour missing in the data. Still it's up and running!

Now I want to put this data on my webserver but I'm not sure how to get access to the RRD file from the web application. And before that I have to get te hardware ready so I can detect when the heat exchanger produce hot water. Some tasks for the weekend. 

It was an exciting night last night when I got the logger running for the first time. I manage to port the rrd4j package to c# in about 20h but then I start to write an administration tool to manage the rrd visually and that took almost as long time as the porting.  Well now I have a visual too that I can use to create and manage my rrd databases. Written with a MVC pattern and using Castle Windsor IOC framework and all!

The data logger was developed in parallel with the porting and in a highly Agile fashion every decision was delayed until the last feasible moment i.e. the model was changed underway. The logger started as a highly generic logger but moved toward a logger for rrd databases with a SQL backing database. The configuration of devices and data channels is stored in a SQL database (SOLite for the moment) and the Logger read this configuration at startup. The logger creates a set of data points from the configuration data and start sample the devices, process the data in a sample processor and persist the result at regular intervals.

Right now the most important thing is to collect data and tune the settings of the rrd data bases. I have to find out how to set tick intervals and what archives needed to store relevant data. Right now I have rrd databases for heat exchange compressor runtime, heat exchange compressor start count and return temperature.  The heat exchange compressor runs for about 20 minutes per hour so I have set the database tick time to 3600. This will give a value around 0.3 in the rrd.

First I wanted to sample the runtime more often, like every 10 minutes, but then there would be several values stored as 1 in the database because the heater would run for the whole sample interval i.e. value overflow. Storing 1 in a ratio value (running seconds/ second) make a MAX archive useless! So I have to be patient and wait for an hour before the firs result show up! Same thing for the start count database, the compressor start 1 to 2 times an hour and the database tick must be set to at least 2 hours to avoid 0 values (making MIN archive useless).  The run time counter store a NaN value at start up and then a 0 value to handle counter restart. The max value is set to 4000 and a counter restart will be detected as a counter wrap. I’m not sure if this behavior I correct thou. The counter restart is a counter wrap and will generate a huge value outside max, and a NaN value is stored in the archive. I have to figure this one out tomorrow.

The data logger project started as a generic data logger for just everything like gasoline usage of my car to outdoor temperature, but has narrowed in to a data source for my RRD databases. The architecture is still flexible and will allow for some extensions in the future (I know it’s over complex but it’s elegant). The components are described by an interface description and the IOC framework (Castle Windsor) takes care of the component injection. Used components are described in the app.config file. This means that I can configure if I want to use the RRD or SQL data repository, or both and what device handlers to use. At the moment there is only an OneWire device handler and I’m planning for a serial handler to read from my old data collector.

The biggest advantage by using configurable components is that it’s possible to mock some components when testing other components. Using a mock device handler enable me to test without the real OneWire network. By using ActiveRecord/NHibernate I can test against an in memory SQLite database without destroying the live database.

Sample processors are used for special handling of the data read from the devices and perform functions for running average filtering, runtime counters and start counters. Running average is used for smoothing of analog signals like temperature, removing the spike in heater output when the heater switch from hot water production to heating. Run time counter detect the digital input that indicate that a device is on, the heat exchange compressor, and count seconds running. The starts counter processor count number of changes from 0 to 1 on a digital input. And then there is the null processor for no processing. 

The device handlers are supposed to be able to scan their devices and report any newly added devices to the configuration database. In the case of the OneWire network this means that the device handler scan the network and report the unique address of each device newly added to the network. The new devices have to be configured by adding ‘data channels’ to them to enable data collection. The channels define the database and data source in the (RRD)database used to store the data together with sample and storage interval. After configuration the logger is restarted, or set to start logging new devices.  At the moment the biggest problem is that there are time intervals all over the place!  The logger runs for the moment every second then the channel have a sample interval, indicating how often the device must be read, a tick interval determine how often the data shall be written to the database. The tick interval must be the same as the tick parameter on the RRD database.  I’m thinking of creating a configuration application to make it easier but this is done very seldom and I’m not sure if it’s worth the effort. But I have to do something for all the intervals!