Hildebrand, the key technology partner of the DEHEMS project has been working with IBM address the issues of traditional databases dealing with time-series data  (i.e. ‘pulses’ of data arriving at regular intervals from one or more sources), because their structure makes it difficult to store and index this data efficiently. As a result, Hildebrand knew that it would be impossible to create a solution around a standard database engine without massive investment in high-end server hardware. The costs of this would be prohibitive, so a new kind of solution was required.  Following research in to leading-edge database platforms Hildebrand looked at the IBM Informix TimeSeries DataBlade and Real-Time Loader, which are specifically designed to handle time-series data.   Working with the software lab at IBM Hursley and they helped run a number of proof-of-concept projects using the facilities in the IBM Hursley Innovation Centre to see if it would be possible to handle the volume of data needed in a cost-effective manner.

The Informix TimeSeries technologies create a single database object for each data-source, and then simply update it with the latest readings whenever a new ‘pulse’ of data arrives. This provides a more manageable data structure, which makes it easier to store, extract and analyse data.

With help from the IBM Hursley team, it was quickly found that Informix TimeSeries could deliver spectacular results. In the first proof-of-concept three million homes were stimulated sending readings once a minute, and was able to capture nearly 40,000 readings per second using only a quad-core, dual-processor Intel server. In the second, using a slightly larger server and it was found it could deliver analytics response times of between one and three seconds for a load of 50,000 readings per second. Without understanding the technical details, energy monitoring for three million homes or more became a practical proposition.

Further details are available in the IBM press release.

The Social Trends report (published July 10) from the UK Office for National Statistics shows that the amount of domestic energy used in homes specifically for lighting and electrical appliances rose from the equivalent of 2.7 million tonnes of oil in 1970 to 6.8 million in 2007 – a 155% increase .  Interestingly the energy used for cooking fell by 41 per cent in the same period.

Comments from National Energy Action said there is a need education about energy efficient appliances with the roll out of smart metering going some way towards behavioural change.