for Mains Power Logging
Arduino Sketches for (Safely) Logging Mains Electricity Consumption
This page covers:
-
Three different Arduino sketches for long-term half-hour logging of mains electricity
-
Methods and sketches to set the correct time for a Real-Time Clock
-
Techniques to help identify the correct operation of loggers through LED flash sequences
-
Writing data to SD Cards and positively identifying data through EEPROM Serial numbers
-
Links to further information about mains data logging
As this project concerns mains electricity (nominally 230V AC in Europe) and potentially high currents,
you must satisfy yourself that you understand and take steps to eliminate risks when working around
the utility meter. You should also check local regulations concerning access around your utility meter.
Three different methods for accurately recording half-hourly mains (230V) electricity consumption are
described below. All Arduino sketches, including setup sketches for the Real-Time Clock and checking
sketches are listed below and can be downloaded from this page as zip files. The three methods described are:
-
Measuring current with a CT and voltage with a VT for true power
-
Monitoring Utility meter LED pulse outputs
-
Monitoring Utility meter Volt-Free pulse contacts
Common Hardware and Software Setup for All Three Logging Methods
All three methods of logging utilise a Real-Time Clock (RTC) and SD Card to record consumption data for every half-hour of the year (17,520 readings). I use the Adafruit Data Logging Shield (Product ID: 1141) - other products are available, but I find this product has been very reliable.
For information about using this shield and libraries see https://learn.adafruit.com/adafruit-data-logger-shield
The above links describe the SD cards that can be used and how they need to be formatted. These sketches do not require very large files unless you modify them to save extra information or to log data at a higher frequency. The largest file size associated with these sketches as presented, will be less than 650 kBytes for a year of recording. Reliable, second-hand, lower capacity SD cards can be purchased for less than £1.
All three of the data logging techniques require the RTC to be set so that
data is accurately recorded. All three will use the same file writing
format. To help identify the loggers a unique identifier is stored
in the Arduino EEPROM (Electrically Erasable Programmable Read-Only Memory -
non-volatile memory).
The process of setting up each board and data logging shield is:
The process of setting up each board and data logging shield is:
-
Edit SetRTCandSerialNumber sketch to set a unique identifier (for EEPROM)
-
Set a RTC time in the sketch to be a minute or so in the future
using the
rtc.adjust( DateTime( yy, MM, dd, hh, mm, ss) ); method in the sketch
-
Insert a new CR1220 button cell to maintain the RTC time
when the Arduino power is off
-
Compile and upload the sketch to the Arduino (with shield)
-
Wait until 2 seconds before the time set during compile
then press the board reset button - this sets the time in the RTC
to the be the time specified - do not turn off or reset
after doing this
-
Overwrite the sketch with TestLoggerSetup sketch to prevent the
time from being reset again and to check that the setup worked
Testing the Arduino Logger RTC, SD Card and Unique Identifier
Following the setup, the TestLoggerSetup sketch is compiled and loaded before
the Arduino is reset by a button press or power cycling. The test sketch does
the following:
The code for the TestLoggerSetup sketch is shown below:
-
Checks for the presence and operating status of the RTC
and SD card
-
Finds the unique identifier in EEPROM and constructs a
file name in the same way that the actual logging sketches will do
-
Opens the file for logging and writes a header - then
shows the same information to the user via the serial monitor
-
Enters a 5 second loop writing data to the file and printing
out the time from the RTC on the serial port so that the user
can check that the correct time was set
-
After running the monitor for a while, the SD card can be checked
to make sure the correct data is being written to the card
-
As the test routine does not reset the RTC the sketch can remain
loaded until you are ready to upload a logging sketch and should
keep the correct time even when turned off
The code for the TestLoggerSetup sketch is shown below:
TestLoggerSetup - output to Arduino IDE Serial Monitor
At the start of the output the status of the RTC and SD Card are shown - both OK here.
The serial number is shown (REDF18-004) and the filename is shown. The string written to the file header is shown to prove the correct start time. The actual time (from the RTC) is then printed every 5 seconds.
This screen shot was taken when the clock showed 10:35 on the 21/11/18, showing the correct timestamp. The analogue clock shown is freely available from this website as a Java runtime file (.jar) or as a zipped file for the Java/Netbeans IDE.
This Arduino (+Data Logging Shield) is now ready to be programmed as a logger. The test sketch may be left in the Arduino until a logging application is chosen and the RTC should maintain the correct time whether powered-up or not - try it!
Logging Sketch #1: Sampled Voltage and Current Waveforms
This sketch samples the current and voltage waveforms at high frequency to calculate the true power and RMS Voltage. This is a true power measurement rather than the indicative measurement based only on current that some clip-on monitors provide.
Current is measured with a Current Transformer (CT) and voltage is measured with a plug-in Voltage Transformer (VT). The two AC waveforms are sampled at a frequency much higher than the mains (50Hz in Europe) and the voltage and current are multiplied to give the instantaneous power at each sample. The sum of the instantaneous power over full cycles gives the true 'real' power. A more detailed description of power monitoring with CT/VT and the way that signals are processed can be found on the OpenEnergyMonitor.org website.
The CT is shown clipped round one of the two thick wires from the supply isolation switch (not always present) to the consumer unit. There is no need to identify positive or negative or to put the CT in the right orientation as the result can be sorted later if current is measured as negative. As this is a clip-on device there is no requirement to cut into supply wires.
The voltage transformer (VT) is more familiar and is like the charger plugs from a few years
ago before switched-mode power supplies (SMPS) became the norm. In this case,
we want a low-voltage AC output to multiply with the CT waveform rather than a rectified DC output.
A suitable plug-in transformer is shown below (230V to 9V AC), plugged-in next
to a SMPS supplying the 5V DC supply for the Arduino. You could rectify the 9V AC to also supply
the Arduino via a rectifier, but SMPS are so cheap this is not worth while! Note that the 9V RMS
output of the VT is a nominal rating under load, so a higher RMS voltage is likely when monitored
with very little load.
The output of the CT and VT are sampled by the Arduino from signals received via an Emon Tx
shield from OpenEnergyMonitor.org - see
https://openenergymonitor.com/emontx-arduino-shield-smt/.
This shield can take up to 4 CT inputs but only one VT input, so the sketch could be extended
to monitor four different circuits leaving the consumer unit - I use the second CT position
which is linked to analogue input pin 2. The Emon Tx shield is shown below:
Combined with an Arduino, the Emon library allows you to extract
various parameters from the monitored circuit(s) including voltage,
current, real-power, power-factor and reactive-power. A full description
of the shield and Arduino libraries and example sketches can be found on
the OpenEnergMonitor.org website at
https://wiki.openenergymonitor.org/index.php/EmonTx_Arduino_Shield .
All the hard work is done for us - we simply call the methods to get the parameters we want!
The Arduino Uno, Data Logging Shield and Emon Tx combine to form the data logger (see left). The CT input is plugged in on the left and the AC voltage at the front of the Emon Tx board. The DC 5V power supply plugged into the Arduino Uno board on the right at the bottom of the assembly. The Data Logging shield is sandwiched between the Arduino Uno and the Emon Tx Shield. For my application, the complete assembly is attached to the lid of a project box, which when assembled includes ports for the CT, AC Volts and DC supply. It also contains a viewing port to be able to see the two LED's to the right of the reset button on the Data Logging shield (front RHS) which let the user know if the logger is working.
Calibrating the CT and VT
As the evaluation of power and voltage depends on the accuracy of the CT and VT, and to some
extent on the tolerance of resistors on the Shield, there is a requirement to set calibration
parameters in the sketch. There is good advice for doing this at
https://learn.openenergymonitor.org/electricity-monitoring/ctac/calibration.
The calibration constants are set in the Setup routine below as follows:
- Voltage calibration: emon1.voltage( 0, 276.85, 1.7 );
- Current calibration: emon1.current( 2, 60.61 );
Although you should follow the calibration procedures, this may be too onerous for multiple logger configurations. As long as the calibration is reasonably correct for a collection of CT's and VT's of a particular make, any small errors can be compensated in the analysis. To do this, I recommend taking a utility meter reading when the logger is installed, when it is removed (or the SD card is removed), and also taking some intermittent readings at recorded times. These readings can then be compared to the logged data to adjust it accordingly; from past experience the variation in total consumption recorded between the meter readings and logger data is rarely more than 5% different - it is reasonable to assume that the error is linear so the logged data can be scaled to match the meter readings whilst maintaining the profile shape.
The installed logger (in a project box) is shown in a meter cupboard. Note two local mains sockets are required to plug in the AC-AC voltage transformer and the SMPS for the Arduino and these must come from the consumer unit with suitable MCB/RCD protection. The CT can be seen clipped over one of the wires between the isolation switch and consumer unit (not shown - inside the house). The meter cupboard shows the usual, damp, mould and spider's webs, so protecting the logger in a project box is essential!
The LogPowerToSD Sketch
The sketch to record the total Wh and average voltage is relatively straightforward now that
the RTC and SD Card writing have been introduced. After initial setup of the Emon library
via the methods described above, the actual data is retrieved via a simple method calcVI,
which samples the waveforms over a specified number of half cycles (20 used) to calculate
various properties, from which we only want the Real Power and RMS Voltage. Loggers are likely to be in position for long periods of time, so it is essential that all possible steps are taken to make sure they are correctly setup and working at the outset. To do this I use a sequence of LED signals; a sequence of flashes to show that the setup went OK, or constantly lit if there is a problem. Throughout the operation, LEDs flash once to show a sample every 5 seconds and a sequence of flashes shows that data has been written to the SD Card every 30 minutes. This considerably reduces the likelihood of simple errors such as forgetting to push the SD Card into the slot.
The sketch operation is described in the following steps:
-
Setup the Emon parameters by passing calibration information to the library
-
Retrieve the logger serial number and check that the RTC and SD Card are present and
working - if not, halt the setup and show constantly lit LEDs
-
Construct the filename and file header and write these to the SD Card
-
Indicate successful setup to installer with a sequence of flashing LEDs
-
Every five seconds, collect the real power and RMS voltage, and sum for total
Wh (positive and negative) and average voltage - flash LED to show sample taken
-
Check to see if a half-hour is completed, and if so complete calculation of
total Wh and average voltage and write to SD Card file with timestamp
-
If the SD Card write is successful, show a sequence of flashing LEDs, otherwise
leave LEDs on constantly to indicate a fault
Data Recorded and Analysis
Analysis of the output of the logged data will be covered in detail in a separate page/article, but you can download a raw (unprocessed) recording made over one year using LogPowerToSD with logger 013 - POWER013.CSV. The first few lines of this file are shown to the left. The file contains single half-hour entries per row and has several restart header blocks due to power outages and some missing and dubious data - typical for a year of data. It was recorded in GMT rather than local time, but analysis has to take account of changes from and to British Summertime on Sunday 29th October 2017 and Sunday 25th March 2018 respectively. The columns are date, time, 'import', 'export', average voltage. In this case, we only had import (buying from the network), so the fact that it appears in the 'export' column means the CT was connected to show negative current for import. This does not affect the result and if there had been some real export (from a PV array for example), the net export per half-hour would have appeared in the third column. Note that you can have both import and export during a half-hourly period but not during one sample period.
The size of file generated by this logging technique is fairly modest at a maximum of 36 Bytes per reading. One year of readings will therefore have a file size of around 620 kBytes. The POWER013.csv file contains more than a year of data, but the 'import' column is all zeros, giving a file size of 644 kBytes for around 14 months of logging. Large SD cards are not required unless more data is recorded or a shorter logging period is chosen.
It is not my intention to describe in detail the analysis techniques that can be applied to this type of data here, but by way of an example, the average daily demand (across all days of the week) divided into Autumn, Early-Winter Late-Winter (after December), Spring and Summer for this property are shown below. The chart displays characteristics for local time so that time-of-day correlated events are aligned across the seasons. This is derived from the POWER013.csv raw data file.
These characteristic show the 'likelihood' of demand (average of similar half-hours over a number of weeks) divided into different seasons of the year for a single property. Normally we are concerned with the average of a group of properties for cost allocation of electricity consumption or network design purposes, but even with data for only one property you can clearly see the strong relationship to the time the household gets up (alarm at 6:30), cooks meals and goes to bed. You can also see that overall daily demand is related to season.
More detailed analysis can be performed and includes the difference between work-days and weekends, effect of temperature (degree-day analysis), relationship to sunrise and sunset, and other factors. Equally important for more detailed analysis is the exclusion of special event data such as holidays, extreme weather events, etc. To get to the stage above requires the reorganisation of data into days (Excel macro) as rows of 48 values, elimination of 'bad' and missing data, marking of special days (Christmas, holidays, etc.) and analysis to identify outliers; for this example I used a simple sinusoidal model of annual demand (per day) and identified 'special events' as days where demand varied by more than 2 standard deviations (+/-2*σ ) from the model; if the error is approximately normally distributed from the model, this should cover >95% of all the data, such that the remaining <5% are likely to be outliers. The daily demand (sum of Wh/half-hour over each day) is shown below with the best sinusoidal model fit and +/-2*σ ranges.
Two holiday periods can clearly be seen in September 2017 and August 2018, and the few other points identified show weather extremes, power outages and excessive use of washing machine after a holiday! I will write more about this type of analysis soon.
Logging Sketch #2: Logging Meter LED Pulse output
The pulse rate (pulses per kWh) is given on the meter nameplate - see below:
The nameplate data tells us that this meter generates 3,200 pulses per kWh - a lot! Every time a pulse is picked-up another 0.3125 Wh of energy has been registered by the meter. To put this in context, if there was a constant 1kW load in the house, then the LED would flash every 1.125 Seconds (3,600 Seconds / 3,200 pulses). The supply is rated for 100Amps at 230V, which corresponds to a maximum of 23kW, so in the most extreme case we would see a pulse every 156.5 mS, so the sketch needs to take account of this, although in reality it is unlikely that you would get more than 10kW of demand unless an electric shower was in use (+ kettle, etc.).
To detect the LED pulses I used a low cost TSL257-LF Light to Voltage Converter. These are readily available for less than £3. Given a 5V supply, the output gives a voltage proportional to light intensity. The actual sensor and its pin configuration are shown to the right. The supply is taken directly from the Arduino 5V and GND pins, and for this sketch I connected the sensor output to analogue input channel 1. For a sensor mounted directly over the meter LED without any other light sources, the pulse can be counted by looking for the transition of the analogue input channel voltage going 'high' - I define anything above 3.5V as being 'high' - indicating that the LED is lit.
The PulseLogtoSDOptic Sketch
This sketch uses many of the components seen in LogPowerToSD described above. The main differences are found in the 'Loop' routine where rather than taking a reading every few seconds, we continually 'poll' for pulses detected by the analogue voltage going 'high'. When a pulse is detected, we also need to be sure that this is a new pulse and not the last pulse still illuminated. Pulses are counted to give the accumulated energy. As in the LogPowerToSD sketch, the time is then checked to see if a half-hour is completed and needs writing to the file on the SD Card. The sequence is described in the steps below:
-
Retrieve the logger serial number and check that the RTC and SD Card are present and
working - if not, halt the setup and show constantly lit LEDs
-
Construct the filename and file header and write these to the SD Card
-
Indicate successful setup to installer with a sequence of flashing LEDs
-
Continually poll analogue channel 1 for a high input indicating a pulse
-
If the input is high and this is the first time a high has been seen (i.e. the
last input was low), then increment the pulse counter
-
Check to see whether a half-hour has elapsed, and if so write the number of
pulses received to the SD Card with a timestamp and reset
the pulse counter ready for the next half-hour
-
If the SD Card write is successful, show a sequence of flashing LEDs, otherwise
leave LEDs on constantly to indicate a fault
The PulseLogtoSDOptic sketch is listed below and can be downloaded as a zip file from the bottom bar of the listing panel.
Analysis of Pulse Data from the PulseLogtoSDOptic Sketch
A short section of a file written by PulseLogtoSDOptic is shown to the left. The actual data retrieved from the logger is shown in the first three columns; day, time and pulses detected in the half hour ending at this time. The fourth column is the conversion made in Excel to get Wh/half-hour, by dividing the number of pulses by 3,200 in this example (see meter above). The analysis of data when it is in Wh/half-hour format is the same as it is for data retrieved from LogPowerToSD, except that you do not have voltage information, and with the exception of very occasional lost pulses during file writing, the logged kWh total should match the quantities retrieved from the utility meter. The file size is up to 23 bytes per entry, so one year should occupy less than 400 kBytes.
Reliably attaching the Light to Voltage sensor over the meter LED without any other light sources getting in and causing false readings has been problematical! A soft, preferably shielded, two-core (+shield for GND) cable is required to connect the sensor to the Arduino + Data Logging Shield. Stiff cable will tend to put forces on the sensor which will eventually cause tape (even the best gaffer tape) to pull away from the meter. I used stiff black foam over the sensor to hold it in place and block out other light sources. This was then taped to the meter.
Although the pulse output should give exactly the same total reading as the meter, possible errors will occur if the logger is not turned on immediately after a half hour ends (to get the complete following half-hour without loss) or if the meter is read part way thorough a half-hour, in addition to very small possible errors due to pulses being missed during SD Card writes. Clearly, any ingress of light may also affect the reading, either by giving false high readings or by holding the reading high so that new pulses are missed.
Logging Sketch #3: Logging Meter Volt-Free Pulse Output
The third method of logging uses 'volt-free' contacts available on some utility meters to provide the same pulse data as the LED for import and sometimes for export as well (2 sets of contacts). By describing these as 'volt-free', I am indicating that the meter contacts are completely isolated from the mains power in the meter, so to register a pulse the contacts are connected between Arduino digital channels with pull-up resistors enabled and GND - see pinMode( importChannel, INPUT_PULLUP ) in the Setup routine below. When the volt-free contact closes to indicate a pulse, this pulls the digital input from 'high' to 'low' state - the opposite of the LED based sketch.
A sub-meter with terminals exposed. The pulse outputs are indicated with the red-arrow between the live and neutral load terminals.
Typical utility owned meter showing sealed terminals which cannot be opened, therefore this technique cannot be used.
The three connections for pulse logging inside this particular meter are difficult to make as
the second and third wires obscure the first and second screw terminals respectively! Ideally
the connecting wire should be shielded two-core, where the shield is connected to the common
pulse logging connector at the meter and the GND pin on the Arduino.
The two wires from the pulse connectors go to Arduino digital I/O Pins
6 and 7 for import and export respectively. See notes above - when one of the volt-free contacts
closes, the state of the corresponding digital input changes from 'high' to 'low'. Operation
of the logger can be checked by simply shorting either pin to GND to create a pulse.
The PulseLogtoSDVoltFree Sketch
The PulseLogtoSDVoltFree sketch is very similar to the PulesLogtoSDOptic sketch. The main differences being that pulses are detected as a falling level on digital channels 6 and 7, i.e. when a change from 'high' to 'low' is detected. There is also a mechanism to make sure we only measure each pulse once by detecting whether the last state was also 'low'. Two channels are detected in this sketch as the meter (for sub-metering) had both import and export contacts to show micro-generation export. one other minor difference is that a 50ms 'flash' and debounce delay is built into this sketch whereas a shorter period was used for the optical logging - this is due to the lower pulse rate associated with the meter concerned where each pulse represented 1Wh - a shorter flash period could be used if this sketch was used with a meter with a higher pulse rate. It is noted in the sketch comments that the flash/debounce period will not cause pulses to be missed from the other channel as it is not possible to have import and export occurring simultaneously.
As the operation of this sketch is very similar to the last, no further introduction is required - see the listing below:
Analysis of Pulse Data from the PulseLogtoSDVotFree Sketch
The data file recorded by PulseLogtoSDVoltFree is very similar to that from the LED pulse recording, except that a further column is included for 'export'. In reality this will only record 'export' if the correct contactor is chosen, but otherwise can be sorted during analysis - in this case the import and export are in the correct columns and as expected there is no export as there was no local generation associated with this logging.
Note that where there is local generation (PV for example) you can get both import and export recorded during a half-hour period. With some load balancing technologies you could also get very low import when the local generator is working; for example where a 'dump' to an immersion heater is used to take excess PV output.
This file was recorded at the same time as logging by both of the previous techniques so that the outputs could be compared. After a minor correction of the VT/CT profile against meter readings (5% below), the three profiles are essentially identical and both of the pulse logging files agree with the meter readings. To show that all three methods produce the same profiles, see the chart below but note that a 'stacked' chart had to be used as the three characteristics otherwise over-write one another.
Potential Improvements/Extensions to Logging Sketches
The three sketches presented above are primarily aimed at getting accurate half-hourly electricity consumption data. There are a number of potential extensions to these sketches, depending on the type used. These include:
-
Record additional parameters such as the highest and lowest power in each half-hour or
average current, reactive power, etc. (CT/VT method)
-
Detect a lack of data over time as being a failure which should be indicated by setting LEDs
to 'high' so that installers can spot problems in the field
-
For short-term logging you could log at higher rates than every 30-minutes - logging at
one minute or 30 second rates will identify the contribution of short-term loads like
kettles and the profile shape of fridges and freezers for example, but file size and
more complex analysis need to be considered (one minute recording is approx 18 GBytes pa)
-
All three types can take additional channels of data - additional CT's, extra pulse
inputs
-
For short-term logging, the Arduino internal clock could be used for the timestamp
although a recording mechanism would still be required
-
Instead of recording locally, data could be transmitted to a central unit with one of
the various communications techniques available as Arduino shields or modules; simple serial,
433MHz radio (very low cost) or Wifi, although transmission from around the meter can
sometimes be difficult
-
Estimate local impedance and voltage profile from change of voltage with load (CT/VT method)
-
Add a display to show instantaneous power and to indicate the logger status (low cost LCD)
-
Log other devices with pulse outputs or individual circuits with CT/VT methods
-
Use interupt driven recording for pulse logging
