Rogue sensor values in GivTCP

Hi Folks

I am using GivTCP 3.0.3 and for the post part it works well and serves my needs which is mainly predbat. However, I noticed that my Energy dashboard shows some obviously rogue data and I started to trace this back to the GivTCP sensors where I have found at least one sensor giving silly values.

The entity state graph for sensor "GivTCP2 Energy Battery Charge Energy Today kWh" looked fine yesterday and dutifully reset to zero at midnight, but at 00:30, it spiked from 0 to 21.6 where it remained until 03:55. At 04:00, the graph drops to 0.3.

The stats data showed 21.6 at 00:30 and -21.3 at 04:00 and I have adjusted both these values to 0 and 0.3 respectively which has adjusted the Energy dashboard values, but I still have some issues, so there must be other sensor data that is in error

I would post images if I knew how to - is there any howto guide on using this forum

After examining all the sensors that cont4ribute to the energy dashboard I found 3 or 4 rogue values at either 00:00, 00:30 and 04:00

I was so keen to fix the data that I did not bother to properly document what and when I had changed, I was all a matter of locating the sensor data for the timeframe where I saw silly data and setting the datapoint to zero from its obviously rogue value.

As I am sure this will happen again, I will make sure to document things better next time

I have looked back a few more days and found several more instances where the sensor "GivTCP2 Energy Battery Charge Energy Today kWh" has implausible values. I have found these after clicking on "outliers" in the developer options -> statistics panel for this sensor.

I realise that there were some "outliers" which were not obviously implausible, just a little higher than expected, so I only set the true outliers to zero

Weasel there’s a couple of givtcp issues raised on this https://github.com/britkat1980/giv_tcp/issues/206 and https://github.com/britkat1980/giv_tcp/issues/237

I have found examples of this in my own energy dashboard, some going back a long way

suggestions so far are to turn MQTT retain on in the givtcp settings, it wasn’t on so I turned it on in mine

another suggestion is to use the total figures in the energy dashboard not the today figures. I haven’t tried that one yet

Thanks, I will look at the MQTT retain setting.

I was considering looking at the filtering capabilities of HA to construct a band-pass filter that would knock out the spikes. The resultant filtered value could then be used by the energy dashboard

geoffreycoan
My GivTCP had "MQTT_Retain": true in allsettings.json, so I will try the Total figures in the dashboard

I couldn't get along with the Total figures and I also discovered that my battery-stack sensor data also had the same type of errors.

I guess I can look at filters, but at least I am not alone and others are experiencing similar

I have created a couple of filtered entities with the following YAML in sensors.yaml

- platform: filter
  name: "Filtered GivTCP2 Battery Charge Today kWh"
  entity_id: sensor.givtcp2_XXXXXXXXXX_battery_charge_energy_today_kwh
  filters:
    - filter: outlier
      window_size: 4
      radius: 2.0
- platform: filter
  name: "Filtered GivTCP2 Battery Discharge Today kWh"
  entity_id: sensor.givtcp2_XXXXXXXXXX_battery_discharge_energy_today_kwh
  filters:
    - filter: outlier
      window_size: 4
      radius: 2.0

where XXXXXXXXXX is my AIO serial number and I added the following line into configuration.yaml

sensor: !include sensors.yaml

After sanity checking the configuration, I restarted HA without issue and I can see the two new entities and they have sensible values.

Tomorrow, I will change my dashboard to use the new entities and let you know what happens

Weasel I’ve added the total charge/discharge kwh sensors alongside the today sensors to my Energy dashboard and have found it works well. There are spikes about once a month from December to April, but none after then and using total also has the advantage of filling in gaps in my data when the connection from givtcp to the inverter has been mushed up. By comparison I think there’s more gaps and spikes in the today sensors.

I plan to fix these Total data spikes with developer tools / statistics and then assuming no other untoward issues I will cut over to using them instead of the Today sensors.

A filter is a good way to go if you can’t find a way to get the sensors working consistently, it does kill any history in the energy dashboard you have though unless you export and import the old history into long term statistics for the new sensor. I’ve been adding a few filters to my own HA to deal with exceptionally noisy power monitors that generate tons of state changes (more on this topic to come in a separate post).

I think you will want to increase the radius to the max charge/discharge rate of your AIO as otherwise you could filter out valid data when the AIO starts charging/discharging

geoffreycoan Looking at the data points on the two sensors I have filtered, they are in the range 0 to 0.4 for 5 minute intervals. This seems OK since my inverter is limited to 3.6kW when connected to the grid and 6kW when off-grid. Therefore a radius of 2.0 is plenty and the rogue data points have historically always been much larger.

I realise that I will be losing the history and will investigate export and import over the next few days or so.

Weasel of course, its cumulative kWh not instantaneous power which is what I was thinking of.

Actually a radius of 2 might be far too big as you are comparing the radius over the previous 4 (window_size) sensor values. GivTCP typically polls every 20-30 seconds depending on what you have it set to, so you’re only going to be measuring the delta change over a minute or two not 5 minutes.

Anyway, regardless, it should do the job at filtering out the rogue values.

In the filters I have created I have set a unique_id for them in the sensor definition so I can configure them in the UI, and once I was happy with them I have excluded the ‘raw’ values from being stored in the HA database with a recorder exclude clause in configuration.yaml just to prevent doubling up on the data storage.

geoffreycoan Saving on storage might be a wise idea, but as I only have 2 filtered sensors at the moment, and probably only a handful when I am done, I might not bother

Weasel you would be amazed at how much guff gets stored in the HA database. I mention it because I am currently on a quest to reduce my database size and have to about 60% of the original size so far. I’m writing up what I do and will share it when finished, I have a lot better insight as to what is in the database and why its there as a result of this.

I think I have narrowed down the problem with my energy dashboard. At midnight, the five sensors that feed my dashboard are being populated with incorrect, negative values. I am quickly able to manually adjust them based upon the last recorded value in the graph at 23:55 and the value recorded at 00:00 Here is an exampe of my findings and adjustments:

• GivTCP2 Energy PV Energy Today kWh = -8.4 - set to zero because previous day's last value was 8.4
• GivTCP2 Energy Battery Charge Energy Today kWh = -17.7 - set to zero because previous day's last value was 17.7
• GivTCP2 Energy Battery Discharge Energy Today kWh = -18.8 - set to 0.2 because previous day's last value was 19.0
• GivTCP2 Energy Import Energy Today kWh = -25.7 - set to 0.1 because previous day's last value was 25.8
• GivTCP2 Energy Export Energy Today kWh = -0.9 - set to zero because previous day's last value was 0.9

I think the root cause is down to how the value is being reset at midnight. The values in the statistics are absolute values, but the graph is cumulative throughout the day, so a set of 0.1 entries at 5-minute intervals will add up to 28.8 in a day. At 00:00, a value of -28.7 will be recorded. This is -28.8 to zero the counter, plus the 0.1 for this period.

Whether this is a bug in GivTCP3 or simply a timing issue on my HA server, I cannot determine and if you ave any suggestions to save me the 5-minutes each day resetting the sensor data, I would appreciate it

Weasel Whether this is a bug in GivTCP3 or simply a timing issue on my HA server, I cannot determine and if you ave any suggestions to save me the 5-minutes each day resetting the sensor data, I would appreciate it

Other people have reported similar issues on the givtcp issue log. You also get rogue values when Home Assistant is rebooted.

Swap to using the Total values in the dashboard instead of the today values, that's what I have done for the battery charge and discharge and that fixed most of my issues. It is not an issue specifically with givtcp3, it happens but less frequently with givtcp2.

You shouldn't lose any data in the dashboard when you do this.

At the moment I am still using the today values for solar, import and export, but if I need to I will swap to the total sensors for those although will be a pain as I have different tariff sensors setup on my import and export today utility meters so I will have to recreate all the history for those

geoffreycoan Thanks - I will try that. I am not bothered about history, being a newbie to HA and the energy dashboard is a nice-to-have rather than a must-have

geoffreycoan I have switched the sensors. It took me a wee while to figure out the correct ones because GivTCP3 has them under "Battery Stack x", where x=1 in my case 🙂 Of course I could add another battery system in the dashboard if I had one.

The Energy dashboard looks no different, so I think the changes are good and I now understand why its better to use values for Total rather than Today. The individual point values are identical, but without the daily reset and the associated negative delta at midnight

I wait with bated breath for midnight tonight 🙂

Weasel and today, there are no rogue values

Thank you for pointing me at the total value counters to resolve this