Download Data in wrong columns

14 comments started 2024-02-27 last 2024-08-06
Cards - Energy GraphCards - Energy FlowCloud Portal
D
#1 Dave M

Since the change to the portal yesterday (26th Feb), the export function from the ENERGY card is providing the half hourly data in the wrong columns.
The field titles and the TOTAL rows match but it is the half-hourly data that is in the incorrect columns.

The headings and TOTAL rows show this order:
Solar To Home, Grid To Home, Battery To Home, Solar To Battery, Solar To Grid, Grid To Battery, Battery To Grid

However, the actual raw half hourly data is in the following order:
Solar To Home, Solar To Battery, Solar to Grid, Grid To Home, Grid To Battery, Battery To Home, Battery To Grid

To see this:
1) Download a daily CSV export from the Energy Card
2) Sum the half hourly data in each column
3) Compare to the totals row - they do not match

I worked out the correct order by looking at the various graphs and comparing to the data - plus a rather obvious high solar import in the middle of the night!

I have raised a Feedback Bug Report for this as I suspect many people who use the CSV data in any way without error checking could find their data very wrong.

J
#2 JonC

I am having the same problem.
I use the daily download and feed it into Excel PowerQuery to feed a set of charts and graphs. This was disrupted earlier in the month when the download format changed from an XLSX spreadsheet to a CSV file - this caused some problems but could be managed by changing the Sheet Name and saving the file as an Xlsx file.
The change that happened on 26th I'm still grappling with - so DaveM's post was a reassurance that I'm not going mad with solar generation at midnight!!
I can understand why there may sometimes be a need to make changes to the data provided but wish there was some way to be notified of impending changes so that they can be planned for. On this occasion though it seems that changes have been made incorrectly with the data structure inconsistent with the column headings - which means that at some point one or the other will be corrected and we'll again be left to our own devices to figure what is changed and how much duff data we've already collected and tried to analyse.
Hopefully some explanation and plan will be forthcoming from GivEnergy.

D
#3 Dave M

I raised this by email with the support desk and this is the response (verbatim) that I received:

Thank you for raising this, we are aware of this and this feature is due to be removed soon due to inaccuracies in the calculated data, for accurate data please use the "meter data" in my inverters > meter data.

The meter data they refer to is the 5 minute data sent back to the server but it is not in 30 minute blocks, only goes to 1 decimal place and is cumulative rather than just for that 5 min block. I'm yet to work out how best to use it.
The download from the Energy card whilst maybe not 100% accurate was at least simple to use.

I don't particularly want to go down the HA route as that seems overkill for my simple need.

V
#4 Vestas

Dave M ...and how does one download the "meter data"? I see no download icon in Firefox, Edge or Chrome....

Edit - also the meter data ISN'T accurate. Ours is over 10% out when compared with the generation meter and the smart meter.

D
#5 Dave M

I found by chance that you could download meter data by clicking on the "Inverter Data" button (9-box-square) at the top right. This actually provides more columns of data when downloaded - although my previous points still apply.

I would agree the meter data is not 100% accurate - but then I presume it is that data which forms the basis for the existing downloadable data anyway.

V
#6 Vestas

Dave M Thanks. Whoever is in charge of portal design needs retraining as their job is clearly too challenging for them.

Edit - oh FFS it only downloads one day's data! This is beyond stupid....

D
#8 Dave M

SteveCook Thanks. I'll definitely take a look over the next few days.

V
#9 Vestas

SteveCook I'm using it for the same reason you did - but I'll just go to the ombudsman asap if they refuse to accept the figures 😉

Edit - I mean Octopus actually INSTALL GE kit so I think they'll struggle....

D
#10 DD

Vestas you can download the data through the API. One URL can get today's meter data, or you can get the full data dump for any day.

On my inverter, the accumulated data for battery in/out is nonsense, though hopefully each day's values are okay.

J
#11 JonC

JonC Having spent some more time looking at this I'm even more confused as to why the data is being issued in the was it is as the fix looks to be perfectly straightforward.

Up until the 21st January the download file from the Energy Graph Page was a single worksheet .XLSX file with the worksheet called 'Sheet 1'.

From 22nd January it became a .CSV file with a worksheet named 'Energy Flows Data {Date}' . Other than tah the data seemed to remain as it was before in the same sequence:
Date: Solar to Home: Sol;ar to Battery: Solar to Grid: Grid to Home: Grid to Battery: Battery to Home: Battery to Grid
I was still able to do my analysis using power Query by just changing the sheet name and saving that as a .xlsx file.

On 26th February, all the data seemed to remain in exactly the same sequence but the headings row was changed and the totals row was changed to be in a different sequence - but the data remained in the old sequence.
I was able to overcome this by just re-juggling the column headings to the old sequence and doing a sum of the data in each column and overwriting the juggled up column totals. I still had to change the sheet name back to the old form of 'Sheet 1' before saving it as a .xlsx file and all the Power Query calculations still worked. Phew..

I don't understand why putting the column headings back in the old sequence is not a fairly trivial change for GivEnergy. The data still seems to be as accurate as it ever was (i.e. not exactly the same as the meter data down to 2 decimal places but good enough for what I was trying to compare to within a couple of pence per day.

The idea that they may 'remove the feature because of inaccuracies in the calculated data' seems to be akin to suggesting they'll shoot a dog because it was sick once after being fed bad food. All that seems to be needed is to put the headings row and the totals row back to the original sequence and the problem would be solved.

Or am I missing something?

T
#12 Typo

I do wish I'd seen this thread earlier - I've just spent the last hour or two trying to get my head around data columns & charts. GE - please fix!

The bottom line, for the hard of thinking, is that the chart figures are still correct, yes?

C
#13 CCourtney

Typo Yes the chart figures are "correct"

P
#14 philipdawsonladds

Me too - wrestled with the data and chart to finally discover that the data is in the wrong column!! This must be easy to fix GE.