Pulling Givenergy cloud data to local database

I'm wanting to pull the data for my plant from the cloud to Influxdb. I can download each day for each inverter from the portal (Dashboard - Inverter Data - Download). That's fine. Once I've got the data where I want it, I'd like to automate the updates to occur a couple of times each day rather than a manual download and import.

I'm already populating Influxdb from various home automation/manufacturer's APIs using Node-RED. I can download the inverter data for any date/period I need using the Get Data Points end point. So far so good. In mapping the dashboard download data to the Get Data Points endpoint output, there's additional data in the former, missing from the latter. The data points missing via the API relate to Inverter AC Energy Output, Hours of Operation, Battery Voltage and so on.

Does anyone have a technique/tool to automate the extraction of these data points from the portal? Automating the data transfer is more important to me than maintaining a complete set, but if there isn't a simple way of doing this, I'll just drop the extra fields available from the portal and use Node-RED and the API. Thanks.

one option is to use givtcp https://github.com/GivEnergy/giv_tcp which will connect locally to the inverter, read the registers (data) and then make it available either via MQTT and / or to influx

I'll probably get a slap for this, but there is technically a second "internal API" that the Web portal uses.
Should you wish to create a script that essentially mimics the login and gets the Web based token you can yank anything the portal shows (including your Solcast predictions and so on)
You can see the calls it makes by simply watching what the portal is up to in your browser (cntrl + Shift × i, then go to the network tab)
Strings are along the lines of .../userID/1 to I think 15 from memory that cover all the different cards and pages)
Others are ...internal-api/something

You should in theory be able to capture the string it issues for the csv download and mimic it with whatever new parameters you want each day (which would presumably just be changing the date)

hoggy Cheers mate 😃 I remember you mentioning this before but had forgotten. That actually makes things simpler as I just need to query the Influxdb to find the latest date downloaded, then download the subsequent day (if it's not today) and import the csv. It means I can write one flow for both initial data load and ongoing updates. This is the process I use to pull 30 minute, 1 hour, daily and weekly electricity consumption from the Smart meter via the Glow API for Influxdb.

locked Thanks for the suggestion. I do use Givtcp but it would rely on the Pi running 24x7 which it should but there are times when I take it offline for various reasons. However, I probably will use it to pull battery cell data every hour or so to see if I can trend SoC drift.

@Tim would be handy if Giv would formalise the API to provide access to the historic data in the cloud.

locked I've added a feature request for this. Fingers crossed.

locked

Or it would be even easier if they released the Modbus registers, even if it's just the read and not write ones!

@Tim I never replied to your direct contact but I have failed in trying to find a way to pull data from the portal itself (beyond just copying the local session from the browser until the token expires)
The portal uses Laravel and XSRF tokens relentlessly for login which means your constantly exchanging the last token for a new one every time you interact with it. This should be easy to mimic in a see-saw style script (scrape the last token from the previous response > shove it in the next request) which is essentially what’s going on in the browser but for whatever reason trying to do this in Powershell just doesn’t play nice.

I haven’t gone back to BATCH yet with cURL (not sure I can bring myself to manipulate Cookies from the command line again) but I have in the meantime had a response from GE who are looking at providing a way to pull the .xlsx file from the API so all is not quite lost.

@DeanoC75 if you speak to support you may be able to get hold of them via the Developer group that they have. You will need to sign an NDA though as they can’t be shared (someone on FB posted a section of them a while back. Said they found them with a Google search but I’d be dammed if I could find them and they shouldn’t be floating around the internet but maybe you Google foo is better)

hoggy

hoggy if you speak to support you may be able to get hold of them via the Developer group that they have. You will need to sign an NDA though as they can’t be shared (someone on FB posted a section of them a while back. Said they found them with a Google search but I’d be dammed if I could find them and they shouldn’t be floating around the internet but maybe you Google foo is better)

That would be ideal for me, as you know as I've posted before, i simply want to have the information so I can use it in my Loxone home automation system. I'm getting to many issues with the giv/tcp data, and I simply haven't got the time to keep restarting/ pulling data/ trouble shooting etc etc.
Most other systems have a modbus interface, I just don't see what the issue is tbh.

hoggy The portal uses Laravel and XSRF tokens relentlessly for login

Hoggy, thanks for looking. Updating the Laravel and XSRF tokens each time a response comes back would be simple to do in Node-RED. Looking at the cURL and powerscript output it looks like they've wrapped the download endpoint using a POST method but I just couldn't find a way to do that with the Node-RED http request node. In the past, I've taken a cURL script, tested it at the Pi OS prompt and then populated a Node-RED function node with elements of the script to create the message url, headers, cookies etc which are then squirted at the http request node. After a bit of jiggling around, I can usually get it working but this time I was just getting 404 (or similar) responses from the end point.

Maybe I'll have another look in a day or two now that I know that parts of the cookies change every time. Cheers.

Tim I wish you good luck. I get 200 OK responses from it and a response but it's just "your session has expired"
Was your errors 404 or 409? There's an extra header field to contend with called "x-inertia-version" which also changes on a random basis. If it's wrong then you just get a 409 error.

I don't think i'm breaking any rules as you can just watch it do this in the browser waterfall anyway:
If it helps my logic is this -
GET Request (with no specific cookies XSRF/Laravel) > https://portal.givenergy.cloud/login
From the response pull out the initial XSRF & Laravel values from the cookies. Also need to dig through the body of he response and find the "inertia version" from the <DIV> part of the HTML. Your looking for something like data-page=....";version;73622be294f0d8693415cd7114fd1230;
POST Request > https://portal.givenergy.cloud/login
Mimic all the headers as per the browser, including the latest XSRF, Laravel Cookies. Also update the inertia version field in the header.
include the body in the request {"username":"YOUR USERNAME","password":"YOUR_PASSWORD","remember":true}
Hopefully this logs you in & returns a response including a 302 redirect as part of the header.
Copy the XSRF & Laravel tokens again and send a GET request > https://portal.givenergy.cloud/dashboard
Enjoy your data?

I may persevere with it as I like a challenge and it's bugging me why something that appears simple just isn't working. It may well be a Powershell limitation I'm not aware of that's overriding sessions or closing a connection or something.
I don't really have much to gain from it other than perhaps the ability to lift out the Solcast card info I guess...

hoggy So frustrating!! I did as you suggested: captured the XSRF-TOKEN and laravel_session objects as they are returned from initial login. Extracted x-inertia-version from the HTML and saved all of these as context variables. (This is a single node that can be re-used for every successful response). Created a new header from the previously saved objects, but concatenated "XSRF-TOKEN=" and ";laravel_session=" with their respective values to create a plain text cookie in the header; added x-inertia-version with the value stored previously, also added x-xsrf-token in the header which is the same as the XSRF-TOKEN value. Posted that to the login end point with basic authentication username and password and hey presto status code 405 with some HTML about robots. Went full out and replicated all the header objects from the web browser waterfall output and this time it timed out.

If I can successfully complete this login process it will be possible to extract a "remember_web_" string which looks like the final component to navigate to the inverter data page. Simply navigating there will be sufficient for my purposes as it is populated by an object that has all of the data I need - I wouldn't need to do a download, simply squirt the data object at the Influxdb. So close and yet not!

Tim yeah i had another stab last night but now its doing my head in - so i built a new live output map of the UKs gas & LNG processing plants instead (& did some tidying up of the TerraVolt site) in the hope that i'll have some sort of epiphany... not arrived yet...

hoggy Hoggy, thanks to the work you shared, I've managed to create a flow to get to the data for my inverters for any given date. The process is convoluted (and they'll probably change it at some point), but if anyone else wants to know the process that works currently:

1. Login using userid and password with a GET to the login URL - capture XSRF-TOKEN, laravel_session and x-inertia version (from within the HTML body) for use later as cookies. You should get a 200 server response. Note that the XSRF-TOKEN and laravel_session values seem to change with every response, so need to be saved each time.
2. Login again using userid and password and: a header cookie containing XSRF-TOKEN and laravel_session and their associated values as a simple string; x-inertia-version set to the previously captured value; x-xsrf-token set to the value of XSRF-TOKEN. Use POST this time to the login url. You'll likely get a 405 server response but capture the new response cookies (XSRF-TOKEN, laravel_session and remember_web?????). If you can capture these data, you're in with a fighting chance.
3. Use a GET to your inverter data page (https://www.givenergy.cloud/inverter/SDxxxxxx/data where SDxxxxxx is your inverter serial number) with the Headers set as above but add remember_web????? and the associated value to the plain text header cookie containing XSRF-TOKEN and laravel_session but use the latest values. You also need to add the x-inertia-version and x-xsrf-token with latest values again. This page gives you the data for today returned as an object. Again, you need to capture the new response cookie values, including the laravel_token which makes its first appearance at this point.
4. To get to historical data, you need to create a plain text header cookie with laravel_token, XSRF-TOKEN, laravel_session, remember_web????? and it's associated value in it along with x-inertia-version (which doesn't seem to change) and x-xsrf-token which needs to be set to whatever XSRF-TOKEN is from the last response. Use a GET to https://www.givenergy.cloud/internal-api/inverter/data/SDxxxxxx/YYYY-MM-DD where the last bit is the date you want your data points for. The data is returned as a JSON object which is perfect for me as it contains the 280ish rows of System Data and Meter Data and each one is properly timestamped.

I will need to run the whole process separately for each inverter as the tokens that change don't seem to be able to be transferred across to the other inverter. Maybe if I navigate from the inverter data page to the dashboard and then to the other one, it might work, but for simplicity, I'll just run it again.

Once I've extracted each day of data for each inverter, I'll have to run this once each day for yesterdays data for each inverter. When it's working fully, I'll post the flow to the Node-RED flows.

Hi Tim,
Glad to see the theory is sound in Node red must be something weird with powershell I'll get to the bottom eventually.
I suspect the ability to only talk to one inverter per session is a side affect of some findings I had back in the earlier incarnations of the portal I'd passed back to GE which they then patched. I think having to run 2 different scripts is a better trade off than relaxing security to be honest.

Hi Tim,
This still working for you? As of yesterday the Portal internal API has moved to “friendly” names (I.e your user name instead of your customer number for instance)
I never did get around to sorting this through powershell but thought I’d let you know if your Node-Red had packed up.

It has removed my ability to work out how many installs there are too which is a bit annoying!

hoggy removed my ability to work out how many installs there

Maybe that's why they did it. Haha.

Yes the process still works. I can pull all the data for one inverter for one day at a time, so an object with 284ish datapoints. That's been working fine since we cracked it a few weeks ago (thanks for your help). My issue has been squirting the objects into Influxdb 2.3 which I got back to at the end of last week (family wedding distracting). My lack of Flux scripting knowledge/skills is slowing me here but I need to query my database to find the latest entry for each inverter and then go and retrieve the data for that inverter for the next day. I could just run the process a couple of hundred times to load the data but I want to be able to leave Node-RED checking every few hours and do a complete day download the following morning. I could do it every hour or 30 minutes but to be honest, I'm really only looking for longer-term trends, so yesterdays data will be sufficient.

I now have an Influxdb bucket called energy with a measurement called InverterData and tags for the relevant datapoints which are the inverter serial number. After that all the data is stored in the datapoint as key value pairs. Just the query "last" that I need to progress now.