Modbus volunteers wanted - register collections

Some of us are working to get the low-level modbus library brought up to date. Unfortunately we all have rather boring hybrid setups, so we can't test against anything more interesting - AIO, 3-phase, EV chargepoints, etc.

It occurred to me that it should be possible to mock up a simple modbus server, just enough to fool the client into believing it's connected to an inverter. So all I need now is some snapshots of register values, to build realistic models. Anyone interested in offering up recordings from their systems? There are various ways to collect traffic, and should be completely safe, other than leaking serial numbers (though I would anonymous anything before publication).

I'm happy to test stuff against my AIO if that helps? Might need a bit of guidance though, I've not used modbus with it before.

Happy to help but gen 1 hybrid with 191/193 firmware for me

KevinB The inverter allows multiple concurrent connections, and a feature of the GE implementation is that when any one client makes a request, they all receive the reply. This means it's possible to capture all the replies by simply making a connection and listening, while using other tools to actually send requests. This can be GivTCP, https://givenergymonitor.wixsite.com/tutorials, the iOS (bbc basic) app, the standard app, or even the portal, to a limited extent.

• ah - I see you're already a predbat user, so that should be covered.

If you use linux, you can make a connection using the socat tool - also available for mac/windows ? https://github.com/3ndG4me/socat/releases

eg socat -x -u TCP:host:8899 CREATE:binfile where 'host' is the ip address of your inverter. This makes a tcp connection to port 8899, and simply writes everything it receives into a file called 'binfile'. (-x means hex-dump to screen, -u means uni-directional LHS to RHS only.) This will only remain connected for a few minutes - the inverter sends out 'heartbeat' requests every so often, and if you don't respond, it will close the connection.

Another way to do it is man-in-the-middle: socat -x -r binfile TCP-LISTEN:8899 TCP:host:8899. This will set up a listening socket on the computer's 8899 port, and when a connection comes it, it will make an outgoing connection to the inverter, and forward all traffic bi-directionally. Here, -r binfile tells it to record all the data from LHS to RHS to the binfile. Then you can tell the official app, or any of the third-party ones, to connect to your computer rather than the real inverter. Because the client should respond to heartbeat requests, this should stay connected indefinitely. Not that I particularly need a lot of traffic - just need to be connected for long enough for whatever tool you're using to explore the various registers.

Either way, I then have a tool which can replay the file through the library, parsing the messages and extracting register values, which I can then dump to create a register file for the server emulator I'm fiddling with. (Just about have that working.)

In principle, it should be possible to use a snooping tool like wireshark and capture the traffic without making a connection to the inverter at all. That's something I could explore if anyone really doesn't want to make an active connection.

If you're actually interested in getting into modbus in more detail, you can use the python client library to explore more actively. But it's a bit catch-22 - until we get support for AIO, it's not so easy to use the tool for that purpose.

There are several versions of the library around - I think there's one in the GivTCP tree which has better support for the more exotic systems - britkat1980 has better access to the givenergy internals. Perhaps here: https://github.com/britkat1980/giv_tcp/tree/dev3/GivTCP/givenergy_modbus_async - you can see in the 'model' subdirectory the different families of devices, and it's these that I'd like to have some way to test, in the absense of actual hardware.

My clean tree is at https://github.com/divenal/givenergy-modbus-async/tree/clean6 (currently awaiting integration into hoggy's master tree), and it's a little behind that one. We will get them to converge eventually, but a simplification I made to one of the other files (to avoid duplication of register definitions) needs a bit of revision to be able to target these different system types. So detecting the system type is the next bit I want to work on.

DD Fantastic info, thanks - I've collected a file from your first socat command, and I clicked refresh on a load of the GivEnergy cloud "remote control" screen settings so it did a read of numerous different things. Would that file be useful? I'm happy to email you it, or upload it somewhere?

DD I see you're already a predbat user

Only using the "cloud" integration, not the direct modbus one, and currently still in read-only mode.

KevinB Yes, that would be very helpful. I've tried creating a discussion on githib at https://github.com/divenal/givenergy-modbus-async/discussions/2 so you might be able to attach a file there. As mentioned, it will include serial numbers. If that worries you, you should be able to get my email address from my github profile.

Many thanks.

KevinB Also running the official app might help - that should prompt the inverter to send an update to the portal every 10 seconds. That should add some of the input registers. The portal tends to worry mostly about holding (= configuration) registers.

@Rubikcube : maybe of interest to you ? I've hacked up a simple python modbus server which looks enough like a hybrid inverter to allow your app to connect to it. (The error count in the top-left corner does tick up - would be interested to know what it's complaining about.)

See https://github.com/divenal/modbus-scripts/blob/main/server.py

DD This is certainly of interest, unfortunately I'm a bit busy with other projects during the summer. I'm impressed that you have built an emulator, even more impressed if you have successfully emulated the flaky unreliability of a GE dongle 🙂

Wireshark may help reveal what is going on with the error count. My app may have bugs. In the early days it was developed mainly using trial and error methodology. With the disclaimer that the comments may be out of date, misleading, or just plain wrong, here is my kotlin code for the modbus header I send so that you can see how I documented my thinking at the time.

            0x59, 0x59,  // Transaction Code (seems fixed)
            0, 1,  // Protocol Identifier (should be zero)
            0, 28,  // Length = number of remaining bytes in frame
            1,  // Unit Identifier = Server Address
            2,  // Function Code (2=Read discrete inputs)
            0, 0, 0, 0, 0, 0, 0, 0, 0, 0,  // Serial Number
            0, 0, 0, 0, 0, 0,  // Padding
            0, 8,  // Secondary length !!
            48,  // slave address (0 GeApp,17 Inverter,48,49,50..54 batteries,55,maybe others)
            4,  // Function code (3 = read holding, 4 = read input, 6 = write single
            0, 0,  // From address (multiples of 60 only)
            0, 60,  // Data length, return packet size double plus 44
            99, 99 // CRC - Cyclic Redundancy Check
        )

I do tend to mix and match decimal and hex according to whatever makes most sense to me.

@DD I've emailed you a couple of captures, hope they're helpful - let me know if anything else would be of use, or if you want anything captured whilst specific things are happening.

KevinB Thanks. Unfortunately, not quite as complete as I'd hoped: when reading registers, it's common to fetch them in blocks of 60, so I'd hoped that when forcing a read from the inverter, the portal would ask for lots, so I'd see the other registers as a side effect. But unfortunately not - it reads them one at a time.

If you have time/inclination, best way would be to use either https://givenergymonitor.wixsite.com/tutorials (android only) or the bbcbasic app which runs on a variety of platforms - see https://community.givenergy.cloud/d/438-how-accurate-are-the-import-from-grid-metrics/16

I've only used the former, but a quick look through the source code for the latter suggests that it should read blocks of 60 registers. I ought to try running against my dummy server to see if that throws up any other insights. No idea if it's been updated to add support for AIO.

Thanks

DD Not a capture, but I was working on a Wireshark dissector for the GiveEnergy protocol a while ago. Is that any use to you? https://github.com/roddypratt/GivEnergy

rjp Oh, that's very interesting - I'll have a look. Thanks.

DD I've emailed you a couple more socat captures, with the BBCBasic app running.

KevinB That worked even better than I'd hoped: not only did the bbc app get (most of) the configuration registers, but I got a lot more of the data registers than I'd expected. I suspect that the run happened to coincide with one of the periodic (5-minute ?) updates the inverter sends to the portal.

In the pause session, I didn't see as many configuration changes as I'd expected. Doesn't affect the results at all, but might just mean that the passive watching mechanism isn't quite as effective as I thought.

Out if interest, do you also have a gateway, or just a standalone AIO? I think gateway can be accessed over its own IP address, but I'm not sure how that behaves.

@hoggy just FYI the captured data includes addresses 0x50-0x53 (bmu's ?), 0x70 (bcu ?) and also 0xa0. Not sure what that one is. BMS, perhaps, if that's something that can be interrogated separately ?

DD Take it thats from an AIO.
So the stack works as so for those (there's no one "BMS" as such like the LV stuff, it's a combined system of battery modules (BMU) and a Master (BCU)

Each AIO has 4x Battery Modules installed with 24 cells in each.
Each Battery Module has a BMU controller (which matches your 0x50 - 0x53 you've seen)
All those 4 talk back to the master, the BCU (0x70)

AIO Inverter > AIO BCU x1 > AIO BMU x4 > 24 Cells, Temp Probes etc...

The BCU can presumably talk to other BCU's such as when you have multiple stacks on 3 phase inverters as they appear to be much the same architecture, although I'm unaware how the addressing works it's probably likely they increment from 0x70 onwards?

as for 0xa0 I'm not sure I've seen that one (from memory anyway) let me have a dig. Initial thoughts are it might be the dongle interface (I'm yet to screw around with that but given GE can set things like 60sec updates and so on remotely, which presumably requires the dongle itself to be tweaked (given there's no other registers kicking around for that?) and I've been curious how this is done.

DD Yes, I have a gateway as well. It does have a dongle and IP address, although it seems to be fairly dumb from a cloud control POV (it's very uninteresting on the GivEnergy portal, there's no battery readings or power flow data or anything like that).

I also have a Gen3 solar inverter, which feeds AC into the gateway, so presumably I can capture data from that as well if you want.

Is the network traffic unencrypted on the wire? If so, I can tcpdump from the wireless AP that they're connected to.

KevinB I also have a Gen3 solar inverter, which feeds AC into the gateway, so presumably I can capture data from that as well if you want.

An AC-coupled inverter? Sure, the more the merrier.

Is the network traffic unencrypted on the wire? If so, I can tcpdump from the wireless AP that they're connected to.

Yes, it's unencrypted. See above for a wireshark decoder (WIP). Looks like there are several tools around which can reconstruct a tcp stream from a packet capture.

DD I'm not sure if I'm happy or sad that it's unencrypted 😆 I've emailed you a tcpdump pcap file with a few things in it.

DD I'm wondering if 0xA0 might be the STM32 chip. Can't really find anything in anything I have and a Google of that address brings up (amongst many things) that it's used as an address to read / write to STM32/EEPROM chips, which kind of sounds plausible. Perhaps how FW updates are done? obviously the dongle must have access to write the control board some way or another. Will keep looking.

happy to be involved but be aware I may need some technical handholding - background is networks and controls with limited exposure to modern dev tools.

Anyway I have a GE 11kw 3 Phase inverter, HV battery stack plus PV's all connected to a flat network hardwired fixed IP.

Let me know

Valex1p If you're running home assistant on some sort of linux system, such as raspberry pi, and you have access to a shell prompt, it should be relatively straightforward - using 'socat' as above to make a passive connection to the inverter, and then recording anything received. Could also use something like
tcpdump -w capture.bin port 8899
which will just record everything involving port 8899 (default inverter port), but that will probably only record packets involving the host machine, not the portal or an inverter app. But givtcp itself should work. Do bear in mind that this may also record broadcasts and potentially other stuff on your network, but I'm sure you know all that.

As you probably know, the inverter tends to upload status to the cloud every 5 minutes. Timing the session to coincide with one of those would probably be helpful.

Once you have a connection, anything done to exercise it will generate traffic. Eg changing settings from the portal. Unfortunately, most of what the standard app does is actually read from the portal, rather than directly from the inverter, except the home screen. Not sure if the bbc basic app mentioned above is able to do anything with the 3phase system.