Security breach - anything we should be doing?

I just saw this article on Ars Technica: https://arstechnica.com/security/2024/08/home-energy-system-gives-researcher-control-of-virtual-power-plant/

A security researcher gained access to, it seems, admin rights over the entire GivEnergy cloud and all installed products (inverters, batteries, etc) connected to it. GivEnergy were using an insecure key that could be easily cracked with modern compute resources. According to the article they have since fixed it at their end.

Is there anything we need to do at home? Should we be concerned about this?

Well that's a bit jaw dropping. Presumably we are due a formal response from someone senior at GE. Assuming the article is legit?

It's already been patched.

https://rya.nc/vpp-hack.html

https://givenergy.co.uk/real-world-example-givenergy-security-and-cyberthreat-response/

Hook Looks like they responded quickly. Did they tell anyone or just put an update on their website? It looks like they have beefed up the overall security so that's encouraging. Fortunately it was an ethical hacker else it could have been nasty!

That is a really good response by GE. I hope that they rewarded Ryan suitably.

sponev
Ars Technica is probably the most trustworthy site for tech news. The journalists tend to be scientists/ technicians with quite an in depth understanding of what's going on behind the scenes. The comments section can be a wealth of information from some of the most technically literate people. With the way the comments are structured, the usual trolls are quickly downvoted, you can still see the original comment but need to click through to reveal it.

sponev
Fortunately it was an owner who spotted it. And before disclosing he approached Giv.

I’m a Head of IT, but probably the least technical I’ve ever been.

But I do pay attention to risk. I don’t need to understand all of the technical details, just the overall gist.

So I was aware of this one.

The ArsTechnica Article is excellent (almost perfect). Ryan's actions are to be commended. The response of the GE technical teams is correct. Technically this particular hole has been filled in. However the article on the GE website, clearly written by marketing types, is significantly flawed.

Simple errors, like the first image has readable code relating to the EV charger. Blur it, or take the picture from further away!

More fundamentally, the statement "all customer data is hosted in UK/EU clouds" is promoted as a good thing from a security perspective. This is misleading. The data is accessible from anywhere in the world, and the method Ryan found could be exploited by anyone anywhere. Where the data is physically held is irrelevant here. And whilst it is true that data is hosted in the UK, the servers are owned and operated by Amazon, a US based organisation.

Amazon could go bust, or just close down AWS. (If it did GE would be the least of our problems!). The number and location of AWS data centres in the UK is secret ( ... as secret as big buildings with lots of aircon on the roof and no windows can be). I can tell you it is not many so this is a vulnerability. Potentially a foreign actor could target them with missiles/bombs or sever the power or data connections in some way.

This is all beyond GE's control, and certainly beyond your control. So the only question you need worry about is what would you do if the cloud portal stopped working? This could be a total outage or just loss or corruption of historic data. And what if someone takes remote control of your system, what will you do? I know what I will do, do you?

Rubikcube Panic !!!!!!

What would you suggest as an emergency action plan?

Robgy Rubikcube's app allows local control of the inverter. However I'm not sure if a GE system can run indefinitely when disconnected from their remote systems - hopefully @Rubikcube can help us with that? If that is the case, then you should be able to disconnect from the WAN and run locally. I have the purchased version of the app (everyone should have it as back-up in my opinion) but really only used it for monitoring so far. It's only a tenner or thereabouts so worth investing. There's also another app which does similar I believe written in BBC Basic (?) but I haven't checked that one.

sponev However I'm not sure if a GE system can run indefinitely when disconnected from their remote systems

It can.

sponev I too have the paid version of Rubikcube's app and only use it for monitoring but I think (not certain) it needs to check online with Google pay if you want to use the controls which could be a problem if the internet is down.
I also have the BBC coded app and a very basic install of home assistant again for monitoring but I only use the Givenergy app for any adjustments to control the inverter.
I don't have any emergency plan in mind but perhaps it would be prudent to consider something.

Robgy Well it looks like at least one of those apps will keep the job running in stand-alone mode. I guess that might be sufficient while GE came back online. If it was sufficiently catastrophic that GE was permanently broken, then not sure what we'd all be doing next, probably looking for alternative equipment. The other issue is security and keeping bad actors out of our local network. Ideally I'd want to separate the GE kit but haven't gone to those lengths so far. But then where do you stop given the large number of other devices, CCTV, various appliances etc which are also connected?

Neat bit of work by Ryan, and a super-fast response from GE. But it reminds me about chains and weakest links... The GE article rightly bigs up the UK dev team for portal and app development, API tokens, etc, but doesn't ever mention how the portal and inverters communicate with each other.

From what I understand, the inverter (or an attacker...) just connects to the portal server (unencrypted, no real authentication) and says "Hi, I'm inverter serial no. XXXX", and the portal replies. "Fine. Just send me data"...

I've seen unexplained garbage data in my portal graphs, which could potentially have been caused by someone exploiting this. You wouldn't be able to directly control the inverter, but poisoned data could modify the behaviour of 3rd party apps using the portal (e.g. Wonder Watt).

Ideally there would be a secure (encrypted, authenticated) link between inverter and portal, but I can't see that happening on legacy hardware. The next-best option would be software that runs on something near the inverter (dongle?, HA Add-on?) to secure the current protocol correctly.

sponev But then where do you stop given the large number of other devices, CCTV, various appliances etc which are also connected?

This is a question I have been mulling on for a while. I've always used two vlans - a trusted one, and an internet-of-things one. There are some things at the edge (why do I trust an Apple laptop but not a media server?) but it's pretty clear - keep non-mainstream stuff off my trusted network.

I have been planning on having a third "energy" vlan. I don't really trust my energy cloud-based, Chinese-owned/controlled kit enough to allow it on to my Trusted vlan, but neither do I want to allow potential access to it from eg my freeware plugs on the IOT vlan.

It's just a bit of a pain to disentangle everything - I'll have to have two MQTT brokers, two Home Assistant instances, etc.

Is the game worth the candle? Am I venturing too far into foil hat territory? Dunno. Some/most people just have everything on one flat network.

Looks like some good discussion here.

It’s good that GivEnergy responded quickly and positively to this, you do have to ask whether there is enough proactive testing/white hat attempts to find weaknesses etc, and some statements about patch policy, security testing etc would have been worthwhile making.

The statement on the website is as @Rubikcube points out, written by the marketing department. Some of it is complete guff, having the software department as a separate legal entity does nothing for IT security, and whether the team is UK based or not matters not either. Given the challenges there are in getting new firmware updates developed, tested and rolled out and the random things that happen sometimes to the portal and app, just what those 30+ software engineers are doing and what the split between support, feature development, operational management, testing, etc is.

I also find it disappointing that there’s no statements about how the systems are secured, encryption at rest, in transit, account security policies (who has access to what), adherence to AWS best practice design, resilience of the infrastructure against single points of failure (which links in to how software changes are developed, tested and rolled out), testing automation (including security and resilience testing), etc.

I’m sure there is a lot more good stuff going on, its just not written about.

PianSom it's a conundrum for sure. We run a web business, a lot of it from the home office and so security is particularly important. We don't runs separate LANS (yet) but ensure we have 3 levels of backup (including off site) so we're only ever one day away from re-establishing if things went badly wrong. Assuming of course they didn't compromise the backups. When/if I get time, I'll look again at the viability of separating systems.

Once again its lazy programming - relying on public libraries which are NEVER checked by the programmers who link them in, they always assume "someone else has".

Didn't the polyfill.io domain takeover teach anyone anything?

I don't agree with Ryan's statement "Expecting developers to know that 512 bit RSA is insecure clearly doesn’t work. They’re not cryptographers. This is not their job."

If this was some sort of new vulnerability then fair enough to some extent. It isn't.

If you don't have a basic grounding in what is and what is not considered secure then you shouldn't be anywhere near the code. Public libraries should NOT be a substitute for competency.

Will say that all the faults I found (haven't been playing for a while) got rectified quickly by the UK team.

hoggy Good to hear but epic fails like using a 512 bit hash function broken more than quarter of a century ago do NOT inspire confidence in either GE's competency or QA (stop laughing at the back there 😉 ).

Edit - or whoever actually codes the portal/back-end which I'm pretty sure isn't GE, its someone in China. Happy to be corrected....

Vestas As a developer myself, I think its surprisingly common for develoeprs now to just "npm install..." or "composer require" or whatever your languege of choice has, and not really think about the actual guts of that component - developers I've seen just treat them as a "working black box" and focus on the bits they need to do around it to achieve. I've got to say, modern frameworks, and centralised dependancy management has kinda spoilt things, because once installed, you never go back to the source to check it unless you need a new feature or something doesn't work.

Another (non web) project I worked on (retro console) I was provided a "working SDK" from the board designers (China) - that was based on Linux 3.4, this project was started in 2020, when Linux Kernel 5.x was out. Again, its not a surprise that things end up old/outdated/not supported.

Its a balance, thats for sure.

Not sure if this had anything to do with it but around 3pm today my battery stopped discharging. Only noticed it this evening. Rebooted inverter didn't make any difference..went into inverter settings and toggled eco mode off and on and then it started discharging again.

@Vestas I'd strongly recommend use of a tool like Snyk, which detects vulnerabilities in 3rd party packages - and your own code.
https://security.snyk.io/vuln/SNYK-JS-JSONWEBTOKEN-3180026

The genie of npm install is out of the bottle, and unless you want to go back to the bad old days of coding every last line yourself (and introducing waaaay more new and exciting vulnerabilities along the way), it's best to accept that and use tools to ensure you manage it well.

The polyfill.io fiasco was (AFAICT) a CDN takeover, and the moral from that is to host/bundle packages yourself, or use browser integrity checks on 3rd party scripts to ensure you don't randomly get a hacked one.

TimI developers I've seen just treat them as a "working black box" and focus on the bits they need to do around it to achieve.

Indeed.

Moxie Marlinspike did an interesting talk on how Agile is killing software development/innovation at this year's Black Hat. One of his points is that black box abstraction has left the majority of software developers unable to do anything other than derivative work - no meaningful innovation takes place, its all reused code, mostly treated as a "black box" which the s/w developer has no knowledge of.

That's pretty much what we've* seen over the last 30 years.

Talented software engineers are rare as rocking horse shit these days.

*our family business has been developing engine management software for decades; if you've flown on any Airbus/Boeing in the last quarter century part of our code keeps you in the air. NB - we had bugger all to do with the GE engines/flight controls on recent Boeings 😉

Vestas One of his points is that black box abstraction has left the majority of software developers unable to do anything other than derivative work - no meaningful innovation takes place, its all reused code, mostly treated as a "black box" which the s/w developer has no knowledge of.

Yeah... but it's black-boxes all the way down. How often do you worry about the voltages travelling along the busses on the motherboard ? Or how a cpu is built out of logic gates ? Or how even a low-level language gets compiled to machine code ? If you're running python, the engine itself is a long way above the bare bones, and then you have the standard libraries on top of that.

So I'm not sure that building high-level software out of lower-level libraries is very different. It's rare these days to address individual pixels when you're writing a gui.

That said, I do find it a little surprising how quickly people reach for add-ons (I'm thinking python specifically) to do something fairly trivial. I've ripped out quite a lot of apparently unnecessary third-party dependencies on some code I've been looking at recently.

DD How often do you worry about the voltages travelling along the busses

Well Intel certainly didn't 😉 13th/14th gen chips are junk.

That however just proves that beancounters running engineering firms results in crap coming out the door.

The fact remains that far too many "developers" rely blindly on what is in a public library and in the case of GE, this appears to have resulted in code utilising a hash function broken 25 years ago.

That in turn indicates no meaningful design process whatsoever (so corp-Agile no doubt).

Decisions about security and cryptographic algorithms should not be made on the fly by a single developer using untested (either that or QA are incompetent too) public code. Not in any sort of competent organisation and certainly not in one wholly reliant on internet comms to control products.

Vestas using untested (either that or QA are incompetent too) public code.

Is there any reason to suppose that the public library providing the 512-bit code was untested? It almost certainly implemented that code correctly. (Many-eyes principle.) The bad decision was the choice of algorithm, not relying on an untested implementation.

DD Precisely, the 512-bit RSA encryption in the library was working fine, just the fact that computing power has exponentially increased so much that 512 bit keys can be easily cracked with some reasonably cheap cloud computing resources.

I'd imagine GE has updated this to a new 2048 bit key.

DD Someone made a design decision to use an insecure hash function and clearly they should never have been in the position to do so. Nobody tested or reviewed that within the company - or they were not competent enough to do so.

There was a video going around pre-YouTube days of two guys using Voodoo2 graphics cards in SLI mode to break the hash function. Voodoo2 cards were 1990s tech so its nothing to do with exponential increases in computing power.

They chose to design in a hash function broken 25 years ago and the guy who found this (Ryan) seems to think that its the job of the library to prevent use of that function, not the programmer/design spec.

If you can't see the basic problem here then I dunno how to assist you further....

Vestas They chose to design in a hash function broken 25 years ago and the guy who found this (Ryan) seems to think that its the job of the library to prevent use of that function, not the programmer/design spec.

If you can't see the basic problem here then I dunno how to assist you further....

I think we're agreeing - the problem is choosing to use the 512-bit algorithm.

DD The problem is that the algorithm was probably the default and the programmer assumed "the library knows best". Same as pretty much all of them do these days.