Much has changed since our very first entry three years ago, and yet much remains the same. We remain on our 10 acres of boreal forest, still living off grid in what we described as our physical off-grid castle. We are in the process of making a number of improvements as mentioned in yesterday’s post, so I thought an overall update might be apropos. On the other hand, since most of what we wrote is still accurate, I will leave that original post up as well.

The enabling changes since 2020 have been updates to our off-grid electrical system. I will dedicate a future entry or two on more technical details, but many of the changes will allow our eccentric homestead to be even MORE…

9kW Generator

Our 6kW Northern Lights generator had been reaching the end of its useful life for quite some time. I wrote about having to replace the generator controller (a Dynagen GS300) before. The NL 673 generator was quite reliable, but, as you may imagine, after 10+ years of daily service, the blue beast was in need of a complete mechanical overhaul. The voltage regulator was failing. The water pump was leaking, which was causing regular overheating and shutdowns, we had a rear bearing replaced, but then the front one began to fail. Seals were starting to leak oil, a glow plug needed to be replaced. But there were other issues: the size of the generator was just barely adequate for our loads, it was not a brushless generator, so the brushes had to be replaced at regular intervals, always at an inconvenient time. And while the brushes were not expensive, the pandemic supply chain problems lead to a deal of anxiety about inventory on hand.

For all of these reasons, we opted to buy the next size up, a larger, brushless 9kW generator. Even with its slightly larger size, our new generator still fits in the same space as the old one, albeit with a larger footprint, and we were able to re-use the old exhaust and fuel lines. Just moving up to a brushless model is such a benefit that I now wonder why anyone who uses depends on a generator for either prime power, or even reliable backup, would be willing to settle for brushes!

Here is the new generator, as you can see, still a sexy blue beast!

Generator controller – a missed opportunity

TL;DR – communications geekery ahead, you may want to scroll down to the next section if you just want to read about updates to our off-grid homestead.

One missed opportunity: the new generator has the same GSC-300 controller as our old one. We could have taken the opportunity to upgrade to a more featureful controller as I mentioned in an earlier post, but at the time the GSC-300 was in stock (and is a regularly stocked product at Alaska Diesel Electric) and the newer TG-410 controller that can speak MODBUS towards a server and CANBus toward the ECU (Engine Control Module) was not in stock. This means that there is no way to remotely monitor the performance or even the fault codes for the generator. The controller responds to a simple auto-start and stop requests from the inverter, but does not communicate status, other than on its local display and a series of indicator LEDs on the front panel. As useful as that is, there is no way of knowing from the house whether the generator has stopped abnormally, and if it has, what the error condition might be. There is no datalogging to help identify performance issues over time. Ideally, I would like to be able to share voltage levels and even AC-waveforms from the generator with remote technicians to allow for a degree of off-site diagnostics and troubleshooting.

One reason that I did not order the newer controller is that I also do not yet have the infrastructure set up to interface with the controller. Ideally, the controller would provide an ethernet or WiFI interface and support either direct API calls, or even better a message passing bus such as MQTT. MQTT in particular would be advantageous, because the direct communications between the controller is and the outside world is through a broker which is then responsible for relaying messages and maintaining connections with all other devices. What this means is that the engine controller could publish a message with a “topic” like “Engine controller” and a payload of “overheat fault” to the broker without being concerned about whether anyone was listening to the topic. Other bits of software (say that provide a graphical display or web site) can subscribe to the “Engine controller” topic on the broker without having to directly contact the controller itself. The broker can hold messages for clients to ensure that they have received all of the topic to which they have subscribed. In other words, MQTT, unlike MODBUS, allows status messages to be treated much like they are on the control panel. It can just send a message, or illuminate a light, and not be concerned about who is watching.

Ok, enough geekery. The point is that no engine controller (that I am aware of) directly speaks MQTT. Rather than using a “modern” computer (or “machine-to-machine”) interface, the controller uses the industrial MODBUS system developed in the 1970’s. Granted there are some advantages to this relatively low-speed archaic method, but to make use of the data from the house or share with a remote technician, I need a way to transfer the MODBUS message over a computer network and a way to process the data.

There are various means to accomplish this task, there are MODBUS TCP adapters that allow MODBUS messages to be sent over an IP network, for example, but it is still necessary to process these messages that are transmitted in this obscure format, and even still, there are limitations in the design of MODBUS itself (it is a “master / slave” protocol that only allows for one master“) that makes it a difficult fit our use case. There are also MODBUS to MQTT gateways that may be worth a look. What I really want is a simple way to:

1. View the status of the generator, including any fault messages
2. Collect statistics on run-time, RPM, voltage, frequency, etc.
3. Remotely start or stop the generator
4. Provide all of the above to a (yet to be chosen) home automation / load management controller (more about this later)

Fortunately, there is an open source project called Genmon that comes very close to these goals. Genmon is not just a gateway. While it does convert MODBUS messages into MQTT, it also provides a useful local generator specific web interface. While it does not have support for DynaGen controllers, the documentation does claim that the interface is customizable.

Note that I cannot use any of this yet, because my primitive GSC-300 controller does not support MODBUS. I suppose that I am hoping that a generator controller will be released that directly meets my needs, otherwise I suppose I will eventually upgrade the controller, and load genmon on a Raspberry Pi.

Sol-Ark 15K inverter

Along with our new generator we also replaced our old Trace SW-4024 Inverter. As previously mentioned, the Inverter is the brains of our whole electrical system, and like HAL, was somewhat evil challenging to live with (as it turns out, our new inverter has its own challenges as well). The Trace was purchased in 2002 and so made it 20 years. It was a reliable unit, but already on the edge of obsolescence when the original homeowner installed it. Its capacity, 4 kW, was not enough to run all needed appliances in the house without assistance from the generator, so we ended up running the generator more hours per day than would have otherwise been required to just charge the batteries. To add to this frustration, the inverter only provided a single “leg” of 120 V AC power. As you may know, most homes in North America are wired with a 240 V AC split-phase configuration, where 120 V AC is available between one of the hot “legs” and a neutral wire, and 240 V AC is available between the two hot wires for devices with motors and other high current appliances. The Trace inverter was designed so that it could be “stacked’ with another inverter and provide this configuration. This would have resulted in a reasonable 8kW inverter system that also had a degree of redundancy, but alas, not only was the second inverter not present, but it was mounted in such a small confined space that there was no way to add the inverter and required support equipment later.

Another limitation of the old inverter was that the DC side and the battery were a 24 volt system. We have over 3kW of solar panels, but because the inverter did not have a integrated charge controller, we would need external solar charge controllers to connect our panels. The only charge controllers for 24 volt systems were relatively low voltage on the solar side and relatively low current on the DC side. We would have needed 3 charge controllers mounted near the batteries (to avoid voltage drop on the 24 VDC side), but as I just mentioned, the inverter and batteries were mounted in such as way as to make room for the required three charge controllers, buss bars, and disconnects was just not possible.

So we replaced the 24V Trace with a new 48V Sol-Ark 15k. The Sol-Ark includes three integrated solar MPTT charge controllers. Each of the Sol-Ark charge controllers supports up to 6kW, so now we can finally connect our panels directly to our system without adding any additional components. The Sol-Ark is a 12kW battery inverter that can also supply 15kW of 240 V AC power if at least 3kW of solar input is available, so that means on a sunny day, with charged batteries, we can support a load of 15 kW (at least for a little while!).

The Sol-ark also has a web-portal that shows everything from voltage and current in and out of the system, to battery state of charge, and even system temperature. It has an internal CANBus and MODBus interface to communicate with batteries that provide CANBus or MODBUS for closed loop communications and controller with an intelligent battery management system (BMS). There is also a generator start/stop control line, but sadly, no direct MODBUS interface towards a generator controller.

The Sol-Arks can also be stacked, in this case, they are stacked in parallel each adding to the total load and battery charging capacity of the system. We have one now, but have room to add up to two more along the rear wall of the garage, where the inverters and batteries now live (that would be 36 – 45 kW of peak power, almost as much as the 48kW that a standard 200A utility electrical service can provide!). More practically, that means, that at some point in the future, if we installed enough solar panels, we could potentially charge an EV during the day and still have enough capacity left to run the house. Even better, if bidirectional EV changing is ever widely supported, a parked EV could supplement the house battery bank.

More communications and control

TL;DR – again more communications geekery ahead, you may want to scroll down to the next section if you just want to read about updates to our off-grid homestead.

Although Sol-Ark does provide a useful web interface, it is not real-time. Rather it is a cloud service that polls the inverter every 5 minutes or so and then displays the most recent data. That means that it only works when our home internet connection is available (to be discussed in a future post). There is no method to import this data into other systems for control or analysis, and any polling intervals that are missed, are just gaps in the data. On the other hand the inverter does sent alerts to mobile devices when a fault is detected and reported.

It does turn out that the inverter does allow the MODBUS port used to communicate with the battery can also poll read-only data from the inverter. There is a product called Solar Assistant that can then attach to the MODBUS port and provide both a local web interface with data logging and an MQTT data feed for use by other systems in the home. Solar Assistant, because it sits on the MODBUS has a “near real-time” view of the system, so that for instance, when an appliance like a dishwasher or the well pump starts consuming power, the load data is available within a second rather than 5 minutes later. This granularity allows for a better view of the system, and just as importantly a way to monitor and control the system from home when the internet is down, or Sol-Ark’s server is inaccessible.

Ideally, Sol-ark would just include this functionality within their product or license the software and embed it. Fortunately, it is not too expensive.

Sol-ark has announced some new products recently, including a smart load controller, solar panel monitoring and optimizers, and a line of microinverters that promise some form of local communications. Lack of local control is currently an industry wide issue (I guess that since most solar installations are grid-tied, most consumers also do not care that they do not have direct access to their production data or inverter status?) In any case, it will be interesting to see the direction Sol-Ark chooses to take with these new products.

In an off-grid setting, dynamic load management and home automation are related, if not tightly coupled concepts, or at least they should be. This is an area that we have not yet really explored, but I have a few ideas. More later.

Homegrid 38.4 kWh Stack’d Battery

As mentioned above, we have upgraded the DC side of our power system to a 48V. Although it would have been possible to convert our 24V lithium battery system to 48V (with a great deal of work, including disassembling the batteries and rearranging the cells) the resulting system would still be only 20 kWh. To upgrade to a 40kWh system, would have cost almost as much and taken up twice the space of the new Homegrid Stack’d system, so we replaced it. The Homegrid has some other great features, one of which is a large display on the front of the Battery Management System that sits at the top of the battery that indicates battery health and state of charge. It is hard to overestimate the importance of knowing the SoC in an off-grid battery powered system. Very quickly one learns to check the SoC before starting a load of laundry. Ideally, of course, this would be something that an automatic load manager would take into consideration, perhaps delaying the start of a dish load by 30 seconds while allowing another high amperage appliance to stop, for example.

Another feature of the Homegrid BMS alluded to above is the closed loop communications with the inverter. The inverter system (including the battery charger and the solar charge controllers) all know the battery’s state of charge and can take actions based on it. For example, starting the generator if the SoC is less than 50%, or redirecting solar photovoltaic panel power from battery charging to loads, or even lowering the output of the panels when the battery is fully charged. The battery parameters (voltage, current, SoC, etc.) that are communicated to the Inverter are also passed on to the Sol-Ark portal for reporting, and also for local monitoring via the MODBUS port to Solar Assistant or the like.

The Homegrid also is designed to be expandable, with up to 15 stacks tied to one BMS acting as a single battery. The stack sizes are also configurable. We chose a fully populated stack of 8 battery modules to yield a 48V 38.4 kWh system as that configuration provides the best match for our Sol-Ark 15k Inverter. When we add an another inverter, we can add a new battery stack as well and both double our energy storage capacity as well as double the load we can supply to loads.

Battery heat

A limitation of all batteries is that they do not perform well in the cold. LiFePO4 batteries in particular are very sensitive to charging in below freezing conditions (although they can be discharged). This could be a problem where we live in Alaska as the temperature can drop to -30. The typical advice is that batteries are as comfortable as you are, which leads many to place the batteries within the heat envelope of their living space. That used to be the case, when our batteries (and inverter) were located in our utility / laundry room. Now that they have moved to the garage a few additional precautions were in order. Firstly, I should note that our garage is insulated. There is a wood stove there, and the generator does produce heat when it is running, but there was no thermostatically controlled heater to protect that batteries. Homegrid does make an internally heated battery that seems reasonable, but it consumes up to 150W to keep the batteries above freezing. We opted instead to install a Toyo oil heater in the garage, set to a fairly low temperature (55 degrees F), that not only keeps the batteries warm, but also makes the generator that much easier to start, and the garage more pleasant to be in general.

More to come…

I hope this update has been interesting. The electrical improvements are making some even more impressive changes possible. Stay tuned…