Or “Pilot Reports.”  Number one.  Because this is the first of them on this blog.

Flying.  It’s awesome.  I do it.  I’m finally able to do more of it.  Keep reading for more!

Yes, I’m a Pilot

And you can’t make the joke about how you tell a pilot in a bar, don’t worry, he’ll tell you – well, at least, until now.  I figure a few years into the blog is long enough to mention such things in passing, especially when I’ve spent my Friday flying, my Saturday re-roofing an old church building, and haven’t had time to put together my more usual technical posts.  I’ve got plenty in the works, just none ready at this time, and I’ve had this percolating for a while.

I’ve actually had my pilot’s license for a hair over 9 years – I got it back in April, 2009.  I flew a lot in Iowa, and then didn’t really do much flying for a while.  It takes me a while to get settled in an area and have the time to really start flying again, and in some areas, I’m just not that comfortable flying.  I learned at an uncontrolled field, and the Seattle airspace, when I lived up there, was just sort of intimidating.  I never flew in it enough to be comfortable in it, and Seattle isn’t a great place to fly single engine anyway – it’s icing conditions in the (low) clouds a depressingly large part of the year, and… just, nope.  I don’t have my instrument rating yet.

Out in Idaho?  It’s a great part of the country to fly in, there are an absurd number of little backcountry strips (dirt, gravel, or grass strips, usually in out of the way places, mostly along rivers in canyons, generally with amazing fishing or camping) scattered around the state, and even if you don’t do that sort of flying, it’s the sort of state that flying makes sense in.  The weather is generally quite good for large parts of the year, and there are plenty of places that simply don’t have direct roads between “here” and “there.”  While a lot of Idaho is quite mountainous, there are also plenty of plains regions, and most of the mountains have valleys between them you can quite safely fly up.

This would be the sort of thing I see out where I live – big, flat, open plains, with snow capped mountains in the back.

Also, the whole commercial drone operator thing (“Part 107”)?  If you already have a pilot’s license, it’s literally an hour or so of some online slides and videos, take a quick quiz, and sit down with your CFI to sign off on the paperwork.  Super easy.  If you’ve got a pilot’s license, there’s no reason not to get your commercial drone operator cert.

Flying to Ogden, Utah for an Aquarium

The opening photo is actually of a Cessna 172 we flew to Ogden, Utah on Friday.  Why?  Well, my daughter has been pestering us to see a different aquarium than the one we usually go to, and there was a a SeaQuest (not DSV… sadly) in one of the malls out in Ogden.  It’s a weird place for an aquarium, but it’s actually a quite good aquarium, and we really enjoyed the trip out and back.  It’s the type of thing you can do with a small airplane that just isn’t easy to do any other way (unless you like an awful lot of time in a car).

Driving, Google says it’s about 5 hours – each way.  And that’s with I-84 running almost directly between Boise and Salt Lake City (seriously, we followed the interstate a large chunk of the way).

Flying, in the Cessna 172 we took (which cruises around 122kt, or 140mph), it was about 2.3 hours each way, with a bit of time on the ground.  Total time logged on the Hobbs (airframe time counter) was 4.9 hours, and that includes ground time, runup, etc.  I didn’t take the most direct route, either, because that would have put us directly over some mountains peaks, and there’s just no good reason to do that when there are perfectly good flatter, lower areas you can follow.  Also, a direct route would have put us over the Great Salt Lake for a good bit longer than I’m comfortable with in a single engine aircraft (especially with a pregnant wife and a 3 year old in the airplane), so we flew around the north edge to keep within easy gliding range of something hard.

The way out, in the morning, was silky smooth the entire way.  The way back, less so.  It wasn’t violently rough, but we were getting rocked around a bit by the mountains and the thermals.  You’d think a Cessna 172 wouldn’t be bothered that much by thermals, but we certainly were!  I had to shove the nose down and pull the throttle back to keep from overspeeding the prop (fixed pitch) quite a few times, and I also found a solid hunk of sinking air that dropped me several hundred feet despite trying to climb.  So… not nearly the nice ride back that it was on the way out.  However, I did get a chance to play in some very nice ridge lift (wind blowing against a mountain side and going up)!

For those who haven’t flown into a halfway decent sized airport in a Cessna, you can usually get a “crew car” out of the FBO (Fixed Base of Operations) you stop and and, ideally, fuel up at.  Crew cars are an interesting little beast, and the best you can normally say is that “they run.”  This FBO had a BMW 7 series – which seemed a good bit excessive as a crew car, until I saw it.  For a 115k mile BMW, it was rough.  The driver’s mirror was cracked, there was body damage on the front bumper, the seat belt and/or airbag system was malfunctioning, and it generally met my expectations for a crew car.  I was afraid it was going to be too nice!  You get nice crew cars at some places, but you usually pay about $1/gal more for fuel at those places…

Why SeaQuest instead of a larger one?  Two reasons.  First, 3 year old.  Their attention span for something like this is limited to a few hours, tops.  Second, I haven’t been doing enough flying in controlled airspace recently to really feel comfortable going into SLC and mixing it up with the jets.  Plus, their 100LL is something like $6/gallon, and the 172 I took is sort of a thirsty girl.

Trip Costs and Alternatives

The round trip, flying, cost us around $400, total.  I’m not going to claim general aviation is cheap, but remember, we took 3 people, could have carried 4, on our own route, seeing what we wanted to see on the way, on our own schedule.  I can find round trip flights into Salt Lake City from about $240, which, for three people, is $720.  Plus, that’s from a good bit further away (Boise vs Nampa), to a good bit further away.  The flight is only an hour, but the rest of the time more than makes up for it (TSA time, boarding, etc).  Plus, their schedule, versus ours, and having to get a rental car at the far end.
Driving, it’s around 650 miles.  At the IRS mileage rate of 54.5 cents per mile, that’s an estimated cost of $354.  Plus over twice the trip time.  Yes, I’m aware the IRS rate is higher than some people can operate a vehicle for (myself included), but it’s relevant to an awful lot of people with newer cars.
So, really, it’s quite comparable to other methods of taking a trip like this – even on literally a best case route (an interstate that goes between the two points).  Add another person (as will be the case in… oh, right about a month now), and the numbers look even better!

Cessna 182s

Another somewhat recent upgrade for me, flying-wise, is that I’m now checked out to fly the club Cessna 182s.  If you’re familiar with small high wing airplanes (which is what Cessna makes), they have three very common airplanes – the 152, the 172, and the 182.  The 152 is a fun little two seat trainer.  I love how they fly, but “gutless wonder” is, perhaps, generous.  They’re just enough airplane to get two people into the sky, they’re tiny (I’ve seen motorcycles with more space between people than you get in a 152), and they don’t have very long legs because they only carry about 20 gallons of fuel.  They don’t burn that much, but they’re not exactly fast, either.  My now-wife and I did a long cross country to visit family in a 152, and we we still liked each other after that enough to get married!
The Cessna 172 is the standard four place (really, two plus two – you can’t usually carry four adults, but you can fit two adults and two kids easily) Cessna.  They also have fixed pitch props, but they carry a useful amount of fuel (typically 40 gallons), have a useful payload, and are the bog standard Cessna you’ll find at every airport in America (and probably the world).  They’re the Honda Civic of airplanes (yes, Piper folks, I know Piper makes airplanes too, but you can’t deny that Cessnas are more common).  That’s what I learned to fly in, and they’re a pretty good “do everything” airplane.  They’re forgiving to fly, fairly easy to land, but the vast majority out there are set up as trainers, and aren’t great traveling airplanes as a result.
Then you get to the Cessna 182.  This is, according to many people, Cessna’s smallest “real airplane.”  They typically have a big snarling 6 cylinder Continental O-470, a constant speed prop (like a transmission for airplanes – it lets the engine be more efficient at a wide range of airspeeds), and enough payload capacity to haul four adults – or, alternately, two adults, two kids, and an awful lot of 100LL avgas (80 gallons is typical).  That leads to a wonderfully long range traveling airplane, and because it’s bigger and heavier, it’s a more stable platform as well.  They have more handy features like rudder trim, and they cruise a good bit faster than a 172, and will climb much more rapidly.  Plus, the constant speed prop helps them be a good bit more stable in cruise (a minor increase in airspeed isn’t met with a faster turning prop, which lets the engine make more power, etc).  Great, great airplanes.
And I’m checked out in them now.  There’s more to keep up with, but for long distance work?  They’re great.  They also are the sort of airplane that’s equally happy potting around the country in 500-600nm legs, or flying into little dirt strips in the middle of nowhere.  They’re not the most efficient airplane out there, but they’ll do just about anything you ask.
So, expect some PIREPs out of this blog about 182 travel at some point going forward!

Some Interesting Area Photos

Finally, I’ll end with a few photos I’ve taken from the air that I happen to like.  This is the Swan Falls Dam on the Snake River – one of many little hydro dams that power southwest Idaho.  This particular dam was installed, originally, to power Silver City – a former mining town up in the Owyhee mountains of Idaho.

Celebration Park, along the river, is another great place to spend time in the summer.

And one of many new solar installations in Idaho.  This one is out by Murphy, but they are popping up like weeds – there are two between Boise and Mountain Home, and they’re just going up everywhere.  I won’t say I see a new one ever single time I fly, but I’ve certainly discovered quite a few from the air!

So, that’s PIREP #1.  I hope to work a variety of others into the mix of posts over time as I do more worth talking about in the air.  One of these days, I even hope to perfect satellite connected Planegressing…

Continue readingPIREPs #1

Off Grid System Design Considerations and Battery Types

People are regularly surprised that my solar powered office is driven by a moderately sized bank of lead acid batteries instead of a fancy modern lithium bank.  I mean, I do work on lithium batteries regularly, and I could certainly build myself a large lithium bank at fairly minimal cost (compared to purchasing one, if I were willing to do that much spot welding).  But, in a deck box behind my office, I have a boring set of 8 Trojan T-105RE batteries in a 48V/225Ah (10.8kWh) bank.  Why?

Quite a few reasons – which I’m about to dive into!  Read on!

Off Grid Power

Before diving too far into the types of batteries used off grid, I’d like to define what I mean by off grid use.  Some of the characteristics I mention are of great value off grid, and of far less value for a grid tied system.
By off grid, I mean a battery system that is the primary energy store for a structure without a power grid connection.  So, a home without the power grid, a business without the power grid, a little solar powered shed without the power grid, etc.
In general, this sort of system is subject to daily cycles.  This means that the battery is charged and discharged (ideally both, but sometimes only discharged) every single day.  Or very close to it.  The system never has a chance to sit and rest, because it’s always under some sort of load (charge or discharge).  And, generally, it means there’s someone around who can pay attention to it.
An alternative I’ll mention occasionally is “cabin use” or “RV use.”  This refers to a system that is used sometimes (often on weekends or for a week at a time), but isn’t used for full time power.  For a cabin in the mountains, the system may get used a few days, and then will have a week or more (unattended) to charge.  For an RV, the same applies.  If you’re full timing in an RV, you have a power use much closer to a standard off grid system, so pay attention to those bits.

Lead Acid Batteries

As much as it stuns a lot of people to learn this, the vast majority of off grid power systems out there today use lead acid batteries.  Most of them use flooded lead acid (it sloshes when you move it), though there are certainly some systems using sealed batteries (typically AGM).  They work – quite well, in fact.  And they have some characteristics that are very well suited to an off grid power system, especially with some of the new developments.
I’ve written a post about keeping lead acid batteries alive in an off grid power system, which you may find very helpful if you’re using them, or if you’re interested in their quirks and how to keep them alive.

In general, lead acid batteries really like being fully charged, and the system should be designed around keeping them as fully charged as possible, as much as possible.  The various lead carbon blends reduce the requirements for this somewhat, but it’s still helpful to charge them fully and regularly – regardless of the type.

Types of Lead Acid Batteries

There are a few different axis you can categorize lead acid batteries on.  Some are great for off grid use, some are reasonably terrible for off grid use.  It’s important to understand this, because using the wrong type of battery for the use case will generally work incredibly poorly, and lead to a uselessly short battery life (and will make off grid power even more expensive than it already is).

Common to all lead acid variants is that they really, really like to be fully charged – regularly.  If lead acid isn’t fully charged quite often, they suffer (to varying degrees) from sulfation.  In the context of battery longevity, sulfation refers to the hard sulfation that builds up on plates, blocking active area and preventing the battery from delivering rated power or capacity.  If you’re interested in more details on lead acid, I’ll refer you to my detailed post on lead acid.

One problem for off grid use with lead acid is that they don’t charge instantly.  To properly charge lead acid, pump current in until the voltage is at a particular temperature compensated value, then hold it there until the charge current drops off.  A “quick” charge cycle for my bank is 5-6 hours from sun up until I drop to float, and that can be a good bit longer (or even all day long) if I don’t get as much sun.

Flooded Lead Acid (FLA)

The most common type of lead acid battery out there, especially for off grid, is flooded lead acid.  This has plates of lead and lead dioxide dunked in a liquid sulfuric acid (“battery acid”) electrolyte.  If it sloshes, it’s flooded lead acid.  They’re very robust and hard to damage, but they require good ventilation as they offgas hydrogen and oxygen during charging, and then watering to make up that hydrogen and oxygen.  The oxygen isn’t a problem, but you don’t want hydrogen building up in enclosed spaces.  It’ll ignite if you look at it wrong (seriously, the ignition energy for hydrogen is frighteningly low), and hydrogen explosions are no fun.

Flooded lead acid is best for a daily cycled system that can get regular maintenance.  They need watering every few months, and if they’re left fully discharged, they can freeze in the cold (as well as badly sulfating the plates up).  However, they’re incredibly tolerant of being moderately overcharged (useful as the string gets older to help keep things balanced), and you can get the “ground truth” about the health of each cell by measuring the specific gravity.  There’s no need to rely on voltages and guesses to figure out battery health.

Absorbed Glass Mat (AGM)

The other common type of lead acid battery (and certainly more common in some uses) is Absorbed Glass Mat.  This type still has the lead plates, but instead of liquid electrolyte sloshing around, the spaces between the plates are filled with a weave of glass fibers, and the electrolyte is soaked into these.  The weave isn’t fully saturated, though, so the acid remains fixed in place.  There’s no liquid to slosh around, which means these batteries can be mounted in almost any orientation, and they don’t (typically) release gasses during charging.  The cells are designed such that the hydrogen and oxygen can recombine in the cell if the charging proceeds at normal rates.  There are vents that will release an overpressure, but they shouldn’t trigger during normal operation.
AGM has some advantages over flooded lead acid for certain uses.  They tend to have a lower internal resistance, which makes for better power delivery and faster charging.  They’re maintenance free (“maintenance impossible”), which can be nice if they’re sitting unused for part of the year.  They also do somewhat better in cold temperatures, but it’s still a bad idea to let them freeze.  And, if you have varying battery angles (say, a sailboat), they won’t spill.

However, they’re more expensive than flooded, typically have a shorter service life (compared to comparably priced flooded), and since you can’t measure the specific gravity (only voltage), it’s harder to make sure that they’re being fully charged.

Gel/”Lead Crystal”/”Lead Silicon”

The final variant of lead acid is the gelled electrolyte type.  This involves some additive in the electrolyte that turns it into a stiff jelly instead of a liquid.  It’s similar to AGM in many ways (orientation independence, maintenance free/impossible), but it’s a lot toucher about charging voltages, and it doesn’t work very well in the cold.  They’re somewhat more than a “gimmick” battery, but an awful lot of the marketing makes them seem like a gimmick.
They have no business in an off grid power system of any sort, and if you run across some quirky sounding lead based battery with slick marketing, it’s probably a gel battery – and you should avoid it.

Lead Carbon

Lead carbon isn’t actually a type of lead acid, but it’s something you’ll see in the marketing.  It refers to blends of lead and carbon, usually on the negative plate, that help prevent hard sulfation from building up.  It’s a good thing for an off grid power system, because it means that if the battery doesn’t get fully charged during some of the winter months, it shouldn’t cause the same sort of capacity loss and damage that you might see without the carbon.
The problem with this sort of technology is that it takes an awful lot of years, in real world conditions, to find out how good it is.  The shorter term tests are very promising, but it’s not until people wear out batteries in the real world that we see how good things really are.
For an off grid system, I think the various carbon additives in the plates are worth the slight extra cost to experiment with, though.

Lithium Ion Batteries

Moving on from the ancient lead acid technology, the next battery technology I’m covering is lithium ion.
I rebuild lithium ion packs for people, and I’m quite familiar with the characteristics of lithium batteries.  They’re absolutely an established technology, and by traditional battery terms, they are impressive.  The energy density (kWh) is awesome, the power density (W) is awesome, and the cycle life ranges from “meh” to “wowzers,” depending heavily on the particular chemistry involved and how they’re treated.  The lithium ion cells people are most familiar with are 18650s – 18mm in diameter, 65mm long.  And a cylinder.

There is an incredible amount of research and investment going into the various lithium ion chemistries, and they’re quite exciting for many applications.  The modern wave of electric cars wouldn’t be possible without them, cell phones rely on them, and pretty much any long lasting device uses lithium ion cells of one chemistry or another.  They also charge quite rapidly and don’t need any sort of “absorb” cycle during charge.  Just pump current into them and don’t exceed the fully charged voltage (you’ll get a tapering of charge current, but it takes far less time than lead acid to fully charge).

We’re seeing more and more lithium ion for stationary power storage, but they’re still not particularly common in off grid systems – and, I’ll argue, most of the variants aren’t a great fit for off grid power storage for a few reasons.

Characteristics of Lithium Ion

Lithium ion batteries are amazing for portable energy storage – cell phones, laptops, and electric cars wouldn’t be anywhere near where they are today without lithium ion batteries.  They store a ton of energy, they can release it incredibly quickly, and they can maintain a reasonable cycle life while doing this.

But, all that said, they have a few characteristics that make them a bit annoying to use in an off grid system.

The first is that they’re most stressed when fully charged and when fully empty (or deeply discharged – you really shouldn’t let a normal lithium chemistry below about 2.5V/cell, and there’s no good reason to be down even that low).  Sitting fully charged (4.2V/cell or 4.35V/cell, depending on the chemistry, though some go even higher) is really hard on lithium cells – and an off grid power system is likely to be doing this for a large part of the year if it’s been designed for year round use.

The second is that lithium ion cells require good thermal management.  Getting them hot really hurts cycle life (which is true of lead acid as well, but lead acid has an awful lot more thermal mass to prevent them from getting hot) – but being hot and fully charged, as one might find in the summer with long, hot days of sun, just slaughters them.  In addition, they can’t be safely charged if they’re below about 45F (7C).  You get lithium plating and that’s permanent capacity loss.

So, what this means for off grid use is that you really either need to have the batteries inside where they’re thermally regulated, or you need a separate heating/cooling system for them that keeps them in a safe temperature range.  This is part of why some of the packs out there have cooling loops.

Thermal Runaway and Venting

Lithium chemistries also have a particularly nasty failure mode called “thermal runaway.”  If they get too hot for whatever reason, they start degrading internally at an ever-increasing rate – which releases energy.  Once they get hot enough, they start gassing their electrolyte and melting their internal bits, which gets really exciting, really fast.  The electrolyte gasses are typically flammable, and it’s not at all uncommon to have molten bits of metal coming out the vent as well – so you get a nice little torch.  This failure mode releases enough energy to propagate to other nearby cells, and YouTube has many videos of this happening (either intentionally or by mistake).
There’s a great paper from 2014 comparing the failure modes of a few different chemistries, and the takeaway should be that you shouldn’t use LiCoO2 anywhere.  LiFePO4 is far better, and the other more recent lithium chemistries sit somewhere in between.
The takeaway here should be that you really want to use LiFePO4 if you’re going with non thermally managed lithium.  Alternately, a pack that does proper thermal management and can prevent runaway.  Yes, I know the “DIY Powerwall” folks do things differently.  No, I don’t think that’s a good idea.  Yes, I think they’re building hazardous fireballs waiting for a place to happen.  Don’t build one of those monstrosities out of junk cells if you plan to keep it within fire spread range of where you sleep.

Slick All-in-One Lithium Packs

You know, “Powerwall” type devices.  If you have the money for one, have the tolerance for the unknown unknowns of one, and can find someone to install one for you in an off grid setup, have a ball.  I don’t treat them as a particularly useful option because they’re almost always seriously short on documentation, and are not (near as I can tell) particularly configurable.  They’re a black box, and I’m not really OK with those in an off grid system I have to maintain myself.  If you can find one with open documentation such that one can interface with it externally (or, ideally, that drives the voltages like a lead acid pack so you don’t even need to talk to it if you don’t want), let me know.  Until then, I stand by my view that they’re likely to be more trouble than they’re worth, off grid.
If you’re going to insist, in the comments, that they’re the future and I should promote them, please answer one question first: How much energy does a Powerwall consume in the winter to stay warm?  Or in the summer to stay cool?  That type of thing matters for off grid use.

Nickel Iron (“Edison Cells”)

Finally, I want to touch on the nickel iron chemistry.  On paper, it looks amazing.  They last nearly forever, don’t care if you overcharge them, aren’t hurt by sitting drained, and are the sort of robust battery that means you can probably buy them for life.
Which… is mostly true.  The problem is that they’re actually quite the royal pain to actually live with.  There are a few people over on SolarPanelTalk who have the NiFe cells, and they try, ever so hard, to like them – but they mostly find them annoying.
The major downsides of NiFe are that they’re insanely inefficient (in terms of round trip energy), they absolutely guzzle water, they’re quite loud when charging, and they suffer (fairly badly) from carbonate poisoning as they absorb carbon from the air.
“Round trip efficiency” means “Power out versus power in” – in watt-hours, not amp-hours.  NiFe cells have a much higher charging voltage than discharge voltage, percentage-wise, and they’re pretty good at gassing during charge (which isn’t “useful” from a charge perspective).  If you get about 60%, you’re doing well.  This means that you need more solar, or more generator use, to get a certain amount of energy out of the system.
The aggressive gassing during charge means that the battery bank is insanely loud (as in, “You should have hearing protection to work around it during charging”).  This also means that a good sized bank will go through many gallons of distilled water a month (you may wish to consider running your own distiller if you have a NiFe bank).  A good sized lead acid bank will go through a few gallons a year.
On top of that, while you can replace the electrolyte every decade or so (and have to), there’s no particularly good place to dump a lot of hazmat (potassium hydroxide and lithium hydroxide).
Plus, the up front costs are staggering.
It’s useful in some situations (I might use them for a mountaintop power outpost if I could find an automatic watering system I trusted), but it’s just not a good general purpose off grid solution for most people, in most systems.

Off Grid Seasons

Before diving into what I think the best battery choice is, I need to discuss the four seasons of off grid power and how the system will typically run – because this directly feeds into why I make the suggestions I do.

Solar Summer

Most people assume that summer is the best for an off grid power system due to the long, sunny days.  It’s actually a bit harder than it seems, at least if you have any sort of mechanical refrigeration (“Air conditioning”) powered from solar.

Yes, the days are long, but the days are also very hot!  As solar panels heat up, they produce less power (at least if you have a MPPT charge controller, which you should).  My panels lose output as they heat up at the rate of 0.41%/C – so on a hot day (40C), with the panels baking in the sun (say, 60C), they produce 20% less power than on a cool, breezy day with the panels at 10C.  You notice the difference.  Plus, summer time tends to have less rain (at least in a lot of the country), so the panels are dirty.  And, worse, in the west, you get wildfires.  That cruds up the sky quite badly, and the smoke coats the panels.  By the fall, my panels are absolutely disgusting, and I can tell the difference in power output.  I’d say they are easily 25-30% down on power by the first rains of fall.

Solar Spring/Fall

Spring and fall are, at least for me, the easiest times of year.  The days are shorter, but the skies are clear, the panels are cool, and I need far less in the way of cooling.  I have plenty of power to ride through the night (admittedly, my night use is fairly low), and there’s a ton of surplus to throw at diversion loads.  Plus, we get rain often enough that my panels are clean.  I don’t clean my panels other than letting them sit outside in the rain, so this makes a difference.  The jump in production from the rain of fall is huge.

For the sunny three seasons (spring, summer, and fall), your battery bank is likely to be fully charged, almost every day.  This means that the battery bank will be spending a lot of time full – which, if you recall, is something lead acid really likes doing!

Solar Winter

Sucks.  You have a backup generator and charger, right?
Winter is the worst time for solar, obviously, but I’d thought I could overpanel my way through it.  Nope!  We get inversions in the winter, which roughly translates to “A week with no sun and no wind.”  You’ll need some variety of backup heat.  I use propane, though one could just as easily use biomass (a wood stove or pellet stove).  

Getting through the winter is the challenge.  If you can do that, the rest of the year is easy.  For winter, the battery bank will be drained a lot more than the rest of the year.  This is a time when lithium’s tolerance of being partially charged is really handy, but it’s only a month or so when it’s really a big problem in most areas.

Panels: Quantity and Orientation

Assuming a heavily solar based system, what should one do with panels and how should one orient them?
My advice would be to put up somewhat more panel than you think you need – in two orientations.
For the bulk of your charging, you should go with south facing panels.  However, absolute max theoretical production is not useful to an off grid system.  You don’t have the grid to soak your power, so what’s far more useful is to have panels that extend your day as long as possible – with a focus on morning power.

East Facing Panels Rock!

After I built my main solar arrays, I had two panels left over, and for lack of any better ideas for them, I built a set of “morning panels” on the east wall of my shed.  They swing south if needed, but mostly they sit docked on the wall and soak up the morning sun.

I assumed these would be generally helpful in the morning – but I didn’t realize just how amazing they would be!  On a typical morning, I get 1-2 hours in which these panels are outproducing my main array – on an absolute basis.  Not producing more per panel – these two panels outproduce all 8 main panels, in terms of amps into the battery bank.  It’s amazing, and what it means is that I have a significantly longer “solar day” to help get my battery bank charged up than I do with just the main array.

When you’re trying to recover from a week of generator use on the first sunny day you get, this sort of thing makes a huge difference in the winter!  And even in the summer, it reduces the time of discharge on my battery bank by 1-2 hours, every single day.  These are absolutely worth it!

Consider a Split Bank

If you have enough area to put panels (and are ground mounting), doing a split panel bank (some facing southeast, some facing southwest) would be worth considering as well.  You can’t do this with the different directions in series, but as long as each direction is in parallel with the other direction, it’ll work fine, and you’ll extend your solar production by a good amount.  Evening production just doesn’t matter as much, since your bank should be full by then, so if you have to pick one, go with a southeast panel facing.

Panel Angle

Another area to think about (if you have the option) is the panel angle to the ground.  Ideally, you’d swing the panels to the sun angle in the sky, which I can do, but it turns out that with enough panels for some of winter, I’m so overpowered most of the year that it doesn’t matter that much.
If you get snow, vertical panels are awesome in the winter.  They shed snow wonderfully!  As soon as they get any light on them at all, they start heating up through the snow, and the snow comes sliding off.  It’s fun to watch, and beats manually clearing snow!  Even though you lose a bit of theoretical production compared to a 20-30 degree angle, the snow clearing aspect of a vertical panel is worth it!


I talk about overpaneling a lot, and it solves a lot of problems with solar.  My system is rather heavily overpaneled (2850W of panels hung for a 300-500W running load during the day), but what it means is that I can rely on solar, without having to run my generator, even on cloudy days most of the year.  I’m able to fully charge the batteries, which is key to longevity, quite often.  It doesn’t work all winter long, but it works wonderfully most of the year!
If your battery bank can’t tolerate the full charge of your panels, no problem.  A good MPPT charge controller will let you limit the max current into the battery bank, which means that you can use more panel area than might be otherwise allowed, and still avoid abusing the battery bank too badly.  Splitting your panels (some southeast, some southwest) is another good way to increase panel area without increasing peak charging current.
What a lot of people don’t realize is that while having panels pointed at the sun is helpful when there’s sun, on a cloudy day, the only thing that matters is raw panel area.  I’ve had many days when swinging my morning panels from east to southwest changes the charge current by no more than 10% – min to max.  For clouds, panel area is king!

So… What’s the Best Off Grid Battery?

Based on all this, what should you pick as your off grid battery?
Well, unpopular though it is in some circles, it’s really hard to beat good old flooded lead acid for a daily cycled system!  They have a wonderfully long lifespan if treated properly, and they absolute love the 9 months of the year when they can be fully charged daily (or nearly so).  Lithium is quite stressed in this situation, which a properly designed system will produce on a very regular basis!
For something like a cabin that’s used on the weekends, you might consider AGM for the lower maintenance requirements.  It depends on the use cases – if you’re up there regularly, flooded is fine, but you need to be careful to not let it deeply discharge during the winter (or it will freeze and burst the case – which is just a nasty mess in the spring).  AGM is going to be more expensive for a daily cycled system, and I don’t suggest it when flooded will work, but it’s a good compromise for infrequent use.
If you do insist on lithium, LiFePO4 is the way to go, but back off the charging voltages in the summer.  It’s a robust chemistry, but sitting fully charged, especially while hot, does wear them out quickly.  It’s rare to find anyone with this type of bank really getting rated life out of the pack, but if you can keep them cool and charge to 80% or so in the summer, they should be fine.  Make sure they’re warm when you charge them in the winter, though!
And, whatever you do, consider pointing your panels something other than due south.  More panel area is useful, and splitting the panel aiming will lead to a much more useful power curve as compared to everything facing due south.
Hopefully this helps out a bit with your system design!  Good luck!

Continue readingOff Grid System Design Considerations and Battery Types

On the Art of Repair: Re-Capacitoring an Old Mainboard

One of the misconceptions about electronics is that if there are no moving parts, the system should last forever.  This is closer to true than not, but one really common item that does wear out on many systems are capacitors.  Electrolytic capacitors are the usual problem, but, fear not!  You can repair them easily and at home!  Beyond just repairing a mainboard, this week is a bit of a discussion on the art of repair, and why you should get good at it.

Get out your soldering iron and join me!

Mainboard Failure and Symptoms

This particular mainboard has been in service since around 2008 in my main home server.  It’s an Abit IB9, an old Core 2 Duo 1.86GHz chip, and a whopping 4GB of RAM.  The CPU was the cheapest one I could find, but this was my home workhorse for many years (it made it a decade, and I haven’t been gentle on it).  But, one day a few months ago, I noticed my fileserver was not responding.  I power cycled it, and that didn’t bring it back on the network, so I dragged it out and attached a monitor to it.
At which point things started to make less and less sense.  It would start booting, but fail to find the device for the root filesystem – which was on the same drive it had just loaded the kernel and initrd from.  Weird.  Before I could puzzle this particular issue out, though, the problem got worse and it simply stopped finding the IDE drives at all.  The IDE controller didn’t even show up in the boot sequence trying to enumerate the drives.  Sigh.  I ordered some replacement parts for the server to bring it up to modern specs, since an ancient CPU, 4GB, and some old IDE drives just don’t work very well for what I’ve been asking of it lately.
I pulled the board out of the system, and tried to boot it without anything attached.  Sometimes it would boot, sometimes not.  Nothing reliable, and nothing particularly repeatable.  Obviously, it was toast.

And then I saw the capacitors.  Now, if you’re not one who stares at electronics regularly, take a look at the capacitors (the round things sticking up) in the upper left.  Now compare them to the ones at the bottom and along the right.  See how the tops are bulging up, and there’s crud leaking out?  They didn’t come from the factory like that!

Looking around the board, I found more bad ones in the center section.  These are all 6.3V, 1500uF capacitors – and about half of them on the board are visibly bad (bulging and leaking).  That’s generally not a good sign, and capacitors (of this size) are usually related to power supply regulation and powering things.  Like an IDE controller!

The Capacitor Plague of the Early 2000s

If you were working with electronics in the early 2000s, this issue probably sounds familiar.  There were a ton of electronics that failed due to bad capacitors in the 1999-2005 era, and it was a bad enough problem that it has it’s own Wikipedia pageEverything was affected by what, as near as researchers can tell, was industrial espionage gone wrong (or right, in that the stolen formula was incomplete).
During that era, it was common to be able to buy “failed electronics” for remarkably little, replace the capacitors with ones that didn’t suck, and go on your way with a screaming deal on a computer, TV, etc.  At least if you were one of the people replacing capacitors.  If not, you were probably stuck replacing things, or paying one of those people a hefty chunk of change to fix your stuff.
But, what it boils down to is that if your capacitors have failed (which some of mine very obviously have), it’s not that big a deal to replace them.  Even if they haven’t obviously failed, capacitors age with time, and it’s a good idea to understand the process of replacing the failed ones on equipment that has gotten flakey, especially on the power front.  If we do hit an era in which new equipment is harder to come by (likely during the decline of an empire), those who can fix things are likely to be in demand – and those skills are seriously missing in our culture today.

Replacing Capacitors

In general, solid state capacitors don’t tend to have issues (which is most of the surface mount ones).  The ones that (usually) fail are the electrolyte ones, and on most systems, they’re through-hole mounted!  This means that they have a pair of pins that go all the way through the circuit board to the backside, where they’re soldered in place.  It also means that they’re the easiest style of component to replace!
You can see quite a few rows of through-hole mounted components here – they’re all the little legs sticking up through the back of the board (this is the back of the mainboard).  Replacing them involves removing them, cleaning out the holes, and putting new ones in.

Of course, one must first figure out what capacitors one is replacing and buy some replacements.  I decided, based on the visible failure of 5 of the 10 of one particular type, that I’d replace all of those.  The failed ones are entirely 6.3V 1500uF capacitors, and eBay is a great source for replacement capacitors (though I’d stay clear of the incredibly cheap ones).  Just make sure the diameter is the same, and the height is close (it usually doesn’t matter, but sometimes there are clearance issues with taller ones).  If you really wanted to hunt down the detailed specifications of the failed ones (ESR, ripple current rating, etc), you could, but my experience has been that “close” is more than good enough for capacitors.  Any decent board will be designed so that the exact ratings don’t matter much on the capacitors that change as they age.

Removing a capacitor is fairly easy.  Get your soldering iron hotter than you think is reasonable.  This is lead free solder (which I hate), so you need a hotter iron than you might use for proper solder.  I normally keep my iron around 250C for lead solder, 300C if it’s a big surface, and for lead free I crank it up to 350C or a bit higher.  You’ll want to use the heaviest tip you can, because a lot of capacitors will have one leg in a ground plane, which will soak a ton of heat.  I’ve met some things I can’t remove from a ground plane, but hopefully you can get the capacitors out.

Put a bit of fresh solder on the tip for heat transfer, and go back and forth melting the solder on the two legs.  You can either get both legs molten, or do one at a time and “rock” the capacitor out.  Eventually, it should come free!

I’ve tested a few of these, and the bulging ones are less than 10% of rated capacity (about 130uF instead of 1500uF).  That’s enough to cause problems…

After the legs pop out, you’ll almost certainly have a hole filled with solder.  To resolve this, use your handy desoldering braid, flux the joint up, put a bit of solder on the braid, and it should suck the solder right of the holes.  You may have to try a few times, but it should work quite reliably.

If that doesn’t work, you can always use a solder sucker, but that runs a non-trivial chance of ripping traces out and trashing the interior plating on the hole, so… use a desoldering braid.

Installing the new capacitors is easy – just solder them in place.  With your proper 60/40 lead solder.  One thing to be aware of: Capacitors are directional!  You must make sure you get the negative on the proper side, or the new capacitors are likely to blow up or, in the best case, just not work.  On this board, the negative side has white silk screening, but pay attention to the alignment when you pull the old ones out and match it.  Capacitors typically have a stripe with “-” along it indicating the negative pin.

You can see my shiny new caps here, next to one of the old ones (the brown one at the top).  That one hasn’t failed (yet), but you can see that the new ones are slightly taller.  It’s not a problem for this board, so I don’t care.  Both of the capacitors I’ve removed are leaking.  They haven’t blown the bottom rubber plug out, but they’re both toast.  Luckily, they’ve only been venting on the top, and haven’t blown crud down onto the board.  That’s no good at all!  If your capacitors have blown gunk onto the board, you’ve got to clean it up – that stuff can cause short circuits and literally burn up your mainboard.

<The Count>Six!  Six new capacitors!  Ah hah hah hah!</>  Yes, I have a toddler.  Why do you ask?

The other four I replaced are in the center of the board, right around the BIOS chip (socketed for your hacking enjoyment, if you’re the kind of person who keeps chip readers around and enjoys messing with BIOS images).  Yes, the mainboard is really dusty.  I blew it off after taking these pictures and realizing just how bad it was.

I’m left with a pile of 10 removed capacitors.  Five of them are definitely shot, but I wasn’t about to give the rest the benefit of the doubt since they’re obviously the same type.  Overall cost?  About $3 (I ordered more capacitors than I needed, because… well, who doesn’t need more capacitors?) and about an hour of time to replace them.  You might need less – any blog project takes longer because I’m taking photos and interrupting my work cycles.

The Results

After replacing all the capacitors, I powered the board on with a drive attached to the IDE controller…
And it properly recognized the drive!  I rebooted a number of times, and it rebooted cleanly without hanging.  Everything is back to working, as near as I can tell!  Yay!

On Repair

With the board working again, I’d like to dive into the (increasingly lost) art of repair, and offer some advice and encouragement for my readers on the advantages of learning to fix things.

I’ve been doing deep level repairs on computers and cars for the past 15 years or so, and it’s been very beneficial.  I’ve been able to get some awesome computers over the years for nearly nothing, I’ve been able to keep my cars running for very little money (when money is tight, that’s a useful skill), and I’ve generally been able to build a reputation as a “Fixer of Things” that’s served me very well.
I can even tell you the event that started it.  Back a long time ago in undergrad, my roommate and I picked up some old Powerbook 520s from the university surplus sale (a dangerous place for what little money one might have).  They worked well enough, until one day, for some reason or another, a can of soda ended up dripping through the keyboard of one, filtering down through the rest of the components.  And the unit stopped working.  After much debate about how one might repair this and if it was worth trying (neither of us having any sort of laptop repair skills in the early 2000s), we realized something that’s been fundamental over the years since.
If it’s already broken, you can’t possibly make it worse!  This isn’t something I grew up with, but it’s something that, once grasped, is the key to getting good at repair.  If some gizmo is not working properly, the absolute worst you can do with a failed repair is… have it not working properly.  But you almost certainly learn something in the process, even if the repair doesn’t work (I’ve learned the most from repairs that failed initially and I had to bodge into working condition – IDE cables are a wonderful source of very fine bits of wire, once you destrand the individual conductors), and that builds your knowledge base for future repairs.  Iterate on this over time, and you get some awesome skills built up that let you tackle things nobody else around you will even dream of doing!

The Value of Broken Things

Quick: How much is a broken laptop worth?  Let’s say the power jack broke internally and it won’t charge anymore.  If you’re most people, that’s a catastrophic failure and the laptop isn’t worth anything at that point, because it doesn’t work after the battery discharges.  If you’re handy with a soldering iron and have a set of geekdrivers (those tiny little screwdrivers you need to work on laptops and phones), that laptop is an hour or two of work away from being almost entirely functional again.  I’ve repaired an awful lot of broken power jacks (the Pentium 4 era laptops were horrible about melting the solder away from the center pin where it went into the mainboard), and for a while, I got a steady stream of used laptops in from people with broken ones who would give them to me for free, or barely more than free.
Even if the laptop doesn’t work for deeper issues, it’s still worth something.  The screen is probably still good (or maybe it’s not, and the rest is).  The keyboard probably still works.  The CD drive is probably fine.  And on and on.  Even a badly shattered laptop that’s fallen out of a car has some parts of value left on it.  If you can build up some common parts for popular laptops of the time, it’s often possible to mash a few together and get something that works.  I ran around with a MacBook for a while that had a mismatched screen and body (I think it was a white screen and a black body) – because I’d been able to scrounge enough broken ones that I could put a working one together.
My favorite repair in this category revolved around an old 12″ Powerbook G4.  Remember those sweet little machines?  This one had what, for nearly everyone who looked at it, seemed to be a catastrophic failure.  It had bad display corruption on power on, and refused to boot more than a few seconds before kernel panicing.  Some analysis indicated that it was likely the onboard RAM chip having gone bad (the machine had 256MB onboard and a slot that could take up to 1GB).  What do you do with bad onboard RAM?
At the time, I didn’t have surface mount soldering skills (or an iron to do so).  So, I did the next best thing and simply disconnected the power for the onboard RAM.  Those Powerbooks were a royal pain to take apart (about a million tiny screws), but I got it apart, found the RAM chips, verified which pins that were the power pins, separated them all with a tiny knife and pick, stuffed some external RAM in, booted it, and… hey!  It worked!  And I had a 12″ Powerbook, nearly free except for the 2 hours it took me to disassemble, analyze, repair, and fix the problem.  It was broken already, the only “official” repair for it was a whole new mainboard, and there was really no risk in trying to fix it this other way – if it failed, well, I’d have to find a new mainboard, which is exactly where I started.

Repairing Cars

Have you ever had one of those deals where both parties think the other side is a sucker for taking the deal?  That describes an awful lot of the cars I’ve owned in college and post-college.  In the days before Cash for Clunkers ruined the used car market, you could find some properly cheap cars.  In order of cost, from cheapest on up, I’ve paid $100, $150, $200, and $350 for cars that more or less ran.  The $350 car was one that involved spending the morning at a junkyard trying to get it running, failing, going back home, removing parts from my other car (that had just tossed a timing belt – non-interference engine so it wasn’t a big deal to get running again), and bumming a ride back to the junkyard.  The people behind the counter thought I was some variety of idiot for spending my day at the junkyard, in the mud, to get this old beat up Subaru wagon out.  I thought they were silly for selling me an ’87 Subaru GL wagon, with dual range, for $350.  Yeah, it pissed fuel and the brakes didn’t work well (you had to pump them a few times to get pressure), but they didn’t realize that the leak was from the fuel pump (which was on a separate plate in front of the fuel tank) and that the rear brakes were just comically out of adjustment because this particular model didn’t have automatic clearance adjusters on the rear brakes.  I didn’t tell them these things, either.
Over the years, people have asked me how I learned to work on cars (mostly when I lived places where most people didn’t work on their cars).  I didn’t grow up with it.  I learned it because, quite honestly, I didn’t like walking that much.  I bicycled a ton as well, even through the winter (studded snow tires on a bike are incredible), but if I wanted a running car, there was a long period of my life when the only person who could make that happen was me.  A shop bill wasn’t an option on my budget.  So I learned.  I’d buy a Chiltons or Haynes manual for a car, and dive in.  They’re really helpful guides, and this was long before YouTube had videos for every car under the sun (that’s one area where video can be helpful).  I’d go hang out with other people who were doing things on their cars and learn.  I’d find forums for cars and read.  The old Subaru forums were awesome, because if there was a way to fix the car with bubble gum and a half piece of duct tape, there were about 5 writeups of it.  Such a lovable bunch of cheap rascals.  General wisdom was that if you bought an EA82 Subaru with “a blown motor cuz the timing belt failed” for $100, you really should get a friend to help you tow it around the corner before you put a new timing belt on and drive it home.  There were a few… incidents… when someone bought a car for $100, spent the half hour to put a new timing belt on it, and drove it off.  The old Subaru motors, unusually for the time, were non-interference engines.  That means that if the timing belt fails, the pistons and valves don’t collide – which means that the repair for a failed timing belt is to simply put a new timing belt on, retime the cams, and drive off.  Most engines were interference engines at the time, and so people would get faulty advice from mechanics.  Their loss!
I hate to dive too far into the personal finance realm (it’s badly overdone, and there are only so many ways to say, “Don’t spend more than you make”), but if you’re on a tight budget and need to own a car, one of the best financial decisions you can make is to learn how to fix your own car.  The vast majority of repairs can be done at home, or in your apartment parking lot (yes, I know that the lease usually says you can’t do that, and I’ve done an awful lot of repair work in those sort of lots – just don’t be obvious about the jack stands if you have the suspension torn apart, and maybe park so the wheel you’re working on is the least visible one).

Getting Started

So, what do you need to get started?  In any realm you choose to start fixing things in, you’ll need the proper set of tools.  For electronics, a set of geek screwdrivers is a requirement, as is a good soldering station.  I’m a huge fan of the iFixit guys – and if you can swing $60, get the iFixit Pro Toolkit.  It’s got just about everything you might need for modern electronics repair.  If you can’t afford $60, try the $20 Essentials kit.
You’ll also need a temperature controlled soldering station.  Yes, you can do some repair work with a Radio Shack woodburner, but a good station makes a huge difference.  I had an Aoyue 936 for quite a while, and I did a ton with that – including some surface mount repair.  I’ve recently upgraded to an Aoyue 968A ($175 on eBay).  This is a far, far nicer station with a solder fume extractor (no more stream of flux smoke in your face) and a hot air rework gun.  It has a digital temperature readout for the various hot bits, and it’s fast.  It gets up to temperature in under 15 seconds and comes with about a dozen tips for all sorts of work.  I really, really like this unit.

You’ll also need some solder wick, some flux, and (obviously) some solder.  Keep your sanity intact and buy lead based solder.  So much easier to work with.  If you have a Micro Center nearby, they may have an amazing DIY electronics section by now!

For car work, if you don’t have any tools, pick up a Mechanic’s Tool Set on eBay (or at your local parts store).  This will take you an awful long way to car repair.  If you don’t have a torque wrench or an adequately calibrated arm, get one of those as well so you can torque fasteners properly.

And then, as you run into things you need for projects, go buy it.  I don’t feel guilty about buying tools for repair work.  In almost every case, the cost of the tool is far less than the cost of having someone else do the repair, so I still come out ahead – and I have a new tool!  New tool day is exciting…

So – Get Fixing!

Hopefully this encourages you to go fix something!  Beyond just fixing your own things, fixing other people’s things is awesome.  And, if we truly are an empire in decline, these sorts of skills will be ever increasingly in demand.  If you’ve got the tools and the skills, you’ll be a very valuable person in times of economic contraction when people need things to last longer.
And, on that note, I’ll end with one of my favorite memes, created by a friend many years ago.  Be that guy!

Continue readingOn the Art of Repair: Re-Capacitoring an Old Mainboard