Articles

GM 8L90 Transmission – P0711

Summary

This is another automotive post on my journey through P0711 on my GM 8L90 transmission for my 2015 Yukon. It is on the K2XX platform. At 5 years old it is starting to get some mileage and wear as seen in GMT K2XX Magnetic Ride Control Shock Inspection.

Background

Monday morning I was out for a drive. With COVID all around us, I find I don’t get out of the house often. My truck still needs some miles on it or it would just sit for a while. All of a sudden I look down and notice the check engine light. I think this is the first time I have actually seen one on my truck. Despite being 5 years old, it only has 38k miles on it.

I’m fairly technical and like to do my own diagnostics when possible. I do it so that I can speak intelligently to the repair shop(dealer in this case). My truck was in the shop a few days on and off last week for some tire issues so I also wanted to avoid an unnecessary visit.

Diagnostic

Recently I acquired a Foxwell NT510 Elite. Its a pretty useful bidirectional scan tool. Bidirectional means it can not only read codes but it can do active tests as well and minor calibration resets. It seems to have most of the features of the more expensive ones but it is locked/licensed by the vehicle brand and usually comes with one brand free. In my case GM. You can purchase other brands and add them to the unit. On this model its a lifetime purchase which is nice.

In my case, I checked the codes starting with the ECM and had a P0700 – Transmission Control Module Requested MIL Illumination. This simply told me to check the Transmission Control Module which showed me P0711 – Transmission Fluid Temperature Sensor Performance. Here is a good link someone on www.gm-trucks.com shared with me – https://www.dtcdecode.com/GMC/P0711

The TCM wants to know transmission temp so it can make informed decisions on shift pressures. It will also do things like not lock up the torque converter clutch if it is too cold. I imagine the shift adapts require certain temperature ranges too, to learn properly.

Doing a bit of reading, seems many people have had failed Transmission Fluid Temperature (TFT) Sensors and they just fail. This was more so on the 2016 models which switched to a 1 piece hardness that encompassed the sensor.

On my dash, if in tow mode, I can tell TFT and it was showing a value. Using my scantool, it showed -40F originally but that ended up being a firmware bug and Foxwell sent me an updated code which fixed it. Cold ambient temp from letting it sit over night seemed in line with TFT. Monitoring it further though, the transmission never warms up to the recommended 195-200F. It also had a few cases where the temp dipped to 100 on the dash (lowest number).

Transmission Thermostat

On the 6L80 and 8L90 series, they introduced a thermostat to help increase the temperature to around 200 on daily driving. It is supposed to support better shifts. Others have said it also helps boil off the moisture which has been a huge problem for these transmissions. There is a small subset of people that feel this is too warm and they actually “delete” the thermostat to make it run cool constantly. They are either in much hotter climates or don’t believe the engineers at GM did it right. There is an old line of thinking that 200F will cook a transmission. They could be right but the warning temp on my dash indicates 300 and these newer fluids are supposed to easily get into the 240-250’s. Nobody on the forums with a properly working transmission seems to hit anywhere above 220 though and that’s under extreme loads. Normally the thermostat opens at 192 and any transmission cooler helps keep it down from much above that.

Freeze Frame

In any case, after clearing the codes one day, the next day only P0711 returned. It requires a second time to trigger before check engine shows up. I realized my Foxwell supports reading Freeze Frame. Its kind of like a black box but under less severe circumstances. In instances where certain codes are set, it will capture parameters that surrounded it. In my case I lucked out and it captured a 48F degree TFT which is definitely out of line

P0711 Freeze Frame - Transmission Fluid Temperature and Malfunction Counter
P0711 Freeze Frame

Graphing

I was even able to graph the temperature over the drive. It does seem like there is a bit of fluctuation but that could be because its not allowed to warm up with a possibly malfunctioning thermostat. Or the harness could be loose. Or the sensor could be inaccurate!

Replacing Transmission Fluid Temperature Sensor

There are quite a few videos on this. Here is a great one if you have the one piece harness, whereas at this time I think they switched back to the two piece which is less involved.

Conclusion

Since I have a GM Protection Plan (extended warranty), I’m not terribly interested in possibly voiding it. I also don’t have the expertise or comfort level to start toying with the transmission itself as the temperature sensor is in the pan. Appointment to the dealer made and I may update this Monday with the results. Regardless, anyone that gets a P0711 on a K2 platform on the 8L90 or even 6L80 transmission, this may help you diagnose it if you’re out of warranty. It could be an economical sensor/harness or even an easier to replace thermostat.

GMT K2XX Magnetic Ride Control Shock Inspection

Yes, this blog does tend to touch on other areas I am passionate about other than IT related topics. I’m an engineer at heart so I love investigating and understanding things, particularly those that tend to lead me down a rabbit hole.

Summary

About 6 months ago I started what would be my journey of understanding how to inspect and diagnose Magnetic Ride Control shocks. I took in my 2015 GMC Yukon Denali to the dealer for what I thought was going to be an oil change and a recall that simply required a code update. I was kindly informed that my front strut was leaking and needed replacement. Hopeful that my extended warranty might cover it, I was also kindly let know that was not the case. This was a maintenance item and parts & labor were going to be roughly $900.

Fast forward a few months and my original extended warranty was about to expire so I purchased a new one that came with a 30 day / 1,000 mile wait period. With COVID-19, I was doing 80 miles a month. I had to step it up and started doing 50-80 miles a day to make up for it. Towards the last 200 miles I realized the ride was very rough. I was not sure if the roads were terrible or not driving so much had increased my expectation of a smooth ride. Remembering that I had a leak on my front strut, I decided to take a look at the remaining strut and shocks.

Magnetic Ride Control

For those that do not know, Magnetic Ride Control is a premium suspension option in select GM vehicles. It has been around circa 2003 but is now on its 3rd iteration. There are many sources that can do a better job of explaining it than me but essentially instead of regular fluid, there is a magnetic fluid and an electromagnetic that is used to in realtime make adjust based on many driving conditions such as gas pedal position, steering position, incline of vehicle and ride height.

The rear shocks in the vehicle also have air springs that are inflated or deflated to provide auto leveling on top of this. This allows the vehicle to maintain its level independent of the load (to a degree).

For my vehicle platform (K2XX) the suspension RPO (Regular Production Option) is Z95. Here’s a good link on RPOs and what they’re used for – https://www.newgmparts.com/decoding-general-motors-rpo-codes

Leaky Shocks?

This is my first time having to deal with these particular shocks so I went to a few sources of information such as r/MechanicAdvice on reddit. Someone responded but they indicated that these shocks can’t leak as the air spring would contain the fluid. I would later find out this was partially correct.

Deciding to wipe it down and see if more fluid showed up, it did. This caused me to reach out to a few personal contacts and their opinion was that they were indeed leaking.

This onee had a ton of build up but was only slightly leaking, perhaps it had almost completely leaked out?
This onee had a ton of build up but was only slightly leaking, perhaps it had almost completely leaked out?
This one was very wet all the way around and you could tell with all the build up
Leaky Leaky

Bite the Bullet and Replace

After wiping down again, I was able to find the part number AC DELCO 23290661 which had been replaced by AC DELCO 84176675. The new part number hopefully indicated I might get more than 5 years / 37k miles on these! I shopped around and found Rock Auto had the best price for OE replacements although Arnott seemed to have a nice rebuild for $100 per shock cheaper. Maybe next time I’ll go that route.

They arrived and they were beautiful! Clear as day what they should look like.

So Pretty!

I’m a hobbyist and have done shocks before but just did not want to mess with this so I hired a local mobile mechanic to throw them in. Check out their Instagram video of them on the job! – I’m not sure why it says its violent but I promise its not!

After removing the old ones, they sure looked like they were in worse shape than I expected.

Anatomy

While learning about this I was curious of the anatomy of this. People had described it through my investigations but I’m a visual learner. I did come across this Youtube video on a Mercedes Air Shock which was similar but I wanted to see mine and how they failed.

Z95 Rear Shock Tear Down

To tear it down I used some rubber gloves as the magnetic fluid can be fairly abrasive, a very sharp box cutter and some pliers/cutters.

Start cutting!

Peekaboo!

At the end you can see a fully opened MRC shock. The lines point to the retaining metallic bands that hold the bladder into place. With it removed its essentially a regular shock in appearance.

So technically the bladder should hold the fluid but apparently the abrasiveness of it causes it to start leaking out over time by eating away at the rubber.

Looking towards the bottom of the shock
Looking towards the top where the air chamber is

Conclusion

Now that I am familiar with these shocks and how they fail, it does appear the remaining strut in the front is leaking and time for a replacement. Luckily I came across this very informative video on how to do it. I have a buddy with a spring compressor and better yet knows how to use it so wish me luck!

Extending Old Hardware Life With ChromeOS

Summary

I came across an interesting idea of extending the life of old by using ChromeOS. What is ChromeOS? We’ll get there. The problem is that many households have an old computer laying around collecting dust. It lacks either memory, hard drive space or compute power to be useable though. At the same time, with everyone social distancing, there’s a need for an extra device. You may have children trying to share devices to get on the internet or need an extra one for a family member that is in your house. This article may help you with that.

What is ChromeOS?

On wikipedia, ChromeOS is described as the following.

Chrome OS is a Linux kernel-based operating system designed by Google. It is derived from the free software Chromium OS and uses the Google Chrome web browser as its principal user interface.

https://en.wikipedia.org/wiki/Chrome_OS

For those non technical, it is basically an Operating System that is minimized to just the basic needs to get on the internet. Its assumption is that most uses cases for computers these days are to access the internet. When it boots up and you get logged in, its primary user interface is Google Chrome.

The problem

Google does not provide installation files to install ChromeOS onto your legacy hardware. ChromeOS is designed to come pre-installed on its Google ChromeBooks.

The Solution!

ChromeOS is open sourced which means other companies can take its code and build upon it. A company named neverware has done just that. It does have paid offerings for education and business but there is also a free download for home users.

What You Need

  • 8GB or larger USB stick
  • Operational computer you wish to install ChromeOS onto that has 2GB of RAM and 16GB harddrive
  • Ideally a newer and operational computer to generate a bootable USB stick.
  • A Google Account (you can create one as a part of this process)

Installing CloudReady

If you have a Windows machine, the installation is fairly straight forward as you will download the windows “USB Maker” and insert your USB drive into that computer and go through the guided wizard.

Once the bootable USB drive is created, insert it into the destination computer and boot or reboot it. You may need to go into that machine’s system BIOS to select a different boot order or ensure that it will boot off the USB drive. Sometimes during boot there is a keystroke to press such as F12 or ESC to select that. You may have to refer to your computer’s user guide on that. neverware has a list of these for the major manufacturers – https://guide.neverware.com/install-and-setup/boot-usb/

When booting off the USB drive, it will boot directly into ChromeOS or neverware’s version of it called CloudReady. It is fully usable at this point so even if the computer’s harddrive is crashed you could perform basic tasks. Ideally after booting you will go through an install process to install it locally and not require the external USB stick.

Booting

During the boot process you will see the following Welcome screen

Welcome!
Welcome!

Next it will require some sort of network connectivity. If you have a wired connection you are good to go. If you are on wifi, it will ask you to authenticate to your network.

Connect to network
Connect to network

Once connected to the network, it will ask you to login to your Google account.

Sign in to your Chromebook
Sign in to your Chromebook

Finally – here we are, a Chrome browser. The above is a one time setup process. Sometimes two times. When doing the above on the Live environment, after installing you may need to go through this once more.

Here we are!
Here we are!

Actually Installing

After you have used the live environment for a while and decided you want to permanently convert that computer to it you can click on the clock and select “Install OS”

https://guide.neverware.com/install-and-setup/home-edition/

Limitations

  • No Google App or Play Store due to licensing issues
  • This is not Microsoft Windows and your apps are limited to what’s in the Chrome Extension store

Final Words

At this point you will have a neverware’s version of ChromeOS installed and have basic web browser functionality on this device. It should be more than sufficient for basic browsing since that is the typical use-case. Hopefully it has helped you make some use of a device that was just collecting dust.

Unboxing My Keychron T10 USB-C Hub

Summary

In preparation for my 2020 13″ MacBook Pro, I realized I am missing some accessories I have not had to upgrade in a while. My 2013 Late model had various ports I needed such as USB2, Thunderbolt 2, HDMI. My needs for a USB-C hub are minimal but I wanted to support Keychron since I have a Keychron K1 v1 and recently upgraded to a v3.

Keychron

If you have never had a mechanical keyboard, check out my article Why I Use Keychron For My Mac. Since I got my K1 I have been following them and have been tracking their T10 USB-C Hub. Up until recently though I have not had a need for one.

Unboxing

Here are a few pictures of the unboxing.

Features

  • 4 x USB 3.0
  • 1 x SD
  • 1 x TF
  • 1 X HDMI ( 60hz at 1080, 30hz at 4k )
  • 1 x VGA
  • 1 x 10/100 Ethernet

Opinions

Only time will tell after using this. To be fair, it will likely be an infrequent use due to most of these ports not necessary for my daily use. I am a little let down that the ethernet port is not a full gig port. Not that I need the speed but some newer switches are only supporting gig or higher so I may run into some compatibility issues.

It supports both USB-C PD and Thunderbolt 3 supporting up to 100W of PD Fast Charging which makes it a great accessory for Macs and Apple ecosystem. Keychron has tried to be a Mac enthusiasts advocate. With that in mind, it works great on Windows 7 or higher.

My Recommendations

My recommendations to Keychron would be to put the input power on the other side. Having it on the same side involves some fun power cable routing. Typically people that own Apple devices are fairly annoyed by things like this.

I would also recommend upgrading the Ethernet port to a port capable of full gig just for compatibility issues.

One last thing I noticed lacking was a USB-C port or ports. While many laptops have extra ports, it would be very nice to be able to plug all of your accessories into one HUB and only plug that into your laptop to minimize insertions.

Final Words

One thing to keep in mind is this USB-C Hub has the markings 5V/3A which is the overhead for it to operate. While it is capable of up to 100W it will siphon off 15W of that. So on my 13″ MBP I may plan on upgrading to a higher wattage power supply to account for that as the 61W one will get reduced to 46W when passing through. This is typical of a USB Hub though as they do require power.

Highly Anticipated 2020 MacBook Pro 13 Inch Purchase

Summary

I have had my eye on a new 13″ MacBook Pro (MBP) for about a year now. My first MBP was a 13″ Late 2013 model. It was maxed out on RAM at the time with 8GB and had a 256GB SSD. I think a 512GB SSD was an option though. This was my first personal SSD. At the time SSDs had a bad name for randomly and prematurely failing so I was a little anxious about that. At the time I purchased it because I was starting to have to support Macs in my day job with some of the software we used, particularly Microsoft Lync and Citrix XenDesktop. It was something new at the time and I wanted to get comfortable with. Steve Jobs did good because based on that purchase I have been hooked!

Why Have I Waited So Long?

Up until about the past year the laptop has worked sufficient for my needs. I do not run a lot of high intensity applications in terms of RAM or CPU. The battery life for what I do is still decent. Ultimately though over the past year I tend to start running low on RAM and see some fairly aggressive swapping. I am also having more difficulties setting up virtual labs in VMware Fusion. I have had to resort to putting the absolute minimum specs on VMs to get them to boot. This is not a bad thing for some Linux Distributions but I have even started running low on disks space. I stood up a Kubenertes cluster on it but that was slightly painful with memory related issues.

Why Not Purchase a 2019 MBP 13″?

I had thought about it for quite a bit starting in December of 2019. To be honest, I was pretty close to doing it. I couldn’t quite settle on 2 or 4 USB-C and balked a little at the price. I ended up sitting on the decision for so long, rumors of the 2020 version came out with some pretty nice features.

Why Not Wait Until 2021?

It seems as though the way Apple is going, 2021 is likely to be the year of ARM processors. That, in and of itself is not a huge deal but I do tend to run x86/x64 based VMs in Fusion and I am not sure if there will be an emulation layer to allow that to happen. Likely not. So I knew I could wait for something after the 2019 version but that I didn’t want to wait until 2021. Ultimately it is now a day to day struggle doing some things that require a few different windows open. I have a few chat apps like Teams, Slack and Google Chat. Combine that with Apple Mail and Calendar for 4-5 email accounts. Then try opening up a few tabs in a browser and start up meeting software to share my screen and it just starts swapping memory like crazy.

Ok, So 2020 Was It!

With all of the above the decision kind of made itself and my fear of paying for it went away. Scary times during COVID-19! Not all of the 2020 rumors panned out. Some speculated that it would come with a 14″ screen. To be fair, if I wanted a larger screen I’d have gone with the 16″. I do prefer the 13″ form factor. It is easier to bring with me places. When I travel I usually put the laptop in the hotel safe but my friends usually have difficulties fitting their 15″ laptops in there.

What drew me closer is the beefed up specs and 10th gen Intel Processors. I had thought I was just going to double my HDD and RAM to 16GB and 512GB. When it came to configuration though, I noticed the higher end models had the option of 32GB RAM. While I was doing that why not go to 1TB of SSD. I probably won’t use the space but right now I have to be so careful with my VMs, I can’t leave any laying around.

In terms of the processor I went with the Core i5 versus i7 due to battery life and unnecessary cost. My use case utilizes the CPU very minimally although 4 cores with hyper-threading is useful for running multiple VMs. I don’t game and my business apps aren’t a huge CPU hog.

Final Words

If you are thinking about the 2020 13″ MBP, here is my reasoning and thought that went behind the purchase. I will most likely do an official unboxing and review post when it arrives in a few weeks. Since I picked custom options, it unfortunately takes that long to ship. Hard to believe in today’s world where we expect instant gratification, there are still some things you have to wait for!

6 Useless Things I Am Doing During COVID-19

Summary

A combination of bored-some and a little obsessive compulsive tendencies lead to some random things. Since we’re all scrolling for content, why not read some of my neurotic tendencies during this. Who else is doing these?

Household Paper Product Inventory Management

Paper Products!
Paper Products!

Who isn’t having to do this? Has anyone figured out why toilet paper and paper towels are still hard to come by weeks later? Long before this I’ve done inventory management to avoid the random $300 trips to target to fill up so we’re good but we are starting to get low. This is my first time post COVID-19 to try to acquire Toilet Paper & Paper Towels. I’ve got a good Amazon, Costco and Target stalking routine.

Exercise

One of my morning runs
One of my morning runs

I had a nice running routine since my Gym closed but let’s face it, I’m out of shape and could stand to lose a few pounds so I am on the sidelines nursing a knee injury. After a few days of running I thought I”d take my 4 year old with me. He ran about 100 ft and then wanted to be pushed. I thought I was in shape enough to do that. My knees told me otherwise!

Electronic Devices

Android Update
Android Update

Last week I decided to upgrade the firmware/software on all of my electronic devices. As I mentioned, I can be a little obsessive-compulsive. It started with my wife’s iPhone and iPad and then the apps on them. I then went to her laptop and updated all the firmware, windows updates, printer drivers and 3rd party software. My laptop is almost always up to date so while I checked it there was nothing else to do. Ecobee (smart thermostat) is more of a pain than my threshold for boredom/laziness. It usually requires a chat with support and required way too much work.

All of my My devices are up to date. Shouldn’t I feel accomplished? Not quite but we’ll get there!

Rechargeable Batteries

Recharge!
Recharge!

Ever since the kids came, they use batteries like nobody’s business. Maybe that’s how they get all their energy because its non stop. In any case I decided to replace all of the rechargeable batteries with freshly charged ones. This included both TV remotes, kids toys, noise hair trimmer, DVD remotes, magic mouse, wireless headsets and a few flashlights.

Sometimes I put in older previously charged batteries to see if they need recharging.

Vehicular maintenance

Vehicles sitting for long periods of time need some maintenance. If you have keyless remote, it does have some drain on the battery which can add up over time. This is particularly so if you walk by your vehicle with keys in pocket which tends to wake up the car in preparation for your entrance.

I’ve been starting up my truck for about 10 minutes a week but today took it out for a drive to help avoid flat spots in the tires. Along with avoiding flat spots, you want to make sure they’re properly inflated. In reality it usually takes about a month to develop flat spots. Colder climates see this much more quickly but most people are lucky enough to correct this after 20-30 minutes of driving.

Also not a bad time to check oil and fluids although I already do that nearly every fill up on both of our vehicles. It did come in handy once when I found a coolant leak on my truck.

In all reality though the vehicles can likely sit for a month without it being problematic.

Endlessly and Needlessly Scroll Social Media

Aren’t we all doing this, looking for a new article to waste our time. If you’re not the master of Facebook, NextDoor, Reddit, Snapchat and TikTok, get on it. You can waste a few days on those. If you’re thinking, wait, he missed a few, gold star for you!

Another hobby I’ve taken up when I get bored of this is “Coffee Naps“. Think of it as power napping on steroids. I actually stumbled across this by accident before finding out it was a thing via social media.

Final Words

In all seriousness though, whatever we need to mentally make it through this, don’t feel bad about doing it. Social distance as best as you can and just try to hang in there! Thought you were alone and doing some of these too? You’re not, we’re all in this. Drop a line or a comment if you do any of these.

ISC BIND Look Aside Related Outage

Summary

I had a fun issue today. All of a sudden BIND stopped returning results for recursive queries to external zones.

My logs were filled with lines like the following

Mar 25 09:05:00 XXXXXXX named[XXXXXXX]:   validating @0x7f3458067420: dlv.isc.org NSEC: verify failed due to bad signature (keyid=64263): RRSIG has expired
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]:   validating @0x7f3458067420: dlv.isc.org NSEC: no valid signature found
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]: error (network unreachable) resolving 'com.dlv.isc.org/DS/IN': 2001:500:2c::254#53
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]:   validating @0x7f3450030b60: dlv.isc.org NSEC: verify failed due to bad signature (keyid=64263): RRSIG has expired
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]:   validating @0x7f3450030b60: dlv.isc.org NSEC: no valid signature found
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]:   validating @0x7f3450065e40: dlv.isc.org NSEC: verify failed due to bad signature (keyid=64263): RRSIG has expired
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]:   validating @0x7f3450065e40: dlv.isc.org NSEC: no valid signature found
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]: error (no valid RRSIG) resolving 'com.dlv.isc.org/DS/IN': 149.20.64.4#53
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]:   validating @0x7f3450065e40: dlv.isc.org NSEC: verify failed due to bad signature (keyid=64263): RRSIG has expired
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]:   validating @0x7f3450065e40: dlv.isc.org NSEC: no valid signature found
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]: error (no valid RRSIG) resolving 'com.dlv.isc.org/DS/IN': 156.154.100.23#53

Troubleshooting

Naturally I tried bouncing named without luck. I then thought there was an issue with the root zones and configured forwarders without luck. I had disabled dnssec via “dnssec-enable no” without luck.

This seemed fairly strange. Ultimately since it was DNSSEC related I opted to disable it via as a temp workaround. It appears the validation was the issue.

dnssec-enable no;
dnssec-validation no;

After some investigation and troubleshooting it appeared to be related to ISC’s DLV and letting RRSIG expire accidentally. It failed in an unexpected manner when this happened.

What is DLV?

DLV stands for DNSSEC Lookaside Validation. DLV is a service that ISC has provided since circa 2006. It allowed DNSSEC to be enabled on zones that could not otherwise be enabled. Not all Top Level Domains (TLD) implemented DNSSEC until the past few years. This was a workaround to allow DNSSEC until then.

In 2017 it was finally decommissioned with DNSSEC being fully available to all TLDs. The A record was left in place and many resolvers still attempt to connect but it does not provide any data.

What is RRSIG?

If you want a full view of DNSSEC and how it works, CloudFlare has a great article for that here – https://www.cloudflare.com/dns/dnssec/how-dnssec-works/ . In short though, RRSIG records contain cryptographic details, particularly start and end dates for the validity of that data. This is much like an SSL Certificate that has a valid period.

The RRSIG records are designed to be required to be updated frequently to ensure the security much like SSL Certificates need to be renewed. This helps prevent a replay attack where an older compromised key is reused.

RRSIG Value

Running the following I could see it expired

# dig +dnssec dlv.isc.org

dlv.isc.org.		3599	IN	RRSIG	DNSKEY 5 3 3600 20200325160456 20200224153150 19297 dlv.isc.org. TyUbbNgG/Oru7TQFHbDC9E208hB8Szheu634Q03nawQFz4dosOFg+ZB5 z8Svh8fw/g35a/ZW5AP1jbSKh19u4c7Ujre3iygS0Tjycmi0mYG6dS7I CcWLOxZpOKf8uw9mzgbIR/VDEFmKj0OJKdkxAqfaWxXLqBBWgFqIucC6 9Tb98clinCPW34xgk6Fzi+OKAFmiGH6/e8wk/h5RMWxipx5KAk2NsWsw QMyEDaA7eLzZTbBenftVR86g6QO4bR+LOKzxGBFQ2XW0ArQKDiuoBqEw 8cmRcGKzVJ761d7EK+LDvnktRNxRMJ9y5LPgxlO2Xm3Un8oExjVbLKi7 OigQnA==

20200325160456 was the key, that translated to 3/25/2020 16:04:56 UTC which is about when the issue started. Further down in the “References” section the ISC-USERS list confirmed this was by mistake. I suppose it was a good “scream” test to remove lookaside. Newer BIND versions do not even support this anymore.

What Happened?

On many older BIND servers deployed before 2017, they were configured with the following.

dnssec-lookaside auto;

Auto would try to query dlv.isc.org first and then query root name servers. The expected behavior was that it simply would not return any data and then the root zones would be queried.

Unfortunately with an expired RRSIG it failed in a way that made BIND think the query response was not valid and an expected failure. For all BIND knew, it was preventing a replay attack.

References

I had originally posted on Reddit and was also pointed to ISC-USERS.

Using Certificate Based Authentication

Summary

Recently a client had a need for putting a web application on the internet that end users could access. They wanted to lock it down so that not everyone on the internet could access. Whitelisting IP Address was not an option because they were remote users with dynamic IPs and the overhead of maintaining this whitelist would be problematic.

The use case was a password recovery tool that their remote users could use to reset and recover passwords. Simple authentication would not suffice. For starters if the users’ passwords expired they wouldn’t be able to easily log into the site. Along with that it would be a high profile target for brute forcing.

Why Not IP Whitelisting?

IP whitelisting used to be and still us for some organizations the de-facto method of filtering traffic. It can be very problematic though. Users today are on the go, working remotely or using their mobile device on cellular data as well as home internet. Other times it involves sales staff at client sites. Keeping up with these IP whitelists can be a chore. Updating this whitelist can be time sensitive to avoid halting productivity. When not maintained, there is a chance someone unexpected could gain access due to simply having an IP previously whitelisted.

A workaround for this is VPN but that requires a bit of support overhead in user training and support. This can be clunky for users that are not used to to using VPN.

Why Certificates

Many larger organizations already have internal Certificate Authorities in place. For Microsoft Active Directory deployments, when CA has been installed, end users are likely auto enrolling in user certificates. Domain joined workstations already have these and trust it the internal Root CA.

Certificates also have a built in expiration. In an auto enrollment environment, this expiration could be lowered substantially to below 1 year.

TLS Handshake

Once of the nice features of TLS is that it does include a mechanism for this. Below is an example of a TLS handshake where the server requests a certificate and the client provides it.

TLS Handshake - Certificate Authentication
TLS Handshake – Certificate Authentication

In Frame 19, the client makes the TLS request with a Client Hello. Frame 23 the Server response with a Server Hello. This is where they set parameters and negotiate things like TLS versions and encryption algorithms.

Frame 26 is part of the Server Hello but it was large and split up. Boxed in red is the “Certificate Request” where the server is requesting a certificate to authenticate.

Frame 33 is where the client actually provides it.

From here you can see this happens before the application level (HTTP) protocol communicates starting in frame 43. What this means is that before the user reaches the web application for authentication, the device requiring TLS Certificate Authentication is filtering the requests. Many times this is a reverse proxy or load balancer is not vulnerable to the same exploits as the web servers.

Browsers

When used properly and the client has a certificate, the browser will prompt users for a certificate to use such as pictured below.

Browser Certificate Authentication Prompt
Browser Certificate Authentication Prompt

Other Applications

A really neat application for this when you have a legacy plain text protocol in play but you want to open it up over the internet and secure similarly. Perhaps you have a legacy application that uses raw text and is not SSL/TLS capable. You can still put this on the internet through a reverse proxy like F5 LTM or stunnel.

Traditionally this type of traffic would be protected via IPSEC tunnel that is encrypted or a dedicated circuit such as MPLS. That does require specific hardware and/or monthly circuit costs to accommodate.

stunnel is extremely useful in this scenario as you can install it on the local machine that has the legacy application and configure it to connect to localhost on a random port and proxy information out over TLS and configure it to use the certificate based authentication.

Here is a graphical example of what that may look like with an stunnel server broken out. stunnel could be installed on the end user’s workstation though.

Legacy App Secured with TLS 1.2 or higher & Certificate Based Authentication
Legacy App Secured with TLS 1.2 or higher & Certificate Based Authentication

stunnel could be put on the local end user workstation to minimize that unencrypted leg. Typically on the server side the reverse proxy has a leg directly on the same VLAN/subnet as the application which minimizes exposure over that but this does help secure the application traffic over the untrusted internet.

Final Words

In this article we learned a little on Certificate Based Authentication. We also learned how it may help your organization better secure your applications and possibly avoid more costly solutions.

Internet Routing and BGP Looking Glasses

Summary

From time to time I get requests from colleagues, “Can you ping this address?”. Many times what is going on is they are bringing up a new internet link and want to check routing. Sometimes they areadding a public endpoint and want to make sure its accessible. They are asking me because within their network it works but they need to make sure it is accessible or routing properly over the internet.

BGP Looking Glasses are a great tool for this if you would like to be self sufficient. You can also get a wider view than just a few colleagues. My favorite one is HE.net’s https://lg.he.net

What is BGP?

BGP Stands for Border Gateway Protocol. It is the standard exterior gateway protocol for internet routing. While an internal network may use something like OSPF for its interior, BGP is better suited for exterior routing.

One of the few keys to understanding BGP at a high level is to understand it is a distance vector routing protocol. These routes are typically better suited for WAN routing as they are a bit more light weight. The downside is that each router’s perception of the internet is key. There is no unified single view of the internet routing. Each router has its own perception of the internet based on the BGP routes it receives.

This is why Looking Glasses are so important. You can see the perception of various points on the internet. This can help you determine internet traffic destined to your infrastructure is taking the expected and optimal path.

HE.net’s Looking Glass

Here is a small snippit of HE’s LG. They have various routers all over the world that would not fit into this screen shot. You can also see the various functions you can perform on the right hand side.

HE.net Looking Glass
HE.net Looking Glass

Ping and traceroute are fairly self explanatory. For ping it just returns the results of an ICMP ping and traceroute shows a list of routers in the path of getting to the destination.

HE.net Looking Glass - Traceroute
HE.net Looking Glass – Traceroute

The real value here is the “BGP Route” option.

HE.net BGP Looking Glass Details
HE.net Looking Glass BGP Details

Here we can see all of the BGP peers this particular router has learned the route to 1.1.1.0/24, the AS path it takes and which one it selects as the best path.

Autonomous Systems

If you are new to BGP and dynamic routing protocols you may be wondering what an AS (Autonomous System) is. In BGP world, it is basically a grouping of similar routers that announce a similar set of subnets or prefixes as they are called in BGP. BGP groups systems together by AS. As each router learns a route, it appends its own AS to the AS path before passing it along.

The above is a bad example because it shows a single direct AS path as HE appears to be directly peered with CloudFlare (AS 13335). CloudFlare is very well peered on the internet. Below is a better example. It at least shows it passing through AS 1299 (Telia) to AS 174 (Cogent)

It seems HE.net is fairly well peered but here is another router output that shows some decent AS paths and the differences. AS 174 being Cogent and AS 209 being CenturyLink and AS 3356 being Level3 it chooses the shortest AS path. Keep in mind the traceroute through CenturyLink could possibly be shorter actual router hops. A shorter AS path does not necessarily mean less latency or shorter traceroute hops.

#show ip bgp 73.0.0.0/8 
BGP routing table entry for 73.0.0.0/8, version 767462940
Paths: (2 available, best #1, table default)
  Advertised to update-groups:
     18        
  Refresh Epoch 1
  174 7922, (received & used)
    X.X.X.X from X.X.X.X (X.X.X.X)
      Origin IGP, metric 13031, localpref 100, valid, external, best
      Community: 174:21000 174:22013
      rx pathid: 0, tx pathid: 0x0
  Refresh Epoch 3
  209 3356 7922, (received & used)
    X.X.X.X from X.X.X.X (X.X.X.X)
      Origin IGP, metric 7800026, localpref 100, valid, internal
      rx pathid: 0, tx pathid: 0

Final Words

If you ever find yourself needing to ping or traceroute from a remote endpoint lg.he.net has you covered. Many carriers have their own looking glass. This is useful incase you want to see how your routes are perceived from their end. If you use BGP at your edge and receive full routes, this is another avenue of seeing those prefixes. With looking glasses, you can do this from various endpoints across the internet pretty easily.

Exciting News!

I get to help a diverse set of clients on a full time basis starting early March 2020! If you are wondering why I have not written an article recently, I had this big decision going on and was focusing on that.

I recently made the decision to enter into the consulting world again. I will be transitioning into this over the next month. Over 3 years ago, after recovering from a failed business, I was blessed to be an independent consultant with Pegasus for the greater part of a year. The opportunity came at an excellent time to help get me back on my feet. They then offered me employment when I needed more stability with flexibility and it was a mutual arrangement. A work home was made at Pegasus for the next two and a half years and it was great. I could not have planned this better even if I tried so it had to be His plan!

Most recently a few previous clients needing assistance drove me to adjust my current employment terms. Pegasus was flexible enough to allow me to make this transition and still take care of them.

With all of that said, I am always looking to acquire new clients to share my extensive knowledge base.  Most of you know but I have been doing IT consulting for over 20 years, extremely detailed oriented and thought provoking in all aspects to a fault. Please keep me in mind should you have some IT needs or considerations, now or future. If you’d like to check out my resume, skills list or blog site – https://www.woohoosvcs.com