Today I had my first Tesla Service Center visit. Rewinding a bit my Tesla journey started on 8/2/21 when I decided to downsize my Yukon and make the plunge into an EV. Fast forward a few months of constantly watching the Estimated Delivery Date and finally receiving ownership on 11/12/2021. The car has been great to drive but over the holidays I started getting a strange error.
The Error – DI_a138
On 12/29/2021 I pulled my Model Y out of the garage and helped my mother in law in to take to the doctors. After getting back in and putting into park I received a strange error.
Power Reduced – OK to drive Front motor temporarily disabled DI_a138
I figured this was just a fluke but created a technical support case with Tesla on it via https://www.tesla.com/support. They checked and did not see anything but recommended I create a Service Appointment to request a virtual diagnostic if it persisted. It did and I did!
After some back and forth chatting on the app, it was requested I drop off the vehicle early before the scheduled appointment for this issue to diagnose the drive unit and it would need to be there for 72-96 hours. I had hoped more could be done virtually but I got it and brought it in.
I spoke with the Service Advisor and he consulted with the lead tech. The Service Advisor then told me he had great news. They had already pulled logs and were ordering a new drive unit just incase and I was free to go with my vehicle and would not need to come back until further notice, likely when parts arrived.
This was really neat. I’ve heard the service visits being a mixed bag. Many times on the groups you only hear complaints but in the marketing materials you hear AI powering automatic parts ordering and virtual diagnosis. In this case while it would have been nice to save the drive I didn’t mind it. It also also nice for them not to need to keep my vehicle until parts arrived, particularly with the shortages happening these days.
More DI_a138 Details
So this was an interesting one. There is a higher probability of this happening when running errands sometimes and quick in and outs. This never happens on a “cold” drive where the car has sat for a bit. It appears that when some pre-drive check is done on the drive unit. It hits an error it doesn’t understand or some safety check and shuts down the front motor.
After finding some other articles in this happening in the past I realized I could just get out and back in and it goes away and drives with both motors enabled.
In some Model 3’s and Y’s from 2019 and 2020 there was a glitch where the motor needed to vent heat and it got confused. If this is the case it may just be a regression on the latest software. At this time it is 2021.44.30
As I hear more I’ll update this article but overall a great first experience with the service center.
1/19/2022 – Brought it in this morning and the short version is known firmware issue that’s now fixed.
“Performed diagnostics for front drive unit, Performed inspections of harness and all connectors and connector pins. No issues found. This was a past firmware issue that has been resolved with firmware 2021.44.30.6 and vehicle is now 1 firmware higher. test drove vehicle to verify no further issues.”
It seem every week we see a new article of some sort of IT security related incident. While many of these larger companies tend to have security tools to help combat them, many smaller companies do not know where to start. An eye opening experience would be to run an internal vulnerability scan on your corporate network to see what types of vulnerabilities you have.
These scans results may then drive you to more promptly patch and mitigate vulnerabilities. Other times it can help give data to management to ask for resources like security tools and people to manage them.
What is This Article?
This article is a quick and dirty install guide for installing Greenbone Vulnerability Management on Kali Linux. Both have been around for quite some time and are free to install. You can attempt to install Greenbone on any Linux distrubution but Kali tends to be the easier and more stable platform since its very commonly used for vulnerability scanning and management.
A Server (Physical or VM) to install that has 2GB RAM, 1 Core and 20GB HDD space
Basic understanding of installing Linux
Installing Kali Linux
We won’t get too much into the details of this but if you’re a first time installer, select the Graphical Install and you can’t go too wrong with most of the defaults.
You can install a Desktop but for the purpose of this tutorial, I’ll be using command line so I unselected the Desktops. It saves space and a little compute.
Prepping for Greenbone Vulnerability Management
I always like to start out with a freshly updated operating system. Since Kali is based off Debian we’ll be using apt-get. The first step is to configure network access.
# This will tell us the interface name, most likely eth0
# This will allow us to edit the interfaces file
sudo nano /etc/network/interfaces
We can then populate it with eth0 information for DHCP
The new lines are the “auto eth0” and “iface eth0 inet dhcp”.
Control X to save and then a “sudo systemctl restart networking” and your ifconfig -a should show an ip address for eth0.
From here we can update the system. Its always best to update before the install to ensure you’re running the latest fixes.
# We're going to switch to root since the next few commands will need to be under that context
sudo su - root
# This command updates the package metadata
# This command performs the actual update
# Reboot when done to start fresh (optional but should be done at some point)
Depending on your internet connection, this could take a few minutes.
Installing and GVM
Installing is the simplest part.
sudo apt-get install gvm
And wait a few minutes after answering “Y” to download and install.
Next we’ll run “sudo gvm-setup” which aside from running through the configuration will start pulling down the various feeds. This will take a good 30-40 minutes. It does an rsync on community feeds for CVEs, Vulnerabilities and other scanner options.
Once it has completed this process, we do need to modify it to listen on 0.0.0.0 instead of 127.0.0.1 so that it is network accessible. Before that though it will output the admin username and password which we need to capture before proceeding. It is randomized for every install and since this install will get blown away after this lab, I’ll share it unfiltered.
We can do that by running the following to edit the systemd startup file
From here a “sudo gvm-start” is all that is needed.
If all went well you should be able to https:// to the IP of the server on port 9392
With any luck aside from a certificate error the following screen should show up.
Configuring a Scan
Now that we have access to the tool, we need to configure a scan. This starts with configuring a target list. Under Configuration / Targets. Then create a new target
This requires just a few fields entered. The name is fairly self explanatory and is just a name for the targets to be scanned. For the list of hosts, you can list subnets and/or IPs. In this example I picked the scanner itself.
For the PortList, The default can be used but when possible I like to scan All TCP and Nmap top 100 UDP ports as many times applications run on non standard ports.
If you feel comfortable providing your scanner credentials it can login and perform authenticated vulnerability checks as some may not be able to be detected anonymously.
Under Scans / Tasks we want to create a new Scan Task
Once that has been created and saved a scan can be kicked off.
When running a scan it can take some time and that is determined by the remote endpoint but it is not uncommon for it to take 30 minutes per IP address depending on how quickly the endpoint can be scanned and what it finds. By default it does have quite a bit of concurrency and in order to fully utilize that you may have to tune your CPU and RAM configurations but that is out of scope for this document.
Once a scan is complete you have a report. Unfortunately this scan did not turn up much due to it being already fairly secure but this is what you can expect to see!
That is it! You now have a vulnerability scanner you can use to scan your network for vulnerabilities. If you choose to use this for your vulnerability management needs, tweaks will need to be applied to port ranges, hosts and other settings but in a nutshell this is the nuts and bolts of what you need.
Some time ago when I was just learning Kubernetes, I wrote a series of articles that started with Intro To Kubernetes. This was on an earlier version (1.14) where many of these manual steps worked. Spinning up a new instance on 1.17, these steps really did not hold up and the cluster was not fully functional. I decided to give kubeadm init a try and it made my life infinitely easier.
Many of the provisioning steps in Intro To Kubernetes are similar, if not the same. Instead of manually starting services on hardware though, kubeadm spins up a bit of it as pods which saves time.
For this lab we need a VM we will name kcp1 (Kubernetes Control Plane 1). Since most of the master/slave terminology is going by the wayside to be more sensitive to what it represented in the past, what was previously referred to as master is now a control plane. The nodes were almost always referred to as worker nodes so it makes sense.
kcp1 needs 2GB of RAM and 2 cores/vCPUs. kwn1 can get away with 1GB RAM and 1 core. For storage, since this is a test lab, 8GB is more than sufficient. For the OS, I still am using VMware Photon OS version 3. I choose the ISO and its a quick install – https://vmware.github.io/photon/. PhotonOS was chosen because it is highly optimized for being a VM and minimized to be a very light weight OS to run docker containers. These traits make it perfect for running Kubernetes.
For this we will need the kubernetes and kubernetes-kubeadm packages installed. It also requires iptables and docker but that comes installed even on the minimal install.
Install and we’re off to the races! We’ll need this on kcp1 and kwn1.
In order for key parts of it to work, we’ll need to open a few firewall rules. This is done by editing /etc/systemd/scripts/ip4save and adding a few lines and then restarting iptables.
kcp1 needs the following line. This is for the API calls which must of the cluster makes to the control plane nodes.
-A INPUT -p udp -m udp --dport 8285 -j ACCEPT -A INPUT -p udp -m udp --dport 8472 -j ACCEPT -A INPUT -p tcp -m tcp --dport 10250 -j ACCEPT
It should look something like this afterwards
Then a simple restart to make the rules take effect.
systemctl restart iptables
kubeadm init Magic!
The magic here is kubeadm init. We need to use something like flannel for the pod network overlay. We don’t get into that in this article but if we don’t pass the right flags for that into kubeadm init, it won’t work. Per flannel
This needs to first be run on the control plane node.
kubeadm init --pod-network-cidr=10.244.0.0/16
On our first run we get some errors about docker not running. We need to enable and start it!
systemctl enable docker systemctl start docker
Giving kubeadm init another run and its off! It may appear to hang for a minute and then spew out lines of actions its performing.
It will then tell you some next steps. It may seem like a lot.
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.83.15:6443 --token xspmbf.kko3d24t74rpaeyo \
For the most part these are copy and paste. The first three lines literally are so we’ll do that so we can use kubectl.
One of the big remaining steps is to deploy a pod network, usually an overlay. As we mentioned earlier, I prefer flannel and it usually just works.
This just needs to be done via kubectl once. It instantiates a DaemonSet which essentially pushes this out to every node that attaches to the cluster.
Over on kwn1 we can run the join command it listed. It will take a little bit of time and then be done. Once done it recommends doing a “kubectl get nodes”.
[email protected] [ ~ ]# kubectl get nodes NAME STATUS ROLES AGE VERSION kcp1 Ready master 6m35s v1.17.11 kwn1 Ready 32s v1.17.11
You can also check all the running pods to see the health. All is well and pretty turn key.
We have a working cluster that was fairly turn key and much easier than my earlier attempts. If you are fairly new to Kubernetes I would recommend deploying the dashboard which I outline Kubernetes Dashboard. This is mostly correct except that the version 2.0 is no longer beta and has a final release that works pretty well now.
My first real exposure to auditd has been deploying Rapid7 InsightIDR. Its been a great tool but I did not understand why they require auditd to be disabled. Is it not more secure having an audit daemon running? After a server had rebooted to completely disable this, it ran into a ton of performance issues and my path lead me down better understanding auditd.
What is Auditd?
At a high level it is a piece of software that allows you to view and audit actions being performed on the operating system. These actions can be directory listing, program execution, audit policy changes.
How is this Facilitated?
There are two pieces to the puzzle. There is a kernel based kauditd which queues up audit events for consumption. Various syscalls and other events cause an audit event to trigger.
These events are consumed via a userland auditd process. This process enables the auditing (via auditctl) and registers its PID and starts consuming events pushed through a PIPE. by default these events are simply logged to /var/log/audit/audit.log.
In the event that the auditd can’t consume these properly there is an audisp process that handles the overflow. Perhaps the auditd crashes but audit events are still pumped to the PIPE. Audisp will generically pump these to syslog which may push to /var/log/messages instead.
You can use the auditctl command to view the status or add rules. These rules would allow you to monitor extra resources or ignore certain ones.
Why Does Rapid7 Want This Disabled?
The Rapid7 ir_agent has its own consumer process that scans the auditd events. Unfortunately (unless compatibility mode is enabled), only one process can consume these and Rapid7 wants its agent to be the consumer. It does so in an interesting way though. kauditd knows events can’t always be consumed in realtime so there are provisions for a backlog buffer which Rapid7 increases. It does this so a background job can run periodically to pull these events from the queue instead of listening in realtime. I’m not sure of the benefit of this method but it seems to work.
Without auditd disabled and stopped though, no others can listen, including Rapid7.
Here you can see, it is currently disabled and no pid is registered. Importantly though you can see the high backlog_limit (buffer). It is normally under 300 because auditd is running constantly.
So Why Did Our Server Blow Up?
Well, in our case its a server that generates a ton of audit logs because it kicks off tens of processes per second. It was easy to dump those to a audit.log file but a security tool actually parsing those just couldn’t keep up. We will either need to filter some of those auditd events or disable the agent.
If you ever need to use auditd, hopefully this gives you a basic insight into it.
Car owners increasingly complain about road vibrations. You may get some new tires and notice a vibration you can’t explain. Over time a vibration may start or get worse. You bring it in and they can’t find anything wrong and recommend a tire balance and rotation. The issue is still there but the issue can’t be found. On the repair shop side, a customer comes in with a vibration complaint. Maybe they are being picky and its a minor vibration that is normal. Or perhaps you find the vibration and fix it but they still complain about it. How do you quantify the mitigation of the vibration?
This is what the science of Noise Vibration Harshness (NVH) aims to address. NVH “scopes” have been around for decades but many times they are difficult to read and understand. Newer scopes that make this much easier are out of budget for many people and repair shops. For repair shops they don’t tend to make money off troubleshooting these kinds of issues so it is hard to justify the expense.
In full disclosure, I came across this with my own vehicle vibration. I took it into the dealer and they did not find anything but recommended a road force balance. It seemed to help a little but not really.
I was researching scientific methods of identifying the vibration and came across NVH. My intent is to use this data to help provide some guidance and extra data for the dealer when asking them to use a PicoScope. I always like going in, informed and being able to provide objective and quantifiable data to the any repair shop. I don’t know that this will actually help narrow it down but understanding the science of how more professional tools works was a neat discovery.
High Level Science
The high level science to how a scope like these work is that they detect vibrations and use calculations to convert that to frequencies in hertz (Hz). That is basically a quantification of how many times per second that vibration happens. Each set of components components that rotate has a set frequency it will vibrate at. For example, an engine at 1200 Rotations Per Minute (RPM) is rotating at 20 Rotations Per Second. A hertz is a unit of measurement that represents one cycle per second. Something that cycles 20 times per second or rotations per second would be 20Hz.
It becomes a little complicated because vibrations can have an order to them. In the above example of an engine at 1200 RPM. Its first order vibration would be 20Hz. A second order vibration would be 40Hz, doubling for each subsequent order. Certain components have natural vibrations at various orders. For example an 8 cylinder engine will have a 1st order vibration and a 4th order vibration. Each engine has a Nth order vibration that is equal to half of their cylinders in a standard 4 stroke engine.
Everything in a vehicle vibrations, so how do we know what to look for? The amplitude of each vibration is the key. Many software packages measure this in meters per second squared or mg or mili-gravity. The universal constant for gravity is 9.81 meters per second squared so something traveling at 1G is accelerating or increasing speed at a rate of 9.81 meters per second every second. In the automotive world, typical vibrations are so minimal that we use mili-gravity or mg.
Understanding the Science For Automotive
The key to this is to find order vibrations that relate to major components. A vibration that relates to the engine does not mean the engine is to blame. It just means that something that spins at the same rate as the engine is the cause. This could be an engine fan, crankshaft, camshaft or other moving part at the same speed or one of the vibration orders.
The same applies to tires. Just because you found a vibration that correlates to tires doesn’t mean it is the tires. It could be a wheel bearing or other part that moves at the same speed as the tire.
The third major component that is measured is a drive shaft or prop shaft(in rear wheel drive vehicles). These typically rotate at a fixed rate that is related to tires but also by a multiplier of what your rear end ratio is. The rear end ratio is usually not a whole number like 4 to 1 or 4:1. It is usually something like 3.73:1, 4.11:1 or 3.23:1. This allows us to differentiate it from tire vibration orders fairly easily.
To combat this, the Dealers have tools at their disposal. In particular, in recent times they have a PicoScope with NVH modules – https://www.picoauto.com/products/noise-vibration-and-balancing/nvh-overview. There is a specific version for GM. It has a price point into the thousands of dollars depending on the model of unit and how many NVH modules. More NVH modules gives you different reference points for the concern and helps isolate it to the area of the vehicle.
Once nice thing about PicoScope though is the software itself is free. If you can get a dealer or someone else to run the diagnostics and send you the file, you can open it up and view/analyze it.
Lower End tools
There are other options. Since smart phones have the necessary equipment to capture many of the metrics necessary, one in particular called “NVH App” by VibrateSoftware caught my eye. These types of software use your phone’s accelerometer to detect vibrations and its GPS to detect speed. It is unfortunately out of most people’s budget @ $399.00USD. It does put it closer to budget for smaller repair shops though or hobbyists that deal with these kinds of issues frequently. Personally I think if they had a 3-4 day subscription for $50, more people would go for it or allow a yearly repair shop price of $399 it may do better. Even AC Delco’s TIS2WEB lets you have 3 day prices on their software. In any case it is very promising.
If you really want to save a buck though, there are quite a few vibration apps not specifically geared towards this use case that you can use. I went through a few of them and came across myFrequency by APPtodate. It was fairly economical at around $9. The main feature you need is the ability to detect multiple frequencies at the same time. I won’t claim to understand the math behind it but I believe it has to do with determining velocities of the vibrations to distinguish them from each other when you’re using one accelerometer.
In this particular example I was driving my truck at 50mph and maintaining 1200rpm. The tires are P275/55R20 and rear end ratio is 3.23:1. We’ll get into later on what that all means. From here you can see the top frequency by amplitude is 26Hz. Drilling into the app it is at 53.52mg which is substantial.
Here you can see the full frequency spectrum
This does however require you to do quite a few calculations and use a constant RPM and speed for the duration of the test. These tests are usually only 10-20 seconds so it is possible.
Deep Dive Into Math
As discussed, engine Hz is the easiest to calculate. Simply divide RPM by 60 to get RPS or Hz. In the above case 1200RPM / 60 = 20Hz.
At 50mph, the tire RPM becomes a little more lenghty. My tires are P275/55R20 which using a tire calculator like https://tiresize.com/calculator/ comes out to 31.9 inch diameter. 31.9 * 3.1416 (constant of PI) gives us 100.2 inch circumference. We already knew that because the calculator provides it. Divide that by 12 and we get 8.35 foot circumference. Every rotation of the tire goes 8.35 feet.
Every mile has 5280 feet in it. 50 miles per hour multiplied by 5280 gives us feet per hour. Divide that by 60 to get 4400 feet per minute. Divide that by 8.35 feet to come to 526.95 RPM. We can then divide that by 60 to get 8.78 as our first order tire vibration.
The rear end of 3.23 means that that prop shaft turns 3.23 times per one turn of the tire or every tire RPM/Hz equals 3.23 turns of the prop shaft. Multiplying 8.78 by 3.23 comes out to 28.36 Hz.
So now we have our first order vibrations of Engine 20Hz, Tire 8.78Hz and 28.36Hz of the prop shaft. We’re also looking for direct multiples of those as well.
Reading the Graph
Pointing out some of the peaks of the graph, you can see we have a 20Hz disturbance with a high amplitude and a 26Hz one as well. 20Hz matches up to the engine and 26 is fairly close. Usually it would have to be closer but I’ve not found the proper mounting point of the vehicle yet. I’m also just starting to take these measurements.
Some things could make the tire and prop shaft skew like not keeping exact speed and the fact that while tire diameters are the factory spec, when you put a vehicle on them they will not have a perfect diameter that matches. Tire wear can also play a little in the numbers.
In any case, this particular one seems to point more to engine RPM so I’ll likely take it for another run and bring up the engine RPM. This particular truck has had torque converter lockup issues and pulsates the TCC at around 1200RPM so I could be catching that pulsation on the TCC.
If you don’t have access to a shop that has a PicoScope or similar NVH and don’t have the budget for the NVH app, you may have what you need to perform these calculations on your own. In my case I created a spreadsheet to calculate these frequencies based on engine RPM, speed and a few vehicle variables. For my test runs I just need to plus in the variables and see where the graphs line up.
Phone mounting is a huge variable in this. The more professional tools have heavy duty magnets that you adhere to the seat rail which is an excellent position to detect vibration. You can move them to various seats to find the source of the vibration as the amplitude is limited.
With a phone it is more difficult to find mounting points in various spots int he vehicle but hopefully this gives you an economical method to track down vibrations.
This is another automotive post on my journey through P0711 on my GM 8L90 transmission for my 2015 Yukon. It is on the K2XX platform. At 5 years old it is starting to get some mileage and wear as seen in GMT K2XX Magnetic Ride Control Shock Inspection.
Monday morning I was out for a drive. With COVID all around us, I find I don’t get out of the house often. My truck still needs some miles on it or it would just sit for a while. All of a sudden I look down and notice the check engine light. I think this is the first time I have actually seen one on my truck. Despite being 5 years old, it only has 38k miles on it.
I’m fairly technical and like to do my own diagnostics when possible. I do it so that I can speak intelligently to the repair shop(dealer in this case). My truck was in the shop a few days on and off last week for some tire issues so I also wanted to avoid an unnecessary visit.
Recently I acquired a Foxwell NT510 Elite. Its a pretty useful bidirectional scan tool. Bidirectional means it can not only read codes but it can do active tests as well and minor calibration resets. It seems to have most of the features of the more expensive ones but it is locked/licensed by the vehicle brand and usually comes with one brand free. In my case GM. You can purchase other brands and add them to the unit. On this model its a lifetime purchase which is nice.
In my case, I checked the codes starting with the ECM and had a P0700 – Transmission Control Module Requested MIL Illumination. This simply told me to check the Transmission Control Module which showed me P0711 – Transmission Fluid Temperature Sensor Performance. Here is a good link someone on www.gm-trucks.com shared with me – https://www.dtcdecode.com/GMC/P0711
The TCM wants to know transmission temp so it can make informed decisions on shift pressures. It will also do things like not lock up the torque converter clutch if it is too cold. I imagine the shift adapts require certain temperature ranges too, to learn properly.
Doing a bit of reading, seems many people have had failed Transmission Fluid Temperature (TFT) Sensors and they just fail. This was more so on the 2016 models which switched to a 1 piece hardness that encompassed the sensor.
On my dash, if in tow mode, I can tell TFT and it was showing a value. Using my scantool, it showed -40F originally but that ended up being a firmware bug and Foxwell sent me an updated code which fixed it. Cold ambient temp from letting it sit over night seemed in line with TFT. Monitoring it further though, the transmission never warms up to the recommended 195-200F. It also had a few cases where the temp dipped to 100 on the dash (lowest number).
On the 6L80 and 8L90 series, they introduced a thermostat to help increase the temperature to around 200 on daily driving. It is supposed to support better shifts. Others have said it also helps boil off the moisture which has been a huge problem for these transmissions. There is a small subset of people that feel this is too warm and they actually “delete” the thermostat to make it run cool constantly. They are either in much hotter climates or don’t believe the engineers at GM did it right. There is an old line of thinking that 200F will cook a transmission. They could be right but the warning temp on my dash indicates 300 and these newer fluids are supposed to easily get into the 240-250’s. Nobody on the forums with a properly working transmission seems to hit anywhere above 220 though and that’s under extreme loads. Normally the thermostat opens at 192 and any transmission cooler helps keep it down from much above that.
In any case, after clearing the codes one day, the next day only P0711 returned. It requires a second time to trigger before check engine shows up. I realized my Foxwell supports reading Freeze Frame. Its kind of like a black box but under less severe circumstances. In instances where certain codes are set, it will capture parameters that surrounded it. In my case I lucked out and it captured a 48F degree TFT which is definitely out of line
I was even able to graph the temperature over the drive. It does seem like there is a bit of fluctuation but that could be because its not allowed to warm up with a possibly malfunctioning thermostat. Or the harness could be loose. Or the sensor could be inaccurate!
Replacing Transmission Fluid Temperature Sensor
There are quite a few videos on this. Here is a great one if you have the one piece harness, whereas at this time I think they switched back to the two piece which is less involved.
Since I have a GM Protection Plan (extended warranty), I’m not terribly interested in possibly voiding it. I also don’t have the expertise or comfort level to start toying with the transmission itself as the temperature sensor is in the pan. Appointment to the dealer made and I may update this Monday with the results. Regardless, anyone that gets a P0711 on a K2 platform on the 8L90 or even 6L80 transmission, this may help you diagnose it if you’re out of warranty. It could be an economical sensor/harness or even an easier to replace thermostat.
Yes, this blog does tend to touch on other areas I am passionate about other than IT related topics. I’m an engineer at heart so I love investigating and understanding things, particularly those that tend to lead me down a rabbit hole.
About 6 months ago I started what would be my journey of understanding how to inspect and diagnose Magnetic Ride Control shocks. I took in my 2015 GMC Yukon Denali to the dealer for what I thought was going to be an oil change and a recall that simply required a code update. I was kindly informed that my front strut was leaking and needed replacement. Hopeful that my extended warranty might cover it, I was also kindly let know that was not the case. This was a maintenance item and parts & labor were going to be roughly $900.
Fast forward a few months and my original extended warranty was about to expire so I purchased a new one that came with a 30 day / 1,000 mile wait period. With COVID-19, I was doing 80 miles a month. I had to step it up and started doing 50-80 miles a day to make up for it. Towards the last 200 miles I realized the ride was very rough. I was not sure if the roads were terrible or not driving so much had increased my expectation of a smooth ride. Remembering that I had a leak on my front strut, I decided to take a look at the remaining strut and shocks.
Magnetic Ride Control
For those that do not know, Magnetic Ride Control is a premium suspension option in select GM vehicles. It has been around circa 2003 but is now on its 3rd iteration. There are many sources that can do a better job of explaining it than me but essentially instead of regular fluid, there is a magnetic fluid and an electromagnetic that is used to in realtime make adjust based on many driving conditions such as gas pedal position, steering position, incline of vehicle and ride height.
The rear shocks in the vehicle also have air springs that are inflated or deflated to provide auto leveling on top of this. This allows the vehicle to maintain its level independent of the load (to a degree).
This is my first time having to deal with these particular shocks so I went to a few sources of information such as r/MechanicAdvice on reddit. Someone responded but they indicated that these shocks can’t leak as the air spring would contain the fluid. I would later find out this was partially correct.
Deciding to wipe it down and see if more fluid showed up, it did. This caused me to reach out to a few personal contacts and their opinion was that they were indeed leaking.
Bite the Bullet and Replace
After wiping down again, I was able to find the part number AC DELCO 23290661 which had been replaced by AC DELCO 84176675. The new part number hopefully indicated I might get more than 5 years / 37k miles on these! I shopped around and found Rock Auto had the best price for OE replacements although Arnott seemed to have a nice rebuild for $100 per shock cheaper. Maybe next time I’ll go that route.
They arrived and they were beautiful! Clear as day what they should look like.
I’m a hobbyist and have done shocks before but just did not want to mess with this so I hired a local mobile mechanic to throw them in. Check out their Instagram video of them on the job! – I’m not sure why it says its violent but I promise its not!
After removing the old ones, they sure looked like they were in worse shape than I expected.
While learning about this I was curious of the anatomy of this. People had described it through my investigations but I’m a visual learner. I did come across this Youtube video on a Mercedes Air Shock which was similar but I wanted to see mine and how they failed.
Z95 Rear Shock Tear Down
To tear it down I used some rubber gloves as the magnetic fluid can be fairly abrasive, a very sharp box cutter and some pliers/cutters.
At the end you can see a fully opened MRC shock. The lines point to the retaining metallic bands that hold the bladder into place. With it removed its essentially a regular shock in appearance.
So technically the bladder should hold the fluid but apparently the abrasiveness of it causes it to start leaking out over time by eating away at the rubber.
Now that I am familiar with these shocks and how they fail, it does appear the remaining strut in the front is leaking and time for a replacement. Luckily I came across this very informative video on how to do it. I have a buddy with a spring compressor and better yet knows how to use it so wish me luck!
I came across an interesting idea of extending the life of old by using ChromeOS. What is ChromeOS? We’ll get there. The problem is that many households have an old computer laying around collecting dust. It lacks either memory, hard drive space or compute power to be useable though. At the same time, with everyone social distancing, there’s a need for an extra device. You may have children trying to share devices to get on the internet or need an extra one for a family member that is in your house. This article may help you with that.
What is ChromeOS?
On wikipedia, ChromeOS is described as the following.
Chrome OS is a Linux kernel-based operating system designed by Google. It is derived from the free software Chromium OS and uses the Google Chrome web browser as its principal user interface.
For those non technical, it is basically an Operating System that is minimized to just the basic needs to get on the internet. Its assumption is that most uses cases for computers these days are to access the internet. When it boots up and you get logged in, its primary user interface is Google Chrome.
Google does not provide installation files to install ChromeOS onto your legacy hardware. ChromeOS is designed to come pre-installed on its Google ChromeBooks.
ChromeOS is open sourced which means other companies can take its code and build upon it. A company named neverware has done just that. It does have paid offerings for education and business but there is also a free download for home users.
What You Need
8GB or larger USB stick
Operational computer you wish to install ChromeOS onto that has 2GB of RAM and 16GB harddrive
Ideally a newer and operational computer to generate a bootable USB stick.
A Google Account (you can create one as a part of this process)
If you have a Windows machine, the installation is fairly straight forward as you will download the windows “USB Maker” and insert your USB drive into that computer and go through the guided wizard.
Once the bootable USB drive is created, insert it into the destination computer and boot or reboot it. You may need to go into that machine’s system BIOS to select a different boot order or ensure that it will boot off the USB drive. Sometimes during boot there is a keystroke to press such as F12 or ESC to select that. You may have to refer to your computer’s user guide on that. neverware has a list of these for the major manufacturers – https://guide.neverware.com/install-and-setup/boot-usb/
When booting off the USB drive, it will boot directly into ChromeOS or neverware’s version of it called CloudReady. It is fully usable at this point so even if the computer’s harddrive is crashed you could perform basic tasks. Ideally after booting you will go through an install process to install it locally and not require the external USB stick.
During the boot process you will see the following Welcome screen
Next it will require some sort of network connectivity. If you have a wired connection you are good to go. If you are on wifi, it will ask you to authenticate to your network.
Once connected to the network, it will ask you to login to your Google account.
Finally – here we are, a Chrome browser. The above is a one time setup process. Sometimes two times. When doing the above on the Live environment, after installing you may need to go through this once more.
After you have used the live environment for a while and decided you want to permanently convert that computer to it you can click on the clock and select “Install OS”
No Google App or Play Store due to licensing issues
This is not Microsoft Windows and your apps are limited to what’s in the Chrome Extension store
At this point you will have a neverware’s version of ChromeOS installed and have basic web browser functionality on this device. It should be more than sufficient for basic browsing since that is the typical use-case. Hopefully it has helped you make some use of a device that was just collecting dust.
I had a fun issue today. All of a sudden BIND stopped returning results for recursive queries to external zones.
My logs were filled with lines like the following
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]: validating @0x7f3458067420: dlv.isc.org NSEC: verify failed due to bad signature (keyid=64263): RRSIG has expired
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]: validating @0x7f3458067420: dlv.isc.org NSEC: no valid signature found
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]: error (network unreachable) resolving 'com.dlv.isc.org/DS/IN': 2001:500:2c::254#53
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]: validating @0x7f3450030b60: dlv.isc.org NSEC: verify failed due to bad signature (keyid=64263): RRSIG has expired
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]: validating @0x7f3450030b60: dlv.isc.org NSEC: no valid signature found
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]: validating @0x7f3450065e40: dlv.isc.org NSEC: verify failed due to bad signature (keyid=64263): RRSIG has expired
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]: validating @0x7f3450065e40: dlv.isc.org NSEC: no valid signature found
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]: error (no valid RRSIG) resolving 'com.dlv.isc.org/DS/IN': 188.8.131.52#53
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]: validating @0x7f3450065e40: dlv.isc.org NSEC: verify failed due to bad signature (keyid=64263): RRSIG has expired
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]: validating @0x7f3450065e40: dlv.isc.org NSEC: no valid signature found
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]: error (no valid RRSIG) resolving 'com.dlv.isc.org/DS/IN': 184.108.40.206#53
Naturally I tried bouncing named without luck. I then thought there was an issue with the root zones and configured forwarders without luck. I had disabled dnssec via “dnssec-enable no” without luck.
This seemed fairly strange. Ultimately since it was DNSSEC related I opted to disable it via as a temp workaround. It appears the validation was the issue.
After some investigation and troubleshooting it appeared to be related to ISC’s DLV and letting RRSIG expire accidentally. It failed in an unexpected manner when this happened.
What is DLV?
DLV stands for DNSSEC Lookaside Validation. DLV is a service that ISC has provided since circa 2006. It allowed DNSSEC to be enabled on zones that could not otherwise be enabled. Not all Top Level Domains (TLD) implemented DNSSEC until the past few years. This was a workaround to allow DNSSEC until then.
In 2017 it was finally decommissioned with DNSSEC being fully available to all TLDs. The A record was left in place and many resolvers still attempt to connect but it does not provide any data.
What is RRSIG?
If you want a full view of DNSSEC and how it works, CloudFlare has a great article for that here – https://www.cloudflare.com/dns/dnssec/how-dnssec-works/ . In short though, RRSIG records contain cryptographic details, particularly start and end dates for the validity of that data. This is much like an SSL Certificate that has a valid period.
The RRSIG records are designed to be required to be updated frequently to ensure the security much like SSL Certificates need to be renewed. This helps prevent a replay attack where an older compromised key is reused.
20200325160456 was the key, that translated to 3/25/2020 16:04:56 UTC which is about when the issue started. Further down in the “References” section the ISC-USERS list confirmed this was by mistake. I suppose it was a good “scream” test to remove lookaside. Newer BIND versions do not even support this anymore.
On many older BIND servers deployed before 2017, they were configured with the following.
Auto would try to query dlv.isc.org first and then query root name servers. The expected behavior was that it simply would not return any data and then the root zones would be queried.
Unfortunately with an expired RRSIG it failed in a way that made BIND think the query response was not valid and an expected failure. For all BIND knew, it was preventing a replay attack.
Recently a client had a need for putting a web application on the internet that end users could access. They wanted to lock it down so that not everyone on the internet could access. Whitelisting IP Address was not an option because they were remote users with dynamic IPs and the overhead of maintaining this whitelist would be problematic.
The use case was a password recovery tool that their remote users could use to reset and recover passwords. Simple authentication would not suffice. For starters if the users’ passwords expired they wouldn’t be able to easily log into the site. Along with that it would be a high profile target for brute forcing.
Why Not IP Whitelisting?
IP whitelisting used to be and still us for some organizations the de-facto method of filtering traffic. It can be very problematic though. Users today are on the go, working remotely or using their mobile device on cellular data as well as home internet. Other times it involves sales staff at client sites. Keeping up with these IP whitelists can be a chore. Updating this whitelist can be time sensitive to avoid halting productivity. When not maintained, there is a chance someone unexpected could gain access due to simply having an IP previously whitelisted.
A workaround for this is VPN but that requires a bit of support overhead in user training and support. This can be clunky for users that are not used to to using VPN.
Many larger organizations already have internal Certificate Authorities in place. For Microsoft Active Directory deployments, when CA has been installed, end users are likely auto enrolling in user certificates. Domain joined workstations already have these and trust it the internal Root CA.
Certificates also have a built in expiration. In an auto enrollment environment, this expiration could be lowered substantially to below 1 year.
Once of the nice features of TLS is that it does include a mechanism for this. Below is an example of a TLS handshake where the server requests a certificate and the client provides it.
In Frame 19, the client makes the TLS request with a Client Hello. Frame 23 the Server response with a Server Hello. This is where they set parameters and negotiate things like TLS versions and encryption algorithms.
Frame 26 is part of the Server Hello but it was large and split up. Boxed in red is the “Certificate Request” where the server is requesting a certificate to authenticate.
Frame 33 is where the client actually provides it.
From here you can see this happens before the application level (HTTP) protocol communicates starting in frame 43. What this means is that before the user reaches the web application for authentication, the device requiring TLS Certificate Authentication is filtering the requests. Many times this is a reverse proxy or load balancer is not vulnerable to the same exploits as the web servers.
When used properly and the client has a certificate, the browser will prompt users for a certificate to use such as pictured below.
A really neat application for this when you have a legacy plain text protocol in play but you want to open it up over the internet and secure similarly. Perhaps you have a legacy application that uses raw text and is not SSL/TLS capable. You can still put this on the internet through a reverse proxy like F5 LTM or stunnel.
Traditionally this type of traffic would be protected via IPSEC tunnel that is encrypted or a dedicated circuit such as MPLS. That does require specific hardware and/or monthly circuit costs to accommodate.
stunnel is extremely useful in this scenario as you can install it on the local machine that has the legacy application and configure it to connect to localhost on a random port and proxy information out over TLS and configure it to use the certificate based authentication.
Here is a graphical example of what that may look like with an stunnel server broken out. stunnel could be installed on the end user’s workstation though.
stunnel could be put on the local end user workstation to minimize that unencrypted leg. Typically on the server side the reverse proxy has a leg directly on the same VLAN/subnet as the application which minimizes exposure over that but this does help secure the application traffic over the untrusted internet.
In this article we learned a little on Certificate Based Authentication. We also learned how it may help your organization better secure your applications and possibly avoid more costly solutions.