ISC BIND Look Aside Related Outage

Summary

I had a fun issue today. All of a sudden BIND stopped returning results for recursive queries to external zones.

My logs were filled with lines like the following

Mar 25 09:05:00 XXXXXXX named[XXXXXXX]:   validating @0x7f3458067420: dlv.isc.org NSEC: verify failed due to bad signature (keyid=64263): RRSIG has expired
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]:   validating @0x7f3458067420: dlv.isc.org NSEC: no valid signature found
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]: error (network unreachable) resolving 'com.dlv.isc.org/DS/IN': 2001:500:2c::254#53
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]:   validating @0x7f3450030b60: dlv.isc.org NSEC: verify failed due to bad signature (keyid=64263): RRSIG has expired
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]:   validating @0x7f3450030b60: dlv.isc.org NSEC: no valid signature found
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]:   validating @0x7f3450065e40: dlv.isc.org NSEC: verify failed due to bad signature (keyid=64263): RRSIG has expired
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]:   validating @0x7f3450065e40: dlv.isc.org NSEC: no valid signature found
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]: error (no valid RRSIG) resolving 'com.dlv.isc.org/DS/IN': 149.20.64.4#53
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]:   validating @0x7f3450065e40: dlv.isc.org NSEC: verify failed due to bad signature (keyid=64263): RRSIG has expired
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]:   validating @0x7f3450065e40: dlv.isc.org NSEC: no valid signature found
Mar 25 09:05:00 XXXXXXX named[XXXXXXX]: error (no valid RRSIG) resolving 'com.dlv.isc.org/DS/IN': 156.154.100.23#53

Troubleshooting

Naturally I tried bouncing named without luck. I then thought there was an issue with the root zones and configured forwarders without luck. I had disabled dnssec via “dnssec-enable no” without luck.

This seemed fairly strange. Ultimately since it was DNSSEC related I opted to disable it via as a temp workaround. It appears the validation was the issue.

dnssec-enable no;
dnssec-validation no;

After some investigation and troubleshooting it appeared to be related to ISC’s DLV and letting RRSIG expire accidentally. It failed in an unexpected manner when this happened.

What is DLV?

DLV stands for DNSSEC Lookaside Validation. DLV is a service that ISC has provided since circa 2006. It allowed DNSSEC to be enabled on zones that could not otherwise be enabled. Not all Top Level Domains (TLD) implemented DNSSEC until the past few years. This was a workaround to allow DNSSEC until then.

In 2017 it was finally decommissioned with DNSSEC being fully available to all TLDs. The A record was left in place and many resolvers still attempt to connect but it does not provide any data.

What is RRSIG?

If you want a full view of DNSSEC and how it works, CloudFlare has a great article for that here – https://www.cloudflare.com/dns/dnssec/how-dnssec-works/ . In short though, RRSIG records contain cryptographic details, particularly start and end dates for the validity of that data. This is much like an SSL Certificate that has a valid period.

The RRSIG records are designed to be required to be updated frequently to ensure the security much like SSL Certificates need to be renewed. This helps prevent a replay attack where an older compromised key is reused.

RRSIG Value

Running the following I could see it expired

# dig +dnssec dlv.isc.org

dlv.isc.org.		3599	IN	RRSIG	DNSKEY 5 3 3600 20200325160456 20200224153150 19297 dlv.isc.org. TyUbbNgG/Oru7TQFHbDC9E208hB8Szheu634Q03nawQFz4dosOFg+ZB5 z8Svh8fw/g35a/ZW5AP1jbSKh19u4c7Ujre3iygS0Tjycmi0mYG6dS7I CcWLOxZpOKf8uw9mzgbIR/VDEFmKj0OJKdkxAqfaWxXLqBBWgFqIucC6 9Tb98clinCPW34xgk6Fzi+OKAFmiGH6/e8wk/h5RMWxipx5KAk2NsWsw QMyEDaA7eLzZTbBenftVR86g6QO4bR+LOKzxGBFQ2XW0ArQKDiuoBqEw 8cmRcGKzVJ761d7EK+LDvnktRNxRMJ9y5LPgxlO2Xm3Un8oExjVbLKi7 OigQnA==

20200325160456 was the key, that translated to 3/25/2020 16:04:56 UTC which is about when the issue started. Further down in the “References” section the ISC-USERS list confirmed this was by mistake. I suppose it was a good “scream” test to remove lookaside. Newer BIND versions do not even support this anymore.

What Happened?

On many older BIND servers deployed before 2017, they were configured with the following.

dnssec-lookaside auto;

Auto would try to query dlv.isc.org first and then query root name servers. The expected behavior was that it simply would not return any data and then the root zones would be queried.

Unfortunately with an expired RRSIG it failed in a way that made BIND think the query response was not valid and an expected failure. For all BIND knew, it was preventing a replay attack.

References

I had originally posted on Reddit and was also pointed to ISC-USERS.

Using Certificate Based Authentication

Summary

Recently a client had a need for putting a web application on the internet that end users could access. They wanted to lock it down so that not everyone on the internet could access. Whitelisting IP Address was not an option because they were remote users with dynamic IPs and the overhead of maintaining this whitelist would be problematic.

The use case was a password recovery tool that their remote users could use to reset and recover passwords. Simple authentication would not suffice. For starters if the users’ passwords expired they wouldn’t be able to easily log into the site. Along with that it would be a high profile target for brute forcing.

Why Not IP Whitelisting?

IP whitelisting used to be and still us for some organizations the de-facto method of filtering traffic. It can be very problematic though. Users today are on the go, working remotely or using their mobile device on cellular data as well as home internet. Other times it involves sales staff at client sites. Keeping up with these IP whitelists can be a chore. Updating this whitelist can be time sensitive to avoid halting productivity. When not maintained, there is a chance someone unexpected could gain access due to simply having an IP previously whitelisted.

A workaround for this is VPN but that requires a bit of support overhead in user training and support. This can be clunky for users that are not used to to using VPN.

Why Certificates

Many larger organizations already have internal Certificate Authorities in place. For Microsoft Active Directory deployments, when CA has been installed, end users are likely auto enrolling in user certificates. Domain joined workstations already have these and trust it the internal Root CA.

Certificates also have a built in expiration. In an auto enrollment environment, this expiration could be lowered substantially to below 1 year.

TLS Handshake

Once of the nice features of TLS is that it does include a mechanism for this. Below is an example of a TLS handshake where the server requests a certificate and the client provides it.

TLS Handshake - Certificate Authentication
TLS Handshake – Certificate Authentication

In Frame 19, the client makes the TLS request with a Client Hello. Frame 23 the Server response with a Server Hello. This is where they set parameters and negotiate things like TLS versions and encryption algorithms.

Frame 26 is part of the Server Hello but it was large and split up. Boxed in red is the “Certificate Request” where the server is requesting a certificate to authenticate.

Frame 33 is where the client actually provides it.

From here you can see this happens before the application level (HTTP) protocol communicates starting in frame 43. What this means is that before the user reaches the web application for authentication, the device requiring TLS Certificate Authentication is filtering the requests. Many times this is a reverse proxy or load balancer is not vulnerable to the same exploits as the web servers.

Browsers

When used properly and the client has a certificate, the browser will prompt users for a certificate to use such as pictured below.

Browser Certificate Authentication Prompt
Browser Certificate Authentication Prompt

Other Applications

A really neat application for this when you have a legacy plain text protocol in play but you want to open it up over the internet and secure similarly. Perhaps you have a legacy application that uses raw text and is not SSL/TLS capable. You can still put this on the internet through a reverse proxy like F5 LTM or stunnel.

Traditionally this type of traffic would be protected via IPSEC tunnel that is encrypted or a dedicated circuit such as MPLS. That does require specific hardware and/or monthly circuit costs to accommodate.

stunnel is extremely useful in this scenario as you can install it on the local machine that has the legacy application and configure it to connect to localhost on a random port and proxy information out over TLS and configure it to use the certificate based authentication.

Here is a graphical example of what that may look like with an stunnel server broken out. stunnel could be installed on the end user’s workstation though.

Legacy App Secured with TLS 1.2 or higher & Certificate Based Authentication
Legacy App Secured with TLS 1.2 or higher & Certificate Based Authentication

stunnel could be put on the local end user workstation to minimize that unencrypted leg. Typically on the server side the reverse proxy has a leg directly on the same VLAN/subnet as the application which minimizes exposure over that but this does help secure the application traffic over the untrusted internet.

Final Words

In this article we learned a little on Certificate Based Authentication. We also learned how it may help your organization better secure your applications and possibly avoid more costly solutions.

Internet Routing and BGP Looking Glasses

Summary

From time to time I get requests from colleagues, “Can you ping this address?”. Many times what is going on is they are bringing up a new internet link and want to check routing. Sometimes they areadding a public endpoint and want to make sure its accessible. They are asking me because within their network it works but they need to make sure it is accessible or routing properly over the internet.

BGP Looking Glasses are a great tool for this if you would like to be self sufficient. You can also get a wider view than just a few colleagues. My favorite one is HE.net’s https://lg.he.net

What is BGP?

BGP Stands for Border Gateway Protocol. It is the standard exterior gateway protocol for internet routing. While an internal network may use something like OSPF for its interior, BGP is better suited for exterior routing.

One of the few keys to understanding BGP at a high level is to understand it is a distance vector routing protocol. These routes are typically better suited for WAN routing as they are a bit more light weight. The downside is that each router’s perception of the internet is key. There is no unified single view of the internet routing. Each router has its own perception of the internet based on the BGP routes it receives.

This is why Looking Glasses are so important. You can see the perception of various points on the internet. This can help you determine internet traffic destined to your infrastructure is taking the expected and optimal path.

HE.net’s Looking Glass

Here is a small snippit of HE’s LG. They have various routers all over the world that would not fit into this screen shot. You can also see the various functions you can perform on the right hand side.

HE.net Looking Glass
HE.net Looking Glass

Ping and traceroute are fairly self explanatory. For ping it just returns the results of an ICMP ping and traceroute shows a list of routers in the path of getting to the destination.

HE.net Looking Glass - Traceroute
HE.net Looking Glass – Traceroute

The real value here is the “BGP Route” option.

HE.net BGP Looking Glass Details
HE.net Looking Glass BGP Details

Here we can see all of the BGP peers this particular router has learned the route to 1.1.1.0/24, the AS path it takes and which one it selects as the best path.

Autonomous Systems

If you are new to BGP and dynamic routing protocols you may be wondering what an AS (Autonomous System) is. In BGP world, it is basically a grouping of similar routers that announce a similar set of subnets or prefixes as they are called in BGP. BGP groups systems together by AS. As each router learns a route, it appends its own AS to the AS path before passing it along.

The above is a bad example because it shows a single direct AS path as HE appears to be directly peered with CloudFlare (AS 13335). CloudFlare is very well peered on the internet. Below is a better example. It at least shows it passing through AS 1299 (Telia) to AS 174 (Cogent)

It seems HE.net is fairly well peered but here is another router output that shows some decent AS paths and the differences. AS 174 being Cogent and AS 209 being CenturyLink and AS 3356 being Level3 it chooses the shortest AS path. Keep in mind the traceroute through CenturyLink could possibly be shorter actual router hops. A shorter AS path does not necessarily mean less latency or shorter traceroute hops.

#show ip bgp 73.0.0.0/8 
BGP routing table entry for 73.0.0.0/8, version 767462940
Paths: (2 available, best #1, table default)
  Advertised to update-groups:
     18        
  Refresh Epoch 1
  174 7922, (received & used)
    X.X.X.X from X.X.X.X (X.X.X.X)
      Origin IGP, metric 13031, localpref 100, valid, external, best
      Community: 174:21000 174:22013
      rx pathid: 0, tx pathid: 0x0
  Refresh Epoch 3
  209 3356 7922, (received & used)
    X.X.X.X from X.X.X.X (X.X.X.X)
      Origin IGP, metric 7800026, localpref 100, valid, internal
      rx pathid: 0, tx pathid: 0

Final Words

If you ever find yourself needing to ping or traceroute from a remote endpoint lg.he.net has you covered. Many carriers have their own looking glass. This is useful incase you want to see how your routes are perceived from their end. If you use BGP at your edge and receive full routes, this is another avenue of seeing those prefixes. With looking glasses, you can do this from various endpoints across the internet pretty easily.

OpenVPN with Encrypted Private Key – Issue Resolved

Summary

I was working on Azure Client VPN with OpenVPN and in testing I had removed the passphrase on the private key for authentication but wanted to put it back on there and it would not work. Some quick searches did not turn up much other than common complaints of this.

Reason

With certificate based authentication on OpenVPN, the public key and private key are put in the ovpn file. This is not the most secure with an unencrypted private key as anyone can simply obtain the file and connect.

With a passphrase on it, there is less concern over the ovpn being disseminated and the key reused.

Issue

The OpenVPN client logs indicated the following

2019-12-20 12:22:42-0600 [-] OVPN 40_89_167_211_p4967 ERR: ">FATAL:CLIENT_EXCEPTION: connect error: PEM_PASSWORD_FAIL: mbed TLS: error parsing config private key : PK - Private key password can't be empty"
2019-12-20 12:22:42-0600 [MyOMIClient,0,] FROM OMI: u">FATAL:CLIENT_EXCEPTION: connect error: PEM_PASSWORD_FAIL: mbed TLS: error parsing config private key : PK - Private key password can't be empty"
2019-12-20 12:22:42-0600 [HTTPChannel,224,] *** API CALL f=xmlrpc_Poll args=['sess_40_89_167_211_p4967_VuxJ7YkTyUVYXeKV_1', 10] kw={} ret=[{'active': True, 'timestamp': 1576866162, 'type': 'ACTIVE', 'last': None}, {'timestamp': 1576866162, 'type': 'FATAL', 'error': u"CLIENT_EXCEPTION: connect error: PEM_PASSWORD_FAIL: mbed TLS: error parsing config private key : PK - Private key password can't be empty"}]

It seemed like it was just not prompting me to enter the passphrase and using a blank one.

Solution

I came across the following article ( https://github.com/pivpn/pivpn/issues/372 ) which had quite a few tangents on it but most recently someone indicated using OpenVPN Connect 3.1.0 and it worked. I tried that and was surprised, it actually prompts me for the passphrase!

It seems the community version of the OpenVPN GUI client supported this but the OpenVPN Connect lacked the feature until recently. As of the time of this writing 3.1.0 is Beta but seems to work great!

I had tried upgrading from 2.5 to 2.7 without luck. Finally after installing 3.1.0 it worked again.

Azure Client VPN with OpenVPN

Summary

In my article Intro To Azure Active Directory Domain Services we discussed environments with minimal infrastructure. With all of the RDP exploits it is typically best not to expose RDP over the internet. Since Bastion is not yet fully available the next best thing aside from setting up a VPN appliance is to use the Point-to-site functionality of a Virtual Network Gateway.

Prerequisites

The first pre-requisite for client VPN using a Virtual Network Gateway is to actually provision one. For OpenVPN compatibility it does require at least SKU VpnGw1 and will not work with basic.

It will require 2 subnets, one for the inside leg of the gateway and another for the client-side pool.

The Virtual Network Gateway does want an inside subnet dedicated for use to the Virtual Network Gateway and not shared amongst other devices.

Authentication is handled either via radius or certificate based. If you are reading this article for a minimized infrastructure you probably do not have radius servers.

Provisioning

The provisioning process is fairly simple although it can take 30-60 minutes for the Virtual Network Gateway to fully provision before you can use the Point-to-site configuration. There are a few simple questions.

Virtual Network Gateway - Create
Virtual Network Gateway – Create

That’s really it for the initial provisioning.

Configuration

Some basic Point-to-site configurations need to be set.

Point-to-site configuration
Point-to-site configuration

The next part is the most difficult part of this. A root and at least one child certificate have to be provisioned. Microsoft has some good documentation on it. To do in Powershell, it does require Windows 10 or Server 2016 or higher.

Root Certificate
Root Certificate

The name is arbritrary but the “Public certificate data” is the area between the “—BEGIN CERTIFICATE—” section and the “—END CERTIFICATE”

The following Microsoft article describes and outlines the process much better than I can do so I will just share it here – https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-certificates-point-to-site

Client Configuration

Again – Microsoft does a really good job on instructions for configuring the client so I will just share this link – https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-howto-openvpn-clients

Final Words

It can be a pain for those of us not familiar with certificates and command line tools like openssl. The idea is that you have a root certificate authority that then issues individual certificates per user or group of users. If that key becomes compromised you can then revoke the individual certificate or untrust the entire certificate authority. I like the idea of creating a CA chain per organization you grant access to.

In this article we walked through creating the resources required and configuring. We did rely heavily upon the Microsoft documentation but it was fairly complete and well shown.

Make sure you distribute your ovpn files with encrypted private keys! – OpenVPN with Encrypted Private Key – Issue Resolved

Intro To Azure Active Directory Domain Services

Summary

Recently I got the pleasure to stand up Azure Active Directory Domain Services (AADDS) for a client. It has some specific use cases and limitations but can be great for your environment.

This article walks you through the background, some of the limitations and also how to provision.

What is Azure Active Directory Domain Services?

This is a natural question. At a high level, let’s talk about the use case for it. In a small cloud environment with a minimal number of VMs, standing up redundant Domain Controllers can chew up a bit of your budget. From there you have to manage and monitor the domain controllers.

AADDS is a managed platform for this. On the back end it appears to be appliances that provide most of the functionality of Active Directory but with some limitations. It also offers a one way sync from Azure Active Directory (AAD) to AADDS. This allows you to use Azure Active Directory credentials to login to VMs.

Use Case

As touched on above, most administrators want a central credential repository for their cloud VMs. If they are used to Active Directory on prem but want to tie this to Azure AD, AADDS makes sense, particularly in small environments where there may not be a full budget to host, manage and maintain full domain controllers with adequate redundancy.

Limitations

There are some notable limitations so if you are looking for the full flexibility of native Domain Controllers, AADDS is not that.

One Way Replication

Replication only occurrs from AAD to AADDS. You can make changes in AADDS via normal Active Directory Users and Computers (ADUC) or other tools but they do not replicate back to AAD. AAD can however overwrite these changes.

Azure Active Directory does support AD Connect to connect to on prem domain controllers but it is not compatible with AADDS at this time.

No Custom Attributes

It does not support extending the schema or adding extra custom attributes

External Guests

While external guest accounts will replicate into AADDS, their password and hashes are inaccessible so you will not be able to login under these accounts. An AAD account local to tenant AD will have to be provisioned.

Lack of Control

When you create or change AAD attributes, you have no control over the replication interval. The initial sync may take an hour or two. The provisioning of AADDS takes quite a bit of time as well. In practice, on a smaller environment, replication does seem to happen within 5-15 minutes.

Provisioning

The first step is to select this from the Market and click create

Azure AD Domain Services
Azure AD Domain Services

On the basics tab, pay particular attention to the DNS Domain name. Best practices for domain naming has changed over the years. These settings cannot be changed after implementing.

Azure AD Domain Services - Basics
Azure AD Domain Services – Basics

For reference here is the output of the “Help me choose the DNS name”. Also – here is another Microsoft article on best practices – https://social.technet.microsoft.com/wiki/contents/articles/34981.active-directory-best-practices-for-internal-domain-and-network-names.aspx

Picking Active Directory DNS Name
Picking Active Directory DNS Name

On the networking tab it will by default create a new Virtual Network and Subnet. You can choose an existing virtual network which is highly likely. It does however, indicate you should use a not in use subnet for this. I have not found much information on that but it may have to do with the Network Security Group it creates and applies.

Azure AD Domain Services - Networking
Azure AD Domain Services – Networking

On the next screen you will elect who can manage this environment. Users in this group will function as the “Domain Admins” you are used to in Active Directory.

Azure AD Domain Services - Administration
Azure AD Domain Services – Administration

Next we can choose which objects to synchronize but typically it will be all objects. If you think you may want to scope it though, choose that first as you can go from scoped to all but have to recreate to reverse.

Azure AD Domain Services - Synchronization
Azure AD Domain Services – Synchronization

Finally, we review the config but pay note to this. In order for this all to work, your AAD hashes have to be stored in the AADDS.

Azure AD Domain Services - Review & Create
Azure AD Domain Services – Review & Create

Final Words

In this article we gave a brief overview of Azure Active Directory Domain Services and some use cases. We also went over some limitations and cases where it would not be a good fit. Finally we walked through the actual deployment.

Parsing CSV Files with Golang

Summary

I love coming across problems that require me to learn something new. I have written a few posts about Go (or Golang) such as Using Recursion on Golang to Solve a Problem and Unit Testing Golang on App Engine. Go is one of my favorite utility languages.

Earlier this year, my wife decided to start her own hair salon and do booth rental (Pretty Hare Salon). We set up square for appointment setting and credit cards. It has nice reporting features but there is literally no forecasting. Forecasting helps us determine pricing and when to increase or decrease marketing/advertisement.

Parameters of the Problem

To work around this lack of forecasting, I found an export feature that exports appointments in csv. Unfortunately it does not list the pricing associated with it. At the time, I implemented something in perl but decided to rewrite in Golang. The application inputs two CSV files (1 – appointments and 2 – pricing) and calculates weekly totals based on that.

Helper Function

Error checking and reporting requirements are fairly minimal for this so I use a helper function that is fairly basic.

func CheckErr(err error) {
	if err != nil {
		log.Fatal(err)
	}
}

Loading CSVs

In order to load the CSV into a variable, I opted to use os.Open() which takes the file path. I pass these in through command-line arguments using os.Args. os.Open returns a pointer to a file.

args := os.Args

	appointmentsFile := args[1]
	pricesFiles := args[2]

	af, err := os.Open(appointmentsFile)
	CheckErr(err)

	pf, err := os.Open(pricesFiles)
	CheckErr(err)

I then open the files return by these with cvs.NewReader

appointments := csv.NewReader(af)
prices := csv.NewReader(pf)

NewReader() expects an io.Reader but *os.File implements a reader so we are good.

These readers return a 2 dimensional slice with the left most dimension being the line number and second dimension being the field.

For the price records I read it in entirety. It is not very long.

priceRecords, err = prices.ReadAll()
CheckErr(err)

Iterating the CSV Line by Line

The Appointment file is the full history and we do not need all the data in it. It can get very large over time so it is best not to completely load it into memory. We also will have logic to ignore anything older than 2 weeks as this is a forecast and we do not care about historicals.

for {
	appointment, err := appointments.Read()
	if err == io.EOF {
		break
	}
	// Business logic in here
}

The appointment array in this can then be read as a single dimension. Something similar to what is below exists in the for loop previously shown.

if appointment[2] == "accepted" {
	// More business logic here
}

Looking Up Data

The pricing information which we fully loaded into the two dimensional slice can be iterated as follows

for _, price := range priceRecords {
	if price[0] == service {
		return strconv.ParseFloat(price[8][1:], 64)
	}
}

return 0, errors.New("Could not find service - " + service)

The “service” variable is the service we are looking up. In the price slice, the first or 0 position is the actual service name we are looking up and position 8 is the actual price.

The price field (8) has a dollar sign so I just used string slicing to omit the first character which is the dollar sign so I could parse the float like this

price[8][1:]

The pricing file is highly controlled so I have very little error checking here.

Final Words

In this article we discussed opening and iterating through CSV files using two methods. One involves fully loading the file into memory and the other involves iterating line by line. At this point you should have a good idea of which method to use and when.

Using Recursion on Golang To Solve A Problem!

Summary

I was writing an SSL Certificate Decoder ( https://tools.woohoosvcs.com/certdecoder/ ) and came across an interesting problem. I wanted to validate a certificate chain but had some difficulty figuring it out. The problem was that I needed to find the most child certificate in a seemingly random list of certificates. What I am calling the most child certificate is the one at the bottom of the chain that is issued but does not issue one of its own. Users can provide the cert chain in any order so we cannot relay on the order they are submitted for this tool.

SSL Certificate Chains

To understand the problem, one needs to understand an SSL Certificate Chain. When traversing a chain, each certificate is an individual certificate. How they are linked is by their issuer. There is a subject section about the certificate itself and then an issuer section which indicates the certificate authority certificate that issued that. It goes all the way up until you hit a root certificate. The root certificates sign themselves.

Here you can see the cert chain for this site. The SAN or alternate names that allow tools.woohoosvcs.com be valid are further down below this screenshot.

SSL Certificate Chain
SSL Certificate Chain

Recursion

It had been a few decades since I sat in a Computer Science class but I do remember a few control structures. One of which was recursion. Recursion can be a difficult concept to grasp and an even more difficult concept to implement. Golang has an example of this here. At a high level a recursive function is one that calls itself. The call stack gets nested in each layer. In the case that you do not know how many certificates and iterations you need to go through to find the chain, recursive functions help with this. Alternatively you would have to use multiple layers of “for” loops to iterate through, matching the current cert against one that may be a child. In a past life I may have implemented a few layers of “for” loops to statically account for the most common scenarios.

Recursion can be tricky and if you do not use it right, it can involve stack overflows if the function indefinitely calls it self. The key to a good recursive function is a way out. It needs to be able to exit without continuing to call itself forever. There are also limits based on the platform as to how deep you can go in this recursion. Your platform will have a limit at which point an exception or error is thrown.

For a much better description of recursion you can read the Wikipedia article here – https://en.wikipedia.org/wiki/Recursion_(computer_science)

The Code

It is still a work in progress but after iterations of playing with linked lists, multiple for loops, this is what I landed on.

func findChildOfCert(certs []x509.Certificate, cert x509.Certificate) x509.Certificate {
	if len(certs) <= 1 {
		return cert
	}
	result := cert
	for _, item := range certs {
		if item.Issuer.CommonName == cert.Subject.CommonName {
			return findChildOfCert(certs, item)
		}
	}

	return result
}

It is kicked off in a parent function via

childCert := findChildOfCert(cs, cs[0])

Where cs is a slice (Golang speak for array) of certificates much like “certs” in the recursive function. We pass it the list of certificates and the very first one.

On the first call it checks to see if this certificate issued any others in the list. If so, it calls the function again with the issued certificate( more child than the current ) and does the process over again.

When it cannot find a certificate issued by the currently iterated certificate (most child record), the for loop exits and it simply passes the original cert that the function was called with. At that point, the stack completely unwinds, passing the “answer” certificate all the way down. That result is provided to the childCert variable.

Validating the Cert

Golang provides a few options for validating the cert. Once you have the most child certificate, you can do something like the below.

for i := range cs {
	if !cs[i].Equal(&childCert) {
		roots.AddCert(&cs[i])
	}
}

opts := x509.VerifyOptions{
	Intermediates: roots,
}

opts2 := x509.VerifyOptions{
	Roots: roots,
}

if _, err := childCert.Verify(opts); err != nil {
	status = append(status, "Not Trusted By Root - failed to verify certificate: "+err.Error())
} else {
	status = append(status, "Trusted Chain")
}

if _, err := childCert.Verify(opts2); err != nil {
	status = append(status, "Not Valid(contiguous) - failed to verify certificate: "+err.Error())
} else {
	status = append(status, "Valid Chain")
}

I load up a “roots” slice of the roots provided. I also exclude the child certificate from this. From there I perform two validations. One that the chain is trusted, meaning it rolls up to one that is trusted by the source used. The other validation is that the chain is valid. Is there continuity in the chain or is it broken. A chain can be valid but un trusted. Knowing the difference may help you in a rare case.

Stack Overflow

I actually found a stack overflow doing a regression test with a self signed certificate. The code above actually ended up comparing the certificate to itself over and over again and trying to keep going down the rabit hole. It ended up with the following

runtime: goroutine stack exceeds 1000000000-byte limit
fatal error: stack overflow

runtime stack:
runtime.throw(0x1332651, 0xe)
	/usr/local/go/src/runtime/panic.go:774 +0x72
runtime.newstack()
	/usr/local/go/src/runtime/stack.go:1046 +0x6e9
runtime.morestack()
	/usr/local/go/src/runtime/asm_amd64.s:449 +0x8f

goroutine 24 [running]:
tools.woohoosvcs.com/certdecoder.findChildOfCert(0xc0000d6b00, 0x1, 0x1, 0xc0000b8800, 0x3da, 0x3ea, 0xc0000b8804, 0x2c2, 0x3e6, 0xc0000b89a0, ...)

tools.woohoosvcs.com/certdecoder/certdecoder.go:180 +0x1d0 fp=0xc020114ef8 sp=0xc020114ef0 pc=0x128a210
tools.woohoosvcs.com/certdecoder.findChildOfCert(0xc0000d6b00, 0x1, 0x1, 0xc0000b8800, 0x3da, 0x3ea, 0xc0000b8804, 0x2c2, 0x3e6, 0xc0000b89a0, ...)

tools.woohoosvcs.com/certdecoder/certdecoder.go:184 +0x1a3 fp=0xc020116920 sp=0xc020114ef8 pc=0x128a1e3
tools.woohoosvcs.com/certdecoder.findChildOfCert(0xc0000d6b00, 0x1, 0x1, 0xc0000b8800, 0x3da, 0x3ea, 0xc0000b8804, 0x2c2, 0x3e6, 0xc0000b89a0, ...)

tools.woohoosvcs.com/certdecoder/certdecoder.go:184 +0x1a3 fp=0xc020118348 sp=0xc020116920 pc=0x128a1e3

Luckily my unit testing caught this and this would have never gone to production due to that unit testing. If you’re not sure what unit testing is check out my article Unit Testing Golang on App Engine. The fix was simple though in that to make sure my recursive function doesn’t compare a cert to itself using the .IsEquals() function

Final Words

In this we walked through a useful use case for recursion and the background of the technology that needed it. I even provided some of the errors that happen when you fail to use recursion properly as I accidentally did!

Unit Testing Golang on App Engine

Summary

If you have landed here, it is most likely because you attempted to implement the subject and ran into errors. In my use case, I ran into errors when splitting up my Golang app into packages and added unit testing to it. To compound it, I was automatically deploying via triggers with CloudBuild.

This builds upon the article IP Subnet Calculator on Google App Engine. Reading that article will help give you a good reference point for this article.

This article is not a “How To” on actual unit testing but getting it to work under this unique combination of tools. A great unit testing article I found is here – https://blog.alexellis.io/golang-writing-unit-tests/

How To Setup

Before we talk about what might have gone wrong, let’s talk about how to set this up properly. There has been a lot of confusion since Golang went to version 1.11 and started supporting modules. After supporting modules it appears using GOPATH is a less of a supported method. With that said some tools still look for it.

Directory Structure

To get a final state visual of my directory structure, it ended up as follows

tools.woohoosvcs.com
tools.woohoosvcs.com/app.yaml
tools.woohoosvcs.com/go.mod
tools.woohoosvcs.com/subnetcalculator
tools.woohoosvcs.com/subnetcalculator/subnetcalculator.go
tools.woohoosvcs.com/subnetcalculator/subnetcalculator_test.go
tools.woohoosvcs.com/cloudbuild.yaml
tools.woohoosvcs.com/html
tools.woohoosvcs.com/html/root.html
tools.woohoosvcs.com/html/view.html
tools.woohoosvcs.com/README.md
tools.woohoosvcs.com/static
tools.woohoosvcs.com/static/submitdata.js
tools.woohoosvcs.com/static/subnetcalculator.css
tools.woohoosvcs.com/root
tools.woohoosvcs.com/root/root.go
tools.woohoosvcs.com/main.go

Modules

Modules help with the dependency needs of Golang applications. Previously, private packages were difficult to manage as well as specific versioned public packages. Modules helps many of these issues. Here is a good article on modules – https://blog.golang.org/using-go-modules

The key to my issues seemed to be involving modules. Google App Engine’s migrating to 1.11 guide recommends the following.

The preferred method is to manually move all http.HandleFunc() calls from your packages to your main()function in your main package.

Alternatively, import your application’s packages into your main package, ensuring each init() function that contains calls to http.HandleFunc() gets run on startup.

https://cloud.google.com/appengine/docs/standard/go111/go-differences

My app’s directory is “tools.woohoosvcs.com” so I named the root, the same. Run “go mod init”

$ go mod init tools.woohoosvcs.com
go: creating new go.mod: module tools.woohoosvcs.com
[email protected] tools.woohoosvcs.com % cat go.mod
module tools.woohoosvcs.com

go 1.13

Importing Private Packages

This then lets us to refer to packages and modules under it via something similar.

import ( "tools.woohoosvcs.com/subnetcalculator" )

CloudBuild

My cloudbuild.yaml ended up as follows

steps:
  - id: subnetcalculator_test
    name: "gcr.io/cloud-builders/go"
    args: ["test","tools.woohoosvcs.com/subnetcalculator"]
    #We use modules but this docker wants GOPATH set and they are not compatible.
    env: ["GOPATH=/fakepath"]
  - name: "gcr.io/cloud-builders/gcloud"
    args: ["app", "deploy"]

Interesting tidbit is the “name” is a docker image from Google Cloud Repository or gcr.io

The first id runs the “go” image and runs “go test tools.woohoosvcs.com/subnetcalculator”. It seems the go image wants GOPATH but go test fails with it set so I had to set it to something fake.

It then uses the gcloud deployer which consumes app.yaml

How I Got Here

Unit tests drove me to wanting to split out logic, particularly the subnetcalculator (check it out via https://tools.woohoosvcs.com/subnetcalculator )

Before implementing modules I could get it to run locally by importing ./subnetcalculator but then “gcloud app deploy” would fail.

2019/11/22 01:39:13 Failed to build app: Your app is not on your GOPATH, please move it there and try again.
building app with command '[go build -o /tmp/staging/usr/local/bin/start ./...]', env '[PATH=/go/bin:/usr/local/go/bin:/builder/google-cloud-sdk/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ace17fcba136 HOME=/builder/home BUILDER_OUTPUT=/builder/outputs DEBIAN_FRONTEND=noninteractive GOROOT=/usr/local/go/ GOPATH=/go GOPATH=/tmp/staging/srv/gopath]': err=exit status 1, out=go build: cannot write multiple packages to non-directory /tmp/staging/usr/local/bin/start.

The error was vague but I noticed it was related to the import path. I tried moving the folder to $GOPATH/src and it seemed to deploy via “gcloud app deploy” but then failed via CloudBuild automated trigger.

------------------------------------ STDERR ------------------------------------
2019/11/21 21:31:39 staging for go1.13
2019/11/21 21:31:39 GO111MODULE=auto, but no go.mod found, so building with dependencies from GOPATH
2019/11/21 21:31:39 Staging second-gen Standard app (GOPATH mode): failed analyzing /workspace: cannot find package "tools.woohoosvcs.com/subnetcalculator" in any of:
	($GOROOT not set)
	/builder/home/go/src/tools.woohoosvcs.com/subnetcalculator (from $GOPATH)
GOPATH: /builder/home/go
--------------------------------------------------------------------------------

ERROR
ERROR: build step 0 "gcr.io/cloud-builders/gcloud" failed: exit status 1

It was like a balancing act with a 3 legged chair! Once I initialized the module though and adjusted the imports it worked great

Final Worlds

Here we worked through a best practice of using modules and an internal package to do automated build deploy on Google App Engine. Unit testing with automated deploys are important so that broken builds do not get pushed to production.

IP Subnet Calculator on Google App Engine

Summary

Over the past few days I put a few hours towards writing a web app to perform some basic CIDR subnet calculations. I wanted to both share the link to it ( https://tools.woohoosvcs.com/subnetcalculator/ ). I also wanted to walk through how I deployed it to Google App Engine.

What is a Subnet Calculator?

When provisioning subnets you typically need to determine your requirements. Determining the size of the subnet based on the number of hosts or what the mask will be can take a little bit of time. This is particularly so when you do not calculate this often. Subnet calculators help save some time on this.

Why did I do it?

I have written a few over the years. My early ones were written in C. Golang has a few libraries though in the new package to help with this. I had to use one the other day and figured why not quite my own. While doing this, why not share how to deploy your own app?

Deploying Golang on Google App Engine

In this, we assume you already have the Google SDK. We walked through this in Hello World From Google App Engine Via PHP.

As with any App Engine app, we need an app.yaml. Mine looks like this

runtime: go112

service: tools

handlers:
 - url: /view\.html
   static_files: view.html
   upload: view.html

I have a main.go file and a static view.html file. Since I already have a default service I am calling this one service. That requires I use a dispatch for my custom domain name.

Don’t forget to add the custom domain to the Settings section!

dispatch:
- url: "tools.woohoosvcs.com/*"
  service: tools

The easiest way to get a Golang app running is as follow. The Google App Engine will run the main function in the main package. Google App Engine sets the PORT environment variable.

package main

func main() {
        port := os.Getenv("PORT")
        if port == "" {
                port = "8080"
                log.Printf("Defaulting to port %s", port)
        }

        http.HandleFunc("/subnetcalculator/", viewSubnetCalculator)

        log.Printf("Listening on port %s", port)
        if err := http.ListenAndServe(":"+port, nil); err != nil {
                log.Fatal(err)
        }
}

// This is the main page to view the Subnet Calculator
func viewSubnetCalculator(w http.ResponseWriter, r *http.Request) {
        http.ServeFile(w, r, "./view.html")
}

The best practice for this would dictate a separate main.go importing your own package but if you were familiar enough with that, you may not need this article.

It is out of scope for this article but “http.HandleFunc(“/subnetcalculator/”, viewSubnetCalculator)” tells Golang that the “/subnetcalculator/” URI be directed to the “viewSubnetCalculator” function. That function simply displays the view.html file

From there we simply run the following. I have tools.woohoosvcs.com under the same parent as dispatch.yaml

cd tools.woohoosvcs.com 
gcloud app deploy

cd ..
gcloud app deploy dispatch/dispatch.yaml

Securing Google App Engine

If you use a Web Application Firewall like Cloudflare, don’t forget to ACL GAE to only allow connections from it. Otherwise, it can be completely bypassed

Google App Engine - Firewall
Google App Engine – Firewall

Final Words

This was just a quick and dirty deploy of a golang app to Google App Engine. As with all the apps though, I run them through Cloudflare or at least some sort of Web Application Firewall.