Okay, okay, we don’t write everything in Go — we have a whole front-end codebase in Typescript/Javascript, we have some Python, we have some straight Bash. Believe it or not, we even had some Tcl up until early 2022 before we ported it, of course, to Go.
At Bitly, we’ve been all in on Go as our preferred back-end language since about 2015.
Why?
We like tools that are efficient and powerful. We like to avoid the unnecessarily complex. We like repeatable patterns, open source, clear standards and high performance. We like easy-to-read code because we like easy-to-maintain code. This is why we like Go.
As you might guess, we power the Bitly ecosystem with a collection of web apps, APIs, microservices and datastores.
The Bitly system processes tens of billions of redirects and hundreds of millions of new links each month. We service customers worldwide and run approximately 15 direct user-facing services (web apps and external APIs), 35 internal APIs, 130 queue consumer processes, 150 cron jobs, 20 datastores and who knows how many ad hoc scripts.
All told, Go accounts for a little over half of our code base.
Once Upon a Time…
There once was a time when we wrote most backend services in Python, and when we wanted serious performance, we dropped into C. We used Python because it was easy to write, reasonably performant, and there was a nice web server framework called Tornado that scaled pretty well with its effective use of non-blocking network I/O.
We used C because, well, it was C and you can’t beat its performance. You can, however, beat its maintainability. (How many C developers do you know out there?)
But what if we could have a simple and straightforward language that gave us high performance as well?
Enter Go
In 2014, we wrote a little open source project called NSQ (nsq.io) and put a promising new language called Go through its paces. We liked what we saw so much that we started writing everything new in Go, and soon thereafter we began porting all legacy services to Go as well.
One of the most compelling findings from that early experience was that we could run the same workload in Go on fewer servers than the Python version and the response times were typically twice as fast. That, in and of itself, would have been enough, but add to the performance gains the fact that developers unfamiliar with Go can be onboarded to it in a matter of weeks, and we were sold.
Performance: Go vs. Python
Serving a simple request
Here’s a simple example of one path served up by a web app. This is simply the serving of a site association file for iOS deep linking for a given custom domain. (If you don’t know what that means, suffice it to say, iOS calls it and it’s a simple json response.)
Just this:
% curl https://m.bitly.is/apple-app-site-association
{
"applinks": {
"apps": [],
"details": [
{
"appID": "2J589H6KTZ.com.bitly.Bitly",
"paths":["NOT /","NOT /*+","NOT /m/*","NOT /.rss/*","NOT /index.php/*","NOT /index/*","NOT /static/*","NOT /etc/*","NOT /media/*","NOT /assets/*","NOT /sites/*","NOT /images/*","/*"]
}
]
}
}
Below you’ll find Python and Go versions of the code to handle this HTTP request.
The two pieces of code are basically doing the same thing:
- Call our internal API to get the site association data for a given custom domain
- Return some json
Here’s the Python/Tornado:
Route definition:
(r"^/apple-app-site-association$", app.site_associations.SiteAssociations, {"os": "ios"}),
Handler:
from lib.deeplinks_api import get_site_associations
class SiteAssociations(app.basic.BaseHandler):
BASE_TMPL = {
"ios": {
"applinks": {
"apps": [],
"details": [],
}
},
}
@tornado.web.asynchronous
def get(self):
callback = functools.partial(self._finish_deeplinks_api_get_sas, os=self.os)
get_site_associations(os=self.os, branded_short_domain=self.host, callback=callback)
def _finish_deeplinks_api_get_sas(self, response, os=None):
if response.error:
statsd.incr("{0}.sas.error.none_found".format(os))
self.set_header("Content-Type", 'application/json')
return self.finish(json.dumps(deepcopy(self.BASE_TMPL[os])))
resp = json.loads(response.body)
data = sorted(resp, key=lambda k: int(k.get('priority', 1)))
return self._generate_response(os=os, data=data)
def _generate_response(self, os=None, data=[]):
response = deepcopy(self.BASE_TMPL[os])
for item in data:
details = {
"appID": item["apple_app_entitlement_id"],
"paths": self.PATHS,
}
response["applinks"]["details"].append(details)
statsd.incr("{0}.sas.success".format(os))
self.set_header("Content-Type", 'application/json')
self.finish(json.dumps(response))
Here’s the Go:
Route definition:
router.GET("/.well-known/apple-app-site-association", wrapper(siteAssociations.Apple)) // wrapper records metrics about the request
Handler:
type SiteAssociationHandler struct {
deeplinksAPI *deeplinksapi.Client
}
type AppleAppLinkDetail struct {
AppID string `json:"appID"`
Paths []string `json:"paths"`
}
func (h *SiteAssociationHandler) Apple(w http.ResponseWriter, r *http.Request) {
host, err := idna.ToASCII(r.Host)
if err != nil {
log.WithContext(r.Context()).Warn("invalid host")
webresponse.BadRequest400(w)
return
}
sa, err := h.deeplinksAPI.GetSiteAssociations(r.Context(), host, "ios")
if err != nil {
switch err.Error() {
case "INVALID_DOMAIN":
webresponse.NotFound404(w)
return
}
webresponse.UnknownError500(r.Context(), w, err)
return
}
sort.Sort(sa)
details := make([]AppleAppLinkDetail, 0, len(sa))
for _, item := range sa {
details = append(details, AppleAppLinkDetail{
AppID: item.AppleAppEntitlementID,
Paths: siteAssociationPaths,
})
}
w.Header().Set("Content-Type", "application/json")
err = json.NewEncoder(w).Encode(AppleSiteAssociation{AppleAppLinks{
Apps: make([]interface{}, 0),
Details: details,
}})
if err != nil {
webresponse.InternalError500(r.Context(), w, err)
return
}
}
Now look at the difference in performance.
Fig 1. Python – Response times in milliseconds. Sample size: 67,179 requests.
Fig 2. Go – Response times in milliseconds. Sample size: 50,192 requests.
That’s almost half the average response time!
A handful of milliseconds improvement may seem like no big deal, but this code does almost nothing. It makes a single API call and marshals some data to return. A 50% improvement in response time at this level increases exponentially as the complexity of the code increases. More complex code is made up of simple building blocks like this.
Need a more dramatic example?
gsutil (Python) to gscopy (Go)
Maybe you use gsutil, like we used to, to copy files to Google Storage. Mundane task, right?
gsutil is a Python-based utility provided by Google (https://cloud.google.com/storage/docs/gsutil). We rewrote the parts of it that we needed in Go and here’s what we found.
The graph below shows the duration of a cron job that runs every 30 minutes. It copies 7,000 to 11,000 files per run. Average file size is 100Mb. (Those of you crunching the numbers will see that the Python version of that cron job didn’t really run every 30 minutes. That version took close to an hour to run.)
Fig 3. Time to copy 7,000 to 11,000 files to Google Storage, Python vs. Go
The Go version is more than a full order of magnitude faster than the Python version. And look at the CPU drop.
Fig 4. Switch from gsutil to gscopy, effect on CPU utilization
Dramatic.
I’m sure you want to see the code. In a nutshell, it uses the Google-provided Go package cloud.google.com/go/storage with a service account. Perhaps we’ll get this into an open source repo near you.
The Impact
The impact of porting to Go should be obvious now.
Our backend services typically take one of only a handful of forms:
- Web apps (bitly.com, internal tools)
- External-facing HTTP APIs (the Bitly API at https://api-ssl.bitly.com/)
- Internal HTTP APIs (shorten API, billing API, user management API, etc. — accessible only to our internal systems)
- Queue readers (async message consumers)
Out of approximately 270 backend services today, about 155 of those are Go. That makes for a performance gain of…divide by 2…carry the 1…A LOT.
But Wait, There’s More!
Beyond the performance benefits of Go, there are several other elements that we’ve come to appreciate about the language.
Community
The Go community is a friendly, welcoming, active and supportive bunch. We love participating in Gophercon and typically have Bitly developers in attendance every year. In addition, at the time of this writing, there are 388 Go Meetups in 247 cities worldwide (https://www.meetup.com/topics/golang/all/).
Documentation, tutorials & tools
Go is not only well documented but documentation is built into the language. Documentation is generated from the inline comments in the source code with godoc (https://pkg.go.dev/golang.org/x/tools/cmd/godoc). The source code is accessible to all, hence when you see the docs saying this:
Fig 5. Go docs. https://pkg.go.dev/builtin#string
… it maps directly to the source code that says this:
Fig 6. Go source code. https://cs.opensource.google/go/go/+/refs/tags/go1.17.6:src/builtin/builtin.go;l=71
The “Tour of Go” tutorial (https://go.dev/tour/welcome/1) provides an excellent introduction to the language and its concepts, making onboarding developers who may have experience in other languages very straightforward.
The Go Playground (https://go.dev/play/) makes it easy to test code snippets without having to set up your entire runtime environment.
Go Standards & Bitly Go Standards
While there are lots of ways to write code in any language, Go provides a guide “for writing clear, idiomatic Go code” at Effective Go (https://go.dev/doc/effective_go).
We also have a set of in-house standards that are laid on top of the Go standards. Go lends itself to clear, consistent patterns in our approach to things like application layout, database access and API design. Some of the topics in our internal Go documentation look like this:
Fig 7. Bitly Internal Go Standards table of contents
We highly value the ability of new developers to quickly become productive on code contributions or code review. Clear patterns from both the language itself and from our internal standards help make the onboarding experience for new developers manageable. With a 12-year old code base, predictability is paramount.
The March Continues…
Fig 8. Thus spake GitHub. Current language breakdown.