An example of how I debug

Last week I had a difficult bug to track down where an Angular frontend was talking to a Node.js backend (made up of multiple microservices) through a load balancer. The browser appeared to be sending 2 POST requests when it should have been sending one.

The first thing I tried to do was shorten the feedback loop. The problem culminated after a long workflow in the browser with a single HTTP POST to the server. I opened Chrome DevTools and converted the request to a cURL command line. Strangely, cURL didn’t show the same behavior as Chrome. Running through the workflow a few times in Firefox also succeeded without error. I used tcpflow to examine the requests but there were no material differences.

Because this error only manifested itself after a substantial upgrade of the backend services I reasoned I could run the frontend locally. I did this, removed all the guards from the Angular frontend and pointed it the test server. This allowed me to navigate directly to the endpoint in the workflow so I could reproduce the request just by clicking a button.

The next thing I wanted to look at was what was happening on the server. I enabled logging on the load balancer and what I saw was unexpected. One request received by the load balancer was being turned into two requests sent to the internal application. Again, this was not behavior that could be reproduced on cURL or Firefox. After adding additional logging to the application, I turned on network logging for Node.js. This led me to the real issue.

I noticed something different between the Firefox and Chrome requests, namely an additional socket timeout when dealing with connections from Firefox. This led me to think that perhaps because of the upgrade, which included bumping Node.js to the next major version (8.0.0), that it could be a platform issue. Indeed, when I downgraded Node.js on the server to 7.10.0, the problem vanished.

The final technique that I want to share is something that I learned from the git bisect command. It uses binary search to narrow problems down to a specific commit. After upgrading Node.js to 8.4.0 and seeing the problem had been fixed, I backtracked through the versions using binary search until I found the version where this issue had been fixed, which was 8.1.1. Reading through the changelog, I ultimately found the bug (#13391) which summarized my problem exactly.

It seems the convergence of keep-alive differences between browsers, the time taken for a request to complete and CORS all contributed to a bug that caused me to want to tear my hair out. At least I got a blog post out of it.

In summary, a list of tools and techniques I used for debugging this problem were:

  • Shorten feedback loops e.g.
    • Reduce the issue to the smallest amount of services or piece of code that still exhibits the problem
    • use cURL to replay problematic network requests
    • use Selenium or other headless browser frameworks to replicate browser issues
  • Use network sniffers like tcpflow or Wireshark to understand exactly what is being sent over the network
  • Ensure an appropriate amount of log statements in code and enable platform logs when required.
  • Binary search (e.g. git bisect)
  • Comments Off on An example of how I debug

2 years in: C# + .NET vs Ruby on Rails

I’m an ex-C# guy (7 years) doing Ruby (2 years) with a little bit of Java thrown in. My company is currently finding it tough hiring Ruby devs (no different from when I was hiring C# devs) and subsequently we’re casting the net wide enough to include C#/Java/etc. guys that want to change to Ruby. We have a standardized technical test that asks the candidate to create a web application in 1.5 hours in the language of their choice.

Coming from a C# background, I’m continuously shocked at how poorly the C#/Java guys fair on the test. Most of the Ruby guys use Rails and a reasonable percentage nail the test but none of the C#/Java guys have even come close to about 10% of the full test. I don’t think it’s because they are necessarily bad coders. Some of the C# code reminds me a lot of what I used to write with IoC containers and design patterns galore. The point is that I think the language and platform ecosystem is so much better that it gets out of the way and let’s you solve the business problem.

Is the test biased to quickstart projects? Possibly. How often do you need to click ‘New Project’ in Visual Studio, anyway? After thinking about it, it just doesn’t seem that way. There is just so much overhead in doing things ‘the right way’ in C#/Java. Tests shouldn’t go to the database. You need interfaces everywhere to inject your repositories into services or your services into controllers. This is not a one off thing. You’ll have to do this everyday to keep your code base clean.

From a developer productivity point of view, it just seems like you can get further, faster with Ruby/Rails then with C#/Java.

  • Comments Off on 2 years in: C# + .NET vs Ruby on Rails

Sinatra or Rails

I was recently asked by my boss that if I had to choose a Ruby web framework, which one would I choose. This is what I found.

I have only written toy examples in Rails (along time ago) and the same goes for Sinatra (except I’ve looked at it more recently). However, I’m quite familiar with two frameworks in the Python world called Django (which is roughly equivalent to Rails) and Bottle (which is inspired by Sinatra).

Short answer: Really an unfair comparison. Rails is a big framework that provides a lot of features while Sinatra is stripped down and aims at handling HTTP request/responses. Rails use case is for a big website or one that is intended to become big. Sinatra’s sweet spot is small websites or a simple apis. When it comes to choosing a complete website framework, a more equal comparison would be between Rails, Padrino or Ramaze.

Longer answer:
For me the most compelling distinction between the two frameworks is the amount of code each has. If I check out the github repo for Sinatra and Rails and run cloc over them, I get the following:

Project Files Blank Comment Code
Sinatra 53 1516 358 8268
Rails 1745 32740 33770 152274

With Sinatra having about 5% of the code of Rails, it should be clear that there’s no way it could hope to be as featureful as Rails. They are therefore addressing very different goals. According to Sinatra’s maintainer:

“…Rails is a framework focused on writing model driven web applications, Sinatra is a library for dealing with HTTP from the server side.”

This is pretty clear from examples where Rails starts off with scaffolding, introducing directory structure and setting up a database, while Sinatra immediately dives in with routing and an explicit GET request.

Because Rails is opinionated software, a lot of design decisions about how to construct a web application has been made for you. In contrast, Sinatra is just a wrapper around Rack and the request/response cycle. All those design decisions e.g. ORM, Ajax, REST, how to handle DB migrations still have to be made when developing with Sinatra. However this can be to be Sinatra’s advantage. Because there is so little code and ‘magic’, this does mean that Sinatra is really fast. Sinatra can be up to 4 times faster than Rails.

What’s more, you don’t need feel like you’ve made the wrong choice and feel hamstrung by Rails if it gets too complicated. In Rails 3.1, you can embed Sinatra within your Rails application.

Ultimately, I think the comparison is probably unfair, given the different goals of the two particular frameworks. I think in terms of full-featured web frameworks, a more equal comparison can be made between Rails, Padrino or Ramaze. These frameworks intend to achieve the same goals but with differing philosophies. The interesting thing about Padrino is that it is built on top of Sinatra. If you don’t like Rails and Sinatra is too stripped down, Padrino represents a nice middle ground in having a lot of the features you need in a web framework but also being as performant as Sinatra.

  • Comments Off on Sinatra or Rails

When you want to deliver a file to a user with SpringMVC you can use the built in FileCopyUtils to put the file into the OutputStream of the response.

This is an example of the method in your controller class.

    @RequestMapping(value = "/yourURLHere")
    public void handleFileDownload(HttpServletResponse response) {
        File file = myFileService.getFile();

        response.setContentType("application/xls"); //in my example this was an xls file
        response.setContentLength(new Long(file.length()).intValue());
        response.setHeader("Content-Disposition","attachment; filename=MyFile.csv");

        try {
            FileCopyUtils.copy(new FileInputStream(file), response.getOutputStream());
        } catch (IOException e) {

CSV to XML with Powershell

My brother had the need to convert a bunch of SNMP data into XML for Zabbix. He was trying to do this in Excel but fortunately, he asked me if there was a quicker way to do this rather then struggle with crazy string concatenation formulas. I thought this should be trivial in Powershell and it seems I was right:

Here’s an example of what the input CSV “might” look like (input.csv)

type key delay
4 13.1001 600
5 13 700

And here’s the powershell file (convert.ps1):


function zabbix_template($csv) { 
    write "					

write ""
Import-Csv $csv_file | ForEach-Object { zabbix_template $_ }
write ""

This is obviously not complete but it’s not far from a working solution. To execute this from a windows command line you would do the following:

c:\>powershell .\convert.ps1 input.csv > zabbix.xml

The two main sources of magic are the Import-Csv cmdlet and string interpolation (No need to use string.format for this). The great thing about the Import-Csv cmdlet is that it creates an object for each row using the header of the csv file. Thus the ‘delay’ column in the csv file becomes a ‘delay’ property on an object for each line. This is brilliant because it means I can pass that object to the function that performs the string interpolation and just reference the columns.

I think this is the day I fell in love with Powershell.

  • Comments Off on CSV to XML with Powershell
« Previous Entries  Next Page »