An issue I had with AWS CodeDeploy

 Hi there

Clearly, I've not really written anything here in 7 years. Let's change that. By writing about an issue we had at Qixxit about 3 years ago. It's pretty specific so really not exciting but I've published a bit of code for it and I wanted to finally do a bit of a write-up on it.

The problem

We ran qixxit's backend on AWS as an auto-scaling group behind an application load-balancer. The auto-scaling group would manage ec2 instances running the application. The auto-scaling group would know about the linked load balancer through a property "target-group" in its configuration. This allows the auto-scaling group to tell the load balancer to stop sending traffic to an instance before removing it. 

We used AWS CodeDeploy's blue/green deployment feature to roll out new versions. This would create a new configuration of the auto-scaling group based on a newly built AMI. It would then bring up new instances for that and redirect traffic from the load balancer to these. Eventually, the old configuration is removed.

The new configuration for the auto-scaling group has all the settings of the previous configuration copied over automatically by CodeDeploy. Except for that "target-group" setting.

Now when the auto-scaling group decides that it wants to scale in and remove an instance it will just immediately shut down that instance. (Sidenote: maybe it's because English isn't my native language but I really don't get why it's scale in/scale out and not scale down/scale up. Is this maybe specific to Amazon's documentation?) Anyway, the instance is gone but the load balancer still sends traffic to it.

I don't actually remember what specific problems this caused and whether this was also an issue when scaling out. It wasn't too dramatic and initially we worked around it by just setting the target group manually and filed a support request.

Amazon's support was helpful and acknowledged the issue. But they didn't provide a way to directly track whether this was being worked on (I guess they generally don't?) and from the communication it sounded like it was pretty unlikely anyone would ever actually work on it.

A solution

And so I wrote a bit of Javascript to solve it for us. CodeDeploy sends notifications for the various steps of the deployment. That can be hooked into to then set the target group of the auto-scaling group. The whole thing runs as a lambda managed via the serverless framework.

This was a nice way for me to practice my javascript test driving skills and also to play around with the serverless framework. And as of recently also to play around with Github Actions. Though I don't actually know if any of it still works or whether it's even still relevant and maybe the bug got fixed by Amazon. 

Anyway, the repository is here: https://github.com/dmt/code-deploy-set-target-group

In closing

I don't like how this write up turned out. It seems impossible to me to describe this in a generic way that doesn't require some knowledge of a bunch of specific AWS products. But maybe that's a good way to keep expectations low for this blog. 

Three tales of browser-based test suites [Part 2] - test harder

In Part 1 I wrote about my initial experience testing with Selenium. In this part I will write about what I learned about browser-based tests on my first project working for ThoughtWorks. It was a step up in terms of coverage and complexity and that also brought its own set of problems.

Three tales of browser-based test suites [Part 1]

The topic of browser-based testing seems to come up again and again and on many occasions I meet people who have expectations that don't match my experience. So I guess it only makes sense to write down these experiences and some of the conclusions I came to.

I'm not even going to try and go into distinctions between acceptance, functional, integration or whatever testing. Whatever you're calling those tests, if they are instrumenting a web browser, that's what I'm talking about. (If you care about the distinctions, or about testing in general, I recommend reading Gojko Adzic's excellent book Specification By Example.)

I want to look at three concrete examples that I have worked on over the past years.

In the first case, nobody on our team had prior experience, except for a short but traumatic brush with IBM's Rational Functional Tester (remember kids: friends don't let friends use IBM products). To get rid of that, we decided to go ahead and try Selenium.

We initially recorded tests with the Firefox plugin but quickly realized that this wasn't maintainable and didn't really offer us good control for selecting elements. Our Tester Benjamin then took it upon him to write tests in Java and started to integrate them into our CI environment. Over time he created a solid set of Page Objects that allowed us to write new tests fairly quickly.

It took the rest of the team quite a while to take some ownership of these tests but eventually we came to a point where we'd sometimes even drive new features from these Selenium Tests.

There were hurdles along the way. Tests were brittle and the execution time was unacceptable (>30 mins). This improved with newer versions of Selenium, use of Selenium Grid to parallelize tests and by keeping a close eye on the VMs running the browser instances. In the case of IE(<8 gave="gave" just="just" p="p" simply="simply" up.="up." we="we">
Due to some relatively complex processes it was also fairly hard to test later stages of these processes. We only slowly got better at setting up test data for this without making use of the browser. This left us with some unfortunate holes in automated coverage.

There were still manual tests for each release for all of our major supported browsers. But with more of the core functionality being handled by automated tests and with the amount of changes shrinking due to shorter release cycles the effort for this went down considerably.

We also became better at writing unit and integration tests and our confidence in these grew.

What did I take away from it:

• the demand for these tests was driven by Benjamin wanting to automate his tests to make his life easier. And also his desire to learn more about Java. It was a relatively clear goal and the benefits of effort spent on automation were fairly clear. Where automation was hard, it allowed him to balance that effort with the effort of doing the tests manually
• setting up and maintaining browser instances for tests and keeping them stable is hard, unthankful work and I'm glad that these days you can just hand money over to saucelabs to do that for you
• getting the CI builds to be reliably green made shared ownership easier. Collaboration on these tests continued to grow.
• closely related to that, the tests have to run fast. If they run for more than ten minutes they might as well not exist
• we had relatively few Selenium tests compared to our unit-tests but there were the odd cases of bugs that weren't caught by our normal test-suites
• when we rewrote a lot of the front-end JavaScript and test-drove those with jasmine, browser compatibility issues became very rare
• as our understanding of all the different tests grew, it became easier to decide when we could avoid writing an extra Selenium test
• working closely with the rest of the team and writing the tests in Java gave our Tester room to grow and learn and turn into a regular dev on the team. Seeing that happen was probably one of the most fun things during my time there. The rest of the team also took up some of the manual testing work.

(Edit 13.9.15: removed some point about testers that I didn't like anymore)

Continue on to part 2

Timetravelling in JavaScript unit tests

At the February berlinjs meet-up Krzysztof Szafranek had a very nice, short talk about TDD in JavaScript (slides here). Among many other things he was showing busterjs and the way it lets you test asynchronous stuff without having to actually wait for time to pass.

I made a note of it to give it a closer look. When I recently had to write some JavaScript again, I remembered about it and was keen to give it a try. Since we were already using jasmine as our testing framework, my colleague Christoph and I were looking for something equivalent there. To cut the intro short, there is: jasmine.Clock.

Thank you, 2011

It’s December. That means it’s time to write a long, rambling post looking back on the year. It’s been a very exciting and eventful year for me, so I’m actually appreciating having the time to do this.

Last new year's eve I tweeted the following. And I wasn't wrong.

2011 is gonna be amazing, you just watch!

I’ve changed jobs, moved from Düsseldorf to Hamburg, took part in a lot of conferences and events, organised a few things myself and in all of that met a huge amount of amazing people that truly inspired me. Thank you, everyone! It was an amazing year.

Javascript TDD with jasmine and maven

Edit[04/2011]: Since I seem to get the odd google hit on this it is worth pointing out that Justin Searls has since updated the plugin to include a lot more useful configuration options and also made it available in the maven central repo. So you can just use it straight away by following the instructions.

Edit[05/2011]: and now there is also this awesome site where you can try out jasmine and even do it in coffee script - all in your browser. Happy pandas all around!

As the amount of javascript in our project has grown quite considerably, it was time to get rid of this black spot in our test coverage. And since I'm getting more and more used to test-drive the code I'm writing, it was especially annoying to have to regress to trial-and-erring for the javascript part of our code.

I was overwhelmed by the choice of testing frameworks but eventually settled on jasmine, as that seems to be the most widely supported one with a reasonable syntax.

As all testing that is not automated might as well not exist it was important that it was possible to integrate it into our Teamcity setup. Thankfully, seraching for jasmine and maven quickly pointed me to jasmine-maven-plugin.

This plugin builds a html test runner that includes all javascript files in your project and then gets accessed with htmlunit and can run during the maven test phase, requiring no changes to our Teamcity build configuration.

Since we have a bunch of library and legacy javascript code that takes offence to being included directly I needed a way to filter which files actually got included in the test. There was some slightly out-of-date fork on github that included changes to configure inclusion and exclusion patterns which I was able to adapt to the current version (Which is available here).

The nice thing here is that all files are still copied to the target directory and the exclusions only affect what files get loaded in the test runner which proved useful soon enough.

I wasn't really happy just expecting all files to be loaded in the tests since that is not likely how it would happen in the deployed code. I am hoping to have the test specs define what other files are required by a specific functionality. Since all files get copied it is easy enough just loading them from inside the tests. I'm not quite happy with that yet as globals don't get reset between different jasmine Specs but it should be easy enough to add that functionality at some point.

Another nice feature of the maven plugin is the inclusion of other javascript artifacts. This way you can deploy libraries like jquery to your local maven repository and have them automatically included in your test runner.

So, with that said, this is what I currently have in our maven configuration:



Most of the options are explained very well in the main jasmine-maven-plugin README. I've only changed the plugin version to our own locally deployed version and made use of the new include option. This will only include files named *_spec.js in the test runner. The specs will then load additional files themselves.