Three tales of browser-based test suites [Part 1]

The topic of browser-based testing seems to come up again and again and on many occasions I meet people who have expectations that don't match my experience. So I guess it only makes sense to write down these experiences and some of the conclusions I came to.

I'm not even going to try and go into distinctions between acceptance, functional, integration or whatever testing. Whatever you're calling those tests, if they are instrumenting a web browser, that's what I'm talking about. (If you care about the distinctions, or about testing in general, I recommend reading Gojko Adzic's excellent book Specification By Example.)

I want to look at three concrete examples that I have worked on over the past years.

In the first case, nobody on our team had prior experience, except for a short but traumatic brush with IBM's Rational Functional Tester (remember kids: friends don't let friends use IBM products). To get rid of that, we decided to go ahead and try Selenium.

We initially recorded tests with the Firefox plugin but quickly realized that this wasn't maintainable and didn't really offer us good control for selecting elements. Our Tester Benjamin then took it upon him to write tests in Java and started to integrate them into our CI environment. Over time he created a solid set of Page Objects that allowed us to write new tests fairly quickly.

It took the rest of the team quite a while to take some ownership of these tests but eventually we came to a point where we'd sometimes even drive new features from these Selenium Tests.

There were hurdles along the way. Tests were brittle and the execution time was unacceptable (>30 mins). This improved with newer versions of Selenium, use of Selenium Grid to parallelize tests and by keeping a close eye on the VMs running the browser instances. In the case of IE(<8 gave="gave" just="just" p="p" simply="simply" up.="up." we="we">
Due to some relatively complex processes it was also fairly hard to test later stages of these processes. We only slowly got better at setting up test data for this without making use of the browser. This left us with some unfortunate holes in automated coverage.

There were still manual tests for each release for all of our major supported browsers. But with more of the core functionality being handled by automated tests and with the amount of changes shrinking due to shorter release cycles the effort for this went down considerably.

We also became better at writing unit and integration tests and our confidence in these grew.

What did I take away from it:

• the demand for these tests was driven by Benjamin wanting to automate his tests to make his life easier. And also his desire to learn more about Java. It was a relatively clear goal and the benefits of effort spent on automation were fairly clear. Where automation was hard, it allowed him to balance that effort with the effort of doing the tests manually
• setting up and maintaining browser instances for tests and keeping them stable is hard, unthankful work and I'm glad that these days you can just hand money over to saucelabs to do that for you
• getting the CI builds to be reliably green made shared ownership easier. Collaboration on these tests continued to grow.
• closely related to that, the tests have to run fast. If they run for more than ten minutes they might as well not exist
• we had relatively few Selenium tests compared to our unit-tests but there were the odd cases of bugs that weren't caught by our normal test-suites
• when we rewrote a lot of the front-end JavaScript and test-drove those with jasmine, browser compatibility issues became very rare
• as our understanding of all the different tests grew, it became easier to decide when we could avoid writing an extra Selenium test
• working closely with the rest of the team and writing the tests in Java gave our Tester room to grow and learn and turn into a regular dev on the team. Seeing that happen was probably one of the most fun things during my time there. The rest of the team also took up some of the manual testing work.

(Edit 13.9.15: removed some point about testers that I didn't like anymore)

Continue on to part 2

Timetravelling in JavaScript unit tests

At the February berlinjs meet-up Krzysztof Szafranek had a very nice, short talk about TDD in JavaScript (slides here). Among many other things he was showing busterjs and the way it lets you test asynchronous stuff without having to actually wait for time to pass.

I made a note of it to give it a closer look. When I recently had to write some JavaScript again, I remembered about it and was keen to give it a try. Since we were already using jasmine as our testing framework, my colleague Christoph and I were looking for something equivalent there. To cut the intro short, there is: jasmine.Clock.

Thank you, 2011

It’s December. That means it’s time to write a long, rambling post looking back on the year. It’s been a very exciting and eventful year for me, so I’m actually appreciating having the time to do this.

Last new year's eve I tweeted the following. And I wasn't wrong.

2011 is gonna be amazing, you just watch!

I’ve changed jobs, moved from Düsseldorf to Hamburg, took part in a lot of conferences and events, organised a few things myself and in all of that met a huge amount of amazing people that truly inspired me. Thank you, everyone! It was an amazing year.

xtc Berlin

Late Tuesday evening, the night before the ALE conference started, I sat in the hotel bar of the venue talking to people who had arrived early. Looking around, I thought, "cool, it's like an extreme tuesday club in Berlin". Since then I've been thinking about what it would take to actually create something like this in Germany.

So what is the extreme tuesday club?

A regular London meeting (weekly) for Agile/XP/Kanban/Scrum newbies, practitioners, and experts. (*)

That's not very elaborate but I like that it casts a fairly wide net in terms of target audience.

There are already a lot of different software and agile related user groups in Berlin and I like what I've seen of them so far. But each of these only cater to a specific topic or technolgy.

I'm looking for something less formal. Have a beer, talk to people about what they're working on, share war stories, cross-pollinate new ideas, ask for help or give help, if wanted. I would hope it would be as inclusive as possible. So, newbie or expert, programmers, PM/POs, design people, start-uppers, whatever - as long as you care about what you're doing and want to talk with people about it, it should be fine.

I did a little open space session at ALE on this and there seemed some general interest and I said I would just try and get this rolling and just see what happens.

Someone suggested the Cafe 100 Wasser as a pub to go to. And that's where I'll be, this Tuesday from around 8pm onwards. I'd be happy for you to join me. If you do plan to come, please let people know on Twitter.
[we've since changed pubs around quite a bit. Twitter is probably the safest bet to find out where people are. We're now usually at the Prater Garten in Prenzlauer Berg]

Some closure (ha!) to 7 Languages in 7 Weeks

I'm currently trying to work through a backlog of things I've been meaning to blog about. And at the top is this post. The whole experience deserves being wrapped up properly. But I'll keep it brief.

When last I wrote about it, I was somewhere in the middle of week 6, working with clojure. This is probably my favourite of the languages in terms of learning new things and offering new perspectives while also having enough momentum that I might actually be able to do something with it at some point. I really hope this doesn't fall victim to my inherent lazyness.

The last week dealt with Haskell. At this point I was so busy with moving apartments and starting my new job that I barely managed to read the chapter but didn't get to try the examples. It sounded interesting but maybe a bit too strict. Others in the study group seemed to like it a lot though.

So, what's left to say. I really appreciated the study group and being able to learn from other people's code on github.

I enjoyed the weekly google hangout sessions with the others, when I was able to join them. I particulary liked how aimee facilitated those, keeping track of threads of discussions and making sure everybody got to say something.

The book requires quite a bit of commitment to get the most out of it but where I did that I felt it was well worth it.

I certainly recommend doing this together with other people. And share your experience with the world. 

All in all, I stick to what I said in my first post on this.We live in amazing times!

Learning clojure just in time

It looks like we've moved to clojure at just the right moment.

As mentioned, I started using emacs for the first time again in ages. While I still struggle remembering all the useful key combos I did get a little bit of help in the transition by installing aquamacs. It keeps buffers in tabs and lets you use cmd-c,x,v while you're trying to remember how to work the yank buffer.

I'm really liking clojure so far. I'm finally writing tests again (not sure I actually ever did for any of the other languages) and it's nice to have that inside emacs and also have the REPL to try things out.

So, Day 1 only had two excercises. I'm going to just point to aimee's entry on that, because we both came up with pretty similar code.

But let's look at the second exercise

Write a function called (collection-type col) that returns :list, :map, or :vector based on the type of collection col.

I got the following:

I wasn't quite happy with repeating the col argument for each condition. So I started to play around with using the various test methods (list? vector? map?) as keys in a map and then filtering the keys based on the collection argument given. I eventually gave up on it, because it is a rather silly approach. But it was an interesting opportunity to play around with the language.


Then I noticed multi-methods and became interested. Method dispatch based on the result of a function applied to the arguments of the method. This is pretty powerful and will probably take a while to get comfortable with. I used the rather simplistic approach of dispatching on the class to do the second exercise again.


This is silly, too. And it works pretty sloppily. The empty list creates an instance of PersistentList$EmptyList which doesn't seem to match PersistenList. But whatever, it's good enough and it got me thinking about multi-methods. There is more on those in this blog post.

And now it's on to day 2, where apparently we see the list comprehensions I was so impressed with in Erlang enter clojure, too.

Excuses, Excuses. Also, Emacs.

My, I've been a bit lazy, I guess. Has it really been that long?

Starting a new job and trying to find a new place to live has taken quite a bit of my time. So it was that during the Prolog week I started to struggle finding the time to do the examples. Then during the Scala week I didn't do much at all other than reading the chapter. But with Erlang I had some more time again and at least managed to do the day 1 and 2 stuff. With Clojure this week I hope to give it even more attention.

For all of it, I neglected writing about it. So to get this rolling again I'll do a quick rundown.

Prolog

Prolog is old and weird. The book compares it to Rain Man, which seems apt. Having never done any work with any logic languages this was very new to me. It was a very big shift in thinking. I appreciated that. It's one of the reasons to work through the book after all.

I think Prolog as a language is not very practical for me. But the chapter provided some really useful basics for understanding the syntax of Erlang and for pattern matching, which shows up in the Scala and Clojure chapters, too.

Scala

Scala was, as mentioned, somewhat neglected by me. Some of the syntax in the examples (especially for the actor stuff) seemed a bit weird to me and I really should actually work through them to get a feel for it. Minor inconsistencies in syntax are probably the main thing bugging me about what I read about Scala so far. And defining your default constructor inside the class definition while still having the option for actual init methods really rubbed me the wrong way.

Still, XML as a first class language construct looks very nice and getting around some of Java's noise is also charming. I reckon we'll cross paths again at some point and I can give it a closer look.

Erlang

Next came Erlang. And that was quite a bit of fun. Having been eased into the syntax by the Prolog chapter helped a great deal. I was quite fascinated by list comprehensions. I'm not sure if their power comes at the cost of maintainability. But it's hard for me to judge that given my lack of experience with the language and syntax.

I didn't even get to play with the actor stuff but I suppose people are happy with it and specifically with the Erlang implementation.

At some point I should really revisit Erlang. It's different enough to challenge the mind to take a new look at old habits. And it looks interesting in general with its focus on concurreny.

Clojure

I'm trying to dig a little deeper this week again. Clojure is interesting to me because for a long time I've wanted to try and make sense of a Lisp language.

In the process of reading up on it I noticed that a lot of people seem to prefer using emacs as an editor for clojure. So I installed that. (As a complete Mac noob let me point out that brew install emacs and brew install emacs --cocoa are two very different things. I have yet to learn why [cf. mac noob] but you probably want the latter.)

I had been using emacs (XEmacs, to be honest) for quite a while as my primary development environment. But that was some 7+ years ago and I never really touched it again. I'm incredibly amused that a lot of the key bindings are coming back to me anyway. And it feels kinda nice. Maybe I will rejoin the battle for the one true editor.

Sidenote: emacs installation notes for anyone who bothers. From someone who has no clue what he's doing:

• brew install leiningen clojure clojure-contrib 
• brew install emacs --cocoa
• git clone https://github.com/technomancy/emacs-starter-kit.git ~/.emacs.d
• alias emacs='/usr/local/Cellar/emacs/23.3/Emacs.app/Contents/MacOS/Emacs -nw'
• lein new someproject
• add :dev-dependencies [[swank-clojure "1.2.1"]] to the (def-project entry in someproject/project.clj
• lein deps
• lein swank
• start emacs in another shell
• (you may or may not need to install some elpa packages for slime and swank and paredit and whatever. See if it works with the just the stuff from the emacs-starter-kit or otherwise ask google. I did get annoying errors doing this and have no idea why it worked in the end. Sorry.)
• M-x slime-connect
• edit e.g. someproject/test/someproject/core.clj
• execute tests with C-c C-,
• you can also do other things with the REPL inside emacs but you best google for that

I hope that can be of help.

Back to the study group...

It seems like most of us have been struggling keeping up with the pace of the book, with everyone taking the odd week off from working through all the examples. But we were lucky that it seems to have mostly been different weeks, so there was always at least one person who could explain more details about a given language. So while I was very lazy there have been a lot of really interesting posts from others in the group.

• aimee on Prolog
• and again on Scala (fuck the pope!)
• Alberto on Erlang (check out the Solution = list comprehension. It's quite impressive)
• Susan with an Erlang follow-up
• More posts here

We've also continued our weekly chats which are always a source of inspiration for me. As of recently we're using the Hangout feature of Google+ and it's really good. Since Skype keeps crashing my little Win7 netbook I'm happy to have found a better replacement.

Tl;dr

It's really nice to see how the chapters of the book build up on each other. I think the book has made me more adventurous to just play around in new languages. It is still nice to have this little virtual study group to help push me along. Technology is still awesome, too (git pushing from a train via tethering to the phone. Magic!) I reinstalled emacs after all those years.