Showing posts with label java. Show all posts
Showing posts with label java. Show all posts

Thursday, April 16, 2026

A bit about relational data in Java / Spring

Stepping away from JPA 

I recently found myself back in the Java world.

tl;dr

I belatedly discover spring data relational jdbc and realise it would have been a better fit for a lot of the things I used to use JPA for. 

the issues

If you've worked with JPA/Hibernate you will likely have encountered one or more of the following issues:

lazy initialisation issues / N+1 query performance issues

You defined your model - now when do you load relations and how does that impact performance? The tendency is to defer thinking about this to a later point in time and probably end up with something hackish like open session in view. (This is perfectly fine depending on the given context, as long as you're aware of the little bit of extra risk you take on.)

problems correctly defining cascading to ensure that related entities get created or deleted when modifying the root entity

You try to stick to domain driven design's definition of repositories and using aggregate roots. E.g. if you're adding a line item to and order, you want that to be persisted by saving the order and not by saving each line item individually through a line item repository. This is not inherently difficult, but configuring this correctly can get complex.

unexpected behaviour because of how proxied entity instances handle calls to internal methods

Hibernate/JPA does its magic (tracking changes, lazy loading, etc) by proxying the entities it handles and wrapping access to entity attributes with aspects. This makes it almost impossible to put any higher order logic into the entity code. This is rarely a problem in itself but tends to lead to excessive use of DTOs.

lots of redundant code mapping back and forth from anaemic DTO objects

It seems all too common to have a complete mirror of all JPA Entities in a set of DTO classes. Then this usually gets mapped back and forth by some automated mapper. This is seen as necessary mainly because of the issues already listed here. In the better cases it will look something like described in this post. Ultimately this makes changes to your models harder while providing very little benefit. 

(Compare to Python's FastAPI with pydantic, especially using different model projections to e.g. hide something like a password attribute)

problems testing all of this because of awkward transaction handling in integration tests

All these problems are solvable and don't mean there is no value in JPA. And a lot of the complexities can be designed and evolved by using tests. But a big issue there used to be in how lazy loading and cascading would behave subtly different depending on how transactions behaved in tests versus in production code. Presumably this should be slightly easier to handle with the proliferation of testcontainers and generally decent integration test support in spring. I haven't had a recent look at this.

a fresh look

So you take on these burdens for the benefit of automagic ways of defining your OR model. "ORMs, it's just what you do to work with relational data in Java." And some of that automagic in e.g. JPARepository is quite useful for basic query operations. 

I've never really questioned this because I do remember the pre-hibernate world of working directly with JDBC. But coming back to the Java world after a two-year period working in a different stack, I was curious to try out some of the project reactor features and in the process looked a bit closer at spring data relational. And with that then finally came the question, why am I even doing this (JPA) to myself? 

The difficulties inherent to trying to map arbitrary complexity of your domain model onto a relational database still exists and don't just disappear. I would just make the claim that in a lot of cases, dealing with those difficulties head on ends up producing simpler, more maintainable code than using JPA. There's still a good amount of querying automagic there (e.g. in CrudRepository/PagingAndSortingRepository) if you want it. And basic relationship handling with @MappedCollection also works well. 

(IntelliJ is also quite good at creating DDL sql for @Table annotated entities that can go straight into, e.g. Flyway scripts)

some examples

Taking the following simplistic model...
@Table("orders")
public record Order(
        @Id UUID id,
        UUID customerId,
        @MappedCollection(idColumn = "order_id", keyColumn = "list_key") List<LineItem> items,
        @Version int version
) {
    static Order forCustomer(UUID customerId, List<LineItem> items) {
        return new Order(null, customerId, items, 0);
    }

    public Order withItems(List<LineItem> items) {
        return new Order(id, customerId, items, version);
    }
}
@Table("order_items")
public record LineItem(
        @Id UUID id,
        String sku,
        int count,
        @Embedded(onEmpty = USE_NULL, prefix = "amount_") Money amount
) {
    static LineItem randItem(String sku, long amount) {
        return new LineItem(null, sku, 1, Money.rands(BigDecimal.valueOf(amount)));
    }
}
public record Money(BigDecimal amount, String currency) {
    static Money rands(BigDecimal value) {
        return new Money(value, "ZAR");
    }
}

 

It maps nicely to the following tables: (sql generated by IntelliJ with the exception of defaulting PKs to uuidv7(), which I added manually)

CREATE TABLE orders
(
    id          UUID PRIMARY KEY DEFAULT uuidv7(),
    customer_id UUID,
    version     INTEGER
);

CREATE TABLE order_items
(
    id              UUID PRIMARY KEY DEFAULT uuidv7(),
    order_id        UUID NOT NULL,
    sku             VARCHAR,
    count           INTEGER,
    amount_amount   DECIMAL,
    amount_currency VARCHAR,
    list_key        VARCHAR
);

ALTER TABLE order_items
    ADD CONSTRAINT fk_order_items_on_order FOREIGN KEY (order_id) REFERENCES orders (id) ON DELETE CASCADE;

 

@MappedCollection and @Embedded work as expected. I also like using records to make it clear that instances aren't mutable. You can then use either CrudRepository or JdbcAggregateTemplate to work with these. E.g.:

public interface OrderRepo extends CrudRepository<Order, UUID> {}

OrderRepo repo;
JdbcAggregateTemplate jdbc;

void examples() {
    UUID someCustomerId; // ...
    Order order = repo.save(Order.forCustomer(someCustomerId, List.of(LineItem.randItem("sku1", 11))));
    order = order.withItems(List.of(
            order.items().get(0),
            LineItem.randItem("sku2", 22)
    ));
    repo.save(order);
    
    List<Order> customerOrders = repo.findByCustomerId(someCustomerId);

    // more complex queries can be done with JdbcTemplate. JdbcAggregateTemplate provides easy mapping to entities though:
    List<Order> customerOrdersByTemplate = jdbc.findAll(Query.query(where("customerId").is(someCustomerId)), Order.class);
}
 

side note: on project reactor

Interesting to dig into a little. Adds a sizable amount of complexity with a lot of caveats (e.g. no support for nested entities in r2dbc). Ultimately not worth the effort except for very constrained use cases. Performance gains for IO bound tasks can instead be achieved with virtual threads.

Wednesday, October 2, 2013

Journeyman weeks - week four @ msgGillardon

Read here about last week at soundcloud...

Nicole Rauch had been a strong supporter even before I actually came up with the idea of doing a journeyman tour. The German Softwerkskammer network has been playing with the idea of craftsmen swaps and Nicole worked hard to make that possible in the company she works for. Hopefully the example will inspire others to do similar things.

During SoCraTes Nicole and her partner Andreas Leidig had offered to host me and eventually got the OK from their employer msgGillardon to let me work there for a week. The company is set in the small town of Bretten, close to Karlsruhe. It's a very beautiful little corner of Germany and I was quite happy with the contrast of small town life compared to frantic Berlin the week before.

View from the top of the office
The part of msgGillardon I was working at makes software doing forecasting for finacial institutions. Most of that is still in C++, with newer parts now being written in Java. I was a little too intimidated by C++ to work on that, which in hindsight might have been unnecessarily self-limiting. But I actually enjoyed working in Java again, since it's been quite a while. The framework used (Eclipse RAP) and the specifications written in FitNesse provided enough things that were new to me and gave opportunity to learn.

I remain skeptical of frameworks that try to hide the complexity of visually representing an application. In the case of Eclipse RAP it at least seems to be thorough and consistent in hiding representation, which makes it more acceptable then Wicket or JSF (*shudder*). If you're happy to stick with what the framework provides and think in terms of desktop applications, as in this case, RAP seems to be helpful. But one of the drawbacks showed itself when we started to write Selenium tests for the application. The effort required to select elements to interact with made automating them too expensive.

FitNesse seems a little crude in some places but I liked working with it. I was missing support to generate code snippets, as Cucumber/JBehave/SpecFlow all do. Aside from that, writing examples in Slim tables fit very nicely for the domain we were working on. There was a lot of combinatorial complexity in the inputs and it looked a lot more comprehensible to have these in FitNesse rather than in java unit tests. It also allowed for a slightly easier active conversation with the product owner about the required functionality.

In terms of culture, msgGillardon is quite different to the start-up companies I've visited in previous weeks (and also quite different compared to ThoughtWorks and its clients). As a more traditional medium sized company working for a lot of customers in banking, things were a little bit more formal and the technology less bleeding edge.

Nevertheless, I was very positively surprised by the willingness to try out new things. (And not just the obvious experiment of letting some random guy show up there and work there for a week. With very convenient and simple organisation.) There seems to be a genuine interest in changing and improving and that is not something I'm taking for granted anymore.

The other thing that definitely stuck out was the diversity of the teams, at least in age and gender. I have no idea how that came to be but it was refreshing to see. I'd be curious if people there have found an explanation for why they are doing so much better than everyone else in Germany.

During the week I also had the opportunity to take part in the Softwerkskammer Karlsruhe meet-up. It was nice to see so many familiar faces from the SoCraTes conference. Nicole and Andreas were running a legacy refactoring workshop that was a very nice alternative to the legacy code retreat format. Like all good workshops, it left me with a lot of things to think about on how I would do things differently if I were to do it again.

This ends my journeyman weeks for now. I feel incredibly grateful for the privilege of having been able to do this. I am in the process of summarising the different experiences and compare and contrast them. I will hopefully also have some useful information for others who want to try something similar. If you have any specific questions or feedback, please do leave a comment, contact me on twitter or write me an email. I'd love to hear from you.

Pictures from the past couple of weeks are on picasa. Also check out Peter Kofler's blog about doing something similar in Vienna and Andy Waite's for something remarkably global.

Tuesday, July 31, 2012

Three tales of browser-based test suites [Part 1]

The topic of browser-based testing seems to come up again and again and on many occasions I meet people who have expectations that don't match my experience. So I guess it only makes sense to write down these experiences and some of the conclusions I came to.

I'm not even going to try and go into distinctions between acceptance, functional, integration or whatever testing. Whatever you're calling those tests, if they are instrumenting a web browser, that's what I'm talking about. (If you care about the distinctions, or about testing in general, I recommend reading Gojko Adzic's excellent book Specification By Example.)

I want to look at three concrete examples that I have worked on over the past years.

In the first case, nobody on our team had prior experience, except for a short but traumatic brush with IBM's Rational Functional Tester (remember kids: friends don't let friends use IBM products). To get rid of that, we decided to go ahead and try Selenium.

We initially recorded tests with the Firefox plugin but quickly realized that this wasn't maintainable and didn't really offer us good control for selecting elements. Our Tester Benjamin then took it upon him to write tests in Java and started to integrate them into our CI environment. Over time he created a solid set of Page Objects that allowed us to write new tests fairly quickly.

It took the rest of the team quite a while to take some ownership of these tests but eventually we came to a point where we'd sometimes even drive new features from these Selenium Tests.

There were hurdles along the way. Tests were brittle and the execution time was unacceptable (>30 mins). This improved with newer versions of Selenium, use of Selenium Grid to parallelize tests and by keeping a close eye on the VMs running the browser instances. In the case of IE(<8 gave="gave" just="just" p="p" simply="simply" up.="up." we="we">
Due to some relatively complex processes it was also fairly hard to test later stages of these processes. We only slowly got better at setting up test data for this without making use of the browser. This left us with some unfortunate holes in automated coverage.

There were still manual tests for each release for all of our major supported browsers. But with more of the core functionality being handled by automated tests and with the amount of changes shrinking due to shorter release cycles the effort for this went down considerably.

We also became better at writing unit and integration tests and our confidence in these grew.

What did I take away from it:


  • the demand for these tests was driven by Benjamin wanting to automate his tests to make his life easier. And also his desire to learn more about Java. It was a relatively clear goal and the benefits of effort spent on automation were fairly clear. Where automation was hard, it allowed him to balance that effort with the effort of doing the tests manually
  • setting up and maintaining browser instances for tests and keeping them stable is hard, unthankful work and I'm glad that these days you can just hand money over to saucelabs to do that for you
  • getting the CI builds to be reliably green made shared ownership easier. Collaboration on these tests continued to grow.
  • closely related to that, the tests have to run fast. If they run for more than ten minutes they might as well not exist
  • we had relatively few Selenium tests compared to our unit-tests but there were the odd cases of bugs that weren't caught by our normal test-suites
  • when we rewrote a lot of the front-end JavaScript and test-drove those with jasmine, browser compatibility issues became very rare
  • as our understanding of all the different tests grew, it became easier to decide when we could avoid writing an extra Selenium test
  • working closely with the rest of the team and writing the tests in Java gave our Tester room to grow and learn and turn into a regular dev on the team. Seeing that happen was probably one of the most fun things during my time there. The rest of the team also took up some of the manual testing work.
(Edit 13.9.15: removed some point about testers that I didn't like anymore)

Continue on to part 2

Sunday, January 9, 2011

Remember back then?

...back in 2010? That was a nice year. I have to say I'm pretty happy with how it turned out. There were some setbacks, first and foremost the departure of two colleagues whose knowledge and support I valued very highly, who left for new cities and new challenges. But such is life. But I learned a lot this year and got to make a lot of interesting experiences and meet a lot of interesting people.

It was a long year and this will be a long post. It will be self-indulgent and most likely only for my own benefit. So there.

First off, I finally took the time to finish reading Domain Driven Design, or "the great blue book in the sky" as I heard someone (probably Neil Robbins) call it. It was very timely as we were just in the process of redoing a fairly large part of our application. It gave me the confidence to split up the refactoring along the life cycle of our main entity, transforming it along the way. That way we were able to concentrate on the first step in the life cycle without having to change everything down the line, too.

Said refactoring was also the first time I remember of deliberately doing "refactoring toward deeper insight". The experience of taking what we had learned about the domain in the previous year and applying that to our domain model was surprisingly rewarding. The end result was a lot simpler and closer to how people actually worked. It gave us the opportunity to introduce new functionality that would otherwise have been extremely cumbersome to implement.

The whole process was not without its flaws though. We underestimated the amount of code we had to change, even with the reduced scope. Especially the front-end proved that good UI design is very hard and time-consuming. In hindsight we probably should have spent more time on considering how the changes might have been divided further into smaller, releasable pieces. In the end we were left with a long time between deployments and suffered all the communication issues that missing regular feedback loops can give you. It was still a good learning experience and it showed me how much I appreciated our previous, shorter deployment iterations.

In march I went to QCon, which I wrote about before. The inspiration I got turned out to have a fairly lasting effect. To get back to the kind of enthusiasm I had when I started out and to be able to really consciously enjoy coding again is something I am extremely grateful for. This wasn't necessarily all due to QCon but it encouraged me to be more introspective about how I work. In particular it had me putting more effort into getting better at TDD. At some point the whole "using tests to drive out the design" bit finally clicked and it made me a happier person.

There is still a lot left to improve though. Most of our bug-fixing work is now purely front-end stuff, ironing out layouting issues and Javascript behaviour across different browsers. But we're getting there. Writing jasmine tests is something I'll hopefully get better at and which integrates well enough into our builds. My colleague Benjamin put an amazing amount of work into getting a useful Selenium tool-set and tests for us. The environment for these is sadly still somewhat brittle but hopefully that will get better with Selenium 2.

We've made a lot of progress in automation anyway. Our test-environment gets automatically redeployed by Teamcity now. These are still very tiny baby-steps towards continuous delivery that conveniently ignore some of the larger issues like schema evolution and the need to switch to a DVCS but at least I think it's stumbling in the right direction.

I was also happy I got to help in making collaboration with our designers on different teams a little easier. Working on a bunch of projects in parallel while trying to consolidate on a consistent look & feel is something that has taken even more effort than I expected. And since look & feel is usually the most immediately apparent contact point for stakeholders, it is always under scrutiny. I underestimated that, too, even though I really should know better. All the more reason to push for further front-end test automation.

Later in the year our team worked on a new project and decided to base it on JSF 2, mostly so we could get the know-how. The other teams were already using it and it seemed reasonable to look for common ground there. I did learn a lot, but I didn't get much done. Having no one on the team with any prior experience meant we had to go through all the little beginners mistakes you get when you stumble into some new technology. Some of those early issues just rubbed me the wrong way which made me get increasingly frustrated. And I'm afraid my growing resentment probably didn't help the mood of the team.

This did bring up a few interesting questions about the effectiveness of aiming for a heterogeneous development environment. Questions which I don't really have answers for but still could ramble on about for hours. So I should probably put that in another post some day.

In November I was lucky enough to take part in a Code Retreat. Having been intrigued by that ever since I saw Corey Haines talk about it at QCon, I jumped at the chance to try it out. This, too, is worthy of its own post and so I won't go into details here. Suffice it to say that I liked it a lot and plan to organize a code retreat in Düsseldorf, to show it to more people.

(As an aside, I just stumbled over a google result for a code retreat in Wroclaw which brought a smile to my face. I always enjoyed my time there, when I visited there for another company, back in another time. And I always had a very high opinion of the people and their programming skills there. So, I guess this fits quite well.)

After Code Retreat Ghent came Google Developer Day Munich. It was nice to meet my now ex-colleague Sebastian again. And same as in Ghent, I was glad for the opportunity to chit-chat with other people in our profession. It never hurts putting things in perspective and hearing what others are passionate about. And of course, rambling about things I'm passionate about. (Something I find myself doing more of these days, I think. But maybe it was just the free beer provided by Google.)

The effort Google puts into these events was impressive and I really liked how non-partisan the speakers were. They openly acknowledged the existence of competitors (foremost Apple) without looking to discredit or awkwardly ignore their work.

I wasn't looking for anything in particular since my work currently targets neither HTML 5-capable browsers nor mobile phones, but at the end of the day I felt more sad about that fact. I wasn't aware how exciting some of the HTML 5 stuff is and I'm really looking forward to that becoming the norm and what people will be able to do with it.

Well, that about does it for two-oh-one-oh. Remaining, then, is the goals I've set myself for 2011 (I'm not yet convinced of spelling it 20!!).


  • Organize a code retreat, possibly making it a recurring event
  • Look for more of these opportunities to learn from others
  • Subversion is a nightmare to merge stuff with and I'm eager to switch to git but it would mean having to get used to a different workflow for everyone in the team. Maybe I'll start with a git kata.
  • Follow through on all the "I should really write a blog post about that"
  • Write more posts in german. 
  • Read GOOS, finish reading REST in Practice, glance over Continuous Delivery.
  • Try to solve more problems event-driven

Hmm... last and possibly least I just realized that I also started writing on this blog (and on twitter) this year. I guess I'll keep that up.

Friday, April 23, 2010

Upgrading to Spring 3

So we decided it was time to upgrade our project to Spring 3. The last time I gave this a try it was a pretty bad experience because at the time I was unable to get the upgraded maven packages without switching to the OSGi names. I'm not sure whether that was an issue with our archiva server or Spring but I didn't run into it this time.

This update also including updating Tiles, Junit and Spring Security.

The latter was not really required and upgrading it turned out to be quite a hassle. Though I guess that hassle had to be dealt with at some point anyway.

Spring Security underwent some major refactoring for version 3, changing a lot of the packages around and also making some minor improvements to parts of the API (UserDetails.getAuthentication and some changes to voters). Adjusting to that was mostly just a matter of organizing imports in eclipse and thankfully most of our access of the relevant classes was wrapped at a few key points.

But it turned out that Webflow hasn't been upgraded to support either Spring Security 3 or Tiles 2.1. Their JIRA has tickets for both and I was able to hack something together from that. I'm curious if there is some better way to handle these versioning issues. Explicitly requiring external dependencies (instead of marking them optional) is too inflexible, if the upgrade is minor. Yet it would be nice to see incompatible changes. Maybe one could put meaning into major and minor version numbers? Oh well, versioning is always tough.

In the case of Tiles, actually removing methods from their api instead of just deprecating them and letting them return null would have brought this particular issue to light more quickly, since Webflow 2.0.9 simply wouldn't have compiled. Why even bother with separate api and core/impl packages?

Other maven changes were limited to removing spring-security-core-tiger (yay to less jdk-specific packages) in favor of spring-security-config. I also had to add commons-codec because it seems to have previously been implicitly required from somewhere else and only been needed since some point after I last ran check-dependencies.

In the end all of this this turned out to be less problematic than I had feared. The knowledge gained from the last attempt (i.e. spring-test update also necessitating a junit update) and the good test coverage helped to sort out most issues before even trying to run the application.

Tuesday, April 6, 2010

Working around type erasure in Java

Due to how generics are implemented in Java, there is no way to determine at runtime, e.g. the type of objects contained in a collection. This can be a bit of an issue if you want to use that type information to look up a specific converter or repository for that type.

What I didn't know was that while the type information is lost on the actual instance, there is still the possibility to get it from the surrounding class declaration. This does require that the instance is declared in a Field and that that Field contains the generic type declaration. Similarly, for classes extending/implementing generic classes or interfaces you can also access type parameters via reflection.

Here's a messy example that hopefully should illustrate both points:

public class LongToStringList extends AbstractList<String> implements List<String> {

private List<Long> someList = new ArrayList();

@Override
public String get(int index) {
return Long.toString(someList.get(index));
}

@Override
public int size() {
return someList.size();
}

@Test
public void test() throws Exception {
// alternatively this.getClass().getGenericInterfaces()[0]
ParameterizedType superclass = (ParameterizedType) this.getClass().getGenericSuperclass();
assertArrayEquals(new Type[]{String.class}, superclass.getActualTypeArguments());

Field field = this.getClass().getDeclaredField("someList");
ParameterizedType fieldType = (ParameterizedType) field.getGenericType();
assertArrayEquals(new Type[]{Long.class}, fieldType.getActualTypeArguments());
}

}
I'm always a bit scared of reflection and so I only learned about it browsing through my colleague Markus' code and then again while stumbling through some of the code in Spring-Binding (which lead here). The latter can use this to determine which converter to use to map form values from an array into a collection on a bound model. And that is pretty nifty if not without its problems.