The Gamelog purchase form refactor, part one

The purchase form

The most complex form in the Gamelog app, by far, is the one used to create and edit video game purchases. I didn’t really appreciate all the complexities when I set out to build the form over a decade ago, but it quickly became apparent that there was a LOT going on here.

The core concept is simple: every purchase consists of some metadata related to where I bought something from, and then a list of purchased items. Each item has something explaining what I’ve bought, some dates for when I purchased it and when it arrived, an MSRP to keep track of the normal price, and the final price I paid (before taxes). Finally, there are some price fields related to the whole purchase, for calculating taxes, store credit, coupons, etc.

The complication is that each purchase item doesn’t just have a text field where you write in whatever item it is you’ve bought. Instead, each item links to an actual Game, DLC, Hardware or Currency object in the database. And because I’m buying stuff, chances are I don’t already own the item in question, so the relevant database entities don’t already exist and need to be created. In addition, there are a bunch of features that facilitate common tasks when inputting a purchase.

Here are a bunch of things the form needs to be able to handle:

• If I’m buying something from a store for the first time, I need to be able to create the store via the form. A new store needs a currency attached to it; if I’ve never bought anything in that currency before, I need to create that too.
• Similarly, if I’m buying a game for the first time, I need to create the game via the form as well. Game objects require a Platform (i.e. PC, Nintendo Switch), which also needs to be created if it doesn’t yet exist. This happens rarely enough that I didn’t bother supporting platform creation in the form, but you can see an obvious problem whenever I DO need to do it.
• If I’m buying a piece of DLC for the first time, there are extra complications. Often, I’ll buy some kind of bundle that includes a game as well as all its DLC. DLC items require a link back to the parent Game. If the Game doesn’t actually exist yet, how can we create DLC items for it? So the form needs to handle this case somehow as well.
• Purchases can be one item or a hundred items. (This has actually happened before!) The form needs to allow me to add or remove items from the form as needed.
• Often, bundles have a single reduced price for all the items in it, but in the database we store everything on a per-item basis, so some math needs to happen to split the cost of a bundle up properly across the various items in it, according to their original MSRP (when possible).
• Purchases in currencies other than my native one need to be converted to CAD so I can do like-for-like comparisons. Of course, when you buy something, you might not know immediately what the conversion rate you got was, so the form needs to be able to handle cases where there’s no currency conversion yet, as well as cases where we know what that figure is.

The original tech stack

• Django 1.x (starting shortly after the concept of model forms was added to Django)
• jQuery 1.11, fun

By this point I’d had some experience with Django already, having built two relatively small production sites for various publications. But since everything I’d built up to that point was essentially a one-person affair, I’d essentially learned how to do everything on my own using my best judgment at the time. This means idiosyncratic or just plain wrong ways of coding can easily work their way into the final product. In this case, probably the biggest single mistake I made here was not learning how to use model forms in Django, and trying to write my own form classes (and later field widgets) from scratch or cobbled together from existing Django code. While these custom form classes have some validation, it’s by no means complete and it’s not at all enforced client-side, meaning form submissions break everything much more often than I’d like (i.e. not at all).

This also causes issues with saving entities to the database: because I wrote my own code to save entities based on the form input, sometimes I’ll save something prematurely before checking that the input for related objects is valid, which then means I’ll have a random Purchase object in the database with no associated PurchaseItem objects—basically, a non-sensical Purchase. Or sometimes there’ll be an issue saving one of the PurchaseItem objects to the database, leaving an incomplete Purchase that needs to be manually cleaned up later.

To my credit, some of this needless DIY effort was because of how I’d originally planned the user flows. For example, in many cases it’s necessary to create an object that doesn’t exist, like a store or a game. To accommodate this in a single form without the user having to split off into another tab or window—pretty common back then, but much less so now that popping up modal forms in the same screen is a common practice—I basically came up with a design pattern that allowed the user to either select an existing object or provide the data necessary to create a new one.

In the above image, the first purchase item shows an existing game, SimCity 3000 Unlimited for PC, attached to the purchase item. The second purchase item shows a new game, Civilization V, as well as additional fields that describe its type (Game) and platform (PC). The third item shows a new piece of DLC, Gathering Storm, as well as the field describing its type (DLC) and an additional set of fields to tie it to an existing or new game (the latter field is used and has the number 6 filled in, to show that the item is connected to the 6th item in the form, which is the entry for Civilization V).

Standard model forms can’t really handle all these extra fields that are technically unrelated to the underlying model; a PurchaseItem doesn’t have a platform or type field because it expects a reference to an existing Item, which should already have all that data. As a result, I ended up building a rather elaborate custom form class to handle all of this, and it’s kind of a Frankenstein’s Monster at this point. But in order to refactor it, some assumptions made in the construction of the original form might need to be rethought.

How to bring this form into the modern age?

I’d begun experimenting with a second version of the form years ago. My primary consideration was to get rid of all of the custom code I’d written for saving entities based on the form input in favor of leveraging the now-quite-mature model forms system in Django. Now that I have an endpoint that uses the new model form, I have an idea of just how much hard-to-maintain custom code I’ve managed to bypass: the old code for saving a Purchase and its associated PurchaseItems was 129 lines, not including stuff like spitting out form errors in the response. The new code uses just FOUR lines. So already there are some benefits. But in order to actually use this endpoint properly, I need to also spit out an HTML form that can submit to it. And here we come to a crossroads: how should I rebuild the form front-end?

As part of this process I decided it was time to change how new related objects get created by the form. Instead of providing fields in the main form itself for creating an object, and letting the user pick between selecting an existing item or filling out the info for a new one, I decided it made more sense to default to selecting an existing item and then having the user take an additional action if they wanted to create a new one instead, which would then change the form’s display so the user could input the relevant data, and then collapse back into the original select-only UI but with the new object already incorporated into the form. In other words, instead of trying to save the Purchase, PurchaseItems and any associated new related objects all on form submission, the user would create the related objects as they filled out the form, and then submit the form to update or create the new Purchase and PurchaseItem objects, which could then reference the new objects that had already been created.

All well and good, but what would the actual interface for this look like, and how would I build it? Time to do some research and experimentation. The last time I tried to do this a few years ago, I tried using React. This would still be my primary option this time around, but React does have its own cons:

• Because you essentially need to describe the entire interface inside React, much of the help Django offers with forms can’t really be reused easily. This isn’t necessarily a bad thing but it does mean you’re sort of joining two systems together instead of working in one integrated system.
• The way PurchaseItem forms worked would have to be completely rethought. The React form, by default, wouldn’t know anything about the various Games, DLC, Hardware, etc. objects that are in the database, and the server can’t really inject that data into the React UI on page load because there’s so many items in the database that requesting all of them and dumping them into the page is pretty heavy. In fact, this is one major incentive for moving away from the existing system, because right now this is what every PurchaseItem fieldset in the form does: each item select widget contains a list of EVERY Game, DLC, Hardware and CurrencyItem object in the system, and worse, Django doesn’t cache this database call. This means just loading the page takes forever. Reducing this to one call is certainly better than the existing status quo but it would still make the page incredibly slow. Better to try a different approach.
• React in general has a lot of moving pieces that all need to have proper testing behind it, so it’s a lot of work to build everything compared to the usual way you’d build a Django form, which is to let Django’s built-in template helpers do the work of constructing a form for you.

Research brought up another potential way of doing things: HTMX. It got some praise from Django users for feeling more in-line with the full-stack development mentality and allowing people who didn’t feel as comfortable working in Javascript to create reasonably interactive front-end interfaces using techniques they were probably already familiar with. I’m reasonably comfortable in Javascript myself, but the thought of being able to build all the form interfaces inside Django instead of creating a second front-end-only system to manage the interactive bits was appealing.

Next time, I’ll talk about the experience of adapting the form to HTMX and where I think it can be useful and where I think it falls short.