Productive Haskell in Enterprise

Oct 4, 2018 - Brian Jones

This post is semi-related to our last Introducing Haskell to a Company article. The goal is to show how a small team in an enterprise-like environment can be productive with Haskell.

I looked back through our git history to pinpoint when exactly we started using Haskell in an official capacity. It turns out it was only a year ago to date when the entire team shifted gears and went all hands on deck with a high priority application (currently at the tail end of development).

Statistics

Hopefully these numbers give more insight into our situation.

Developers

We currently have 8 full time developers and 1 intern.

Non-Haskell Tasks

It’s worth nothing what we do when not writing Haskell.

  • Time is spent supporting the core vendors as they do their upgrades
  • Time is spent supporting existing C# applications
  • Time is spent rewriting and fixing SSRS reports
  • Time is spent handling all of the miscellaneous developer related tasks one tends to do in a support company
    • Updating various websites/wordpress sites
    • Implementing PHP speed test tools
    • Writing SalesForce API tooling
    • Supporting SharePoint forms
    • etc, etc, etc
  • A developer spent a few months writing a complex C# ETL job to snapshot various ISP network device equipment states

Haskell Code

Projects

I just checked, and we currently have close to 100k SLOC of custom Haskell code in our repo.

  • Internet facing Customer Self Service (billing, internet plan management, profile updates, marketing, etc)
    • Servant backend API
    • Customer facing Reflex SPA
    • Internal facing administrative Reflex SPA
    • Android and iOS apps to be built from the same customer facing Reflex code - (thanks Obsidian Systems!)
  • 6 distinct microservices (Servant APIs and Clients)
    • Wrap vendor blackboxes and their monster non-standard APIs and/or access an IBM DB2 database
    • Currently support the Customer Self Service, but allow us to scale out future apps with ease
    • Email/SMS template and notification management
  • Internal Network Device + DNS Management tool
    • Servant backend API
    • Reflex SPA
  • Internal Sales Team Upsale tool
    • Servant backend API
    • Reflex SPA
  • Helpdesk Tool (intern + mentors)
    • Servant backend API leveraging ConnectWise API
    • Reflex SPA
  • Sales Management tool
    • Servant backend API
    • Reflex SPA

No less than 10 projects of varying complexity and priorities on our backlog.

Libraries

One of our primary goals is to continue to abstract out as much as possible, which has made spinning up new projects a breeze. We continue to refine this process.

  • A custom app stack building library which allows us to easily compose together on demand:
    • Servant APIs (with routes proxied from the main app)
    • Katip (logging)
    • Configuration via Dhall
    • EKG (app monitoring)
    • Generic database pool management for multiple sources
    • Task management and smart thread shutdown (ability to run non-web daemons)
  • Custom LDAP wrappers, PostgreSQL transaction helpers, general utility functions
  • Custom Reflex widget library wrapping Material Design CSS
  • A custom Prelude (based on Protolude)

Some attempts at open source tools:

That last one lays some groundwork for interfacing with ISP network devices over telnet, relevant to future projects in our pipeline.

Maintenance Goals

A story Chris told me helps provide perspective.

One day Chris finished writing a piece of software, the users were pleased, and life went on. Roughly 5 years later Chris was asked to add a few small features to the project. Chris told the customer that it would take no less than 2 days to implement, and the customer clapped their hands with joy. Chris got blind sided by the bit rot and spent 2 unplanned weeks catching the application up to the latest version of its core Java libraries.

Opinions about how things could have been better handled aside (it was a decade ago), we’ve probably all had to update an application we haven’t touched in forever, and it may have taken a lot more work to update than we had planned. Whether that was due to remembering the scope, lack of really good tests, spending time asking yourself if it really was you who wrote that terrible code, or simply spending all of the time catching the application libraries up to the current year, it was ultimately an unplanned maintenance time sink. In certain cases this is where an inexperienced developer may even suggest writing version 2, much to the horror of management.

Pinning Versions

To get around bit rot we lockstep every single application we write with a nixpkgs commit pin. This allows us to:

  1. Deterministically build every single application from a known state
  2. Walk our system and Haskell dependencies forward and allow the CI to catch anything that breaks
  3. Do incremental maintenance on everything we’ve ever written

Does it take extra time to do this? Absolutely. However, it’s much easier to spend a day of planned maintenance on everything today than potentially 2 weeks of unplanned maintenance in the future. It also forces us to make very conscientious decisions about breaking changes to shared libraries.

The best part of all comes right back to Haskell. The compiler allows us to refactor a dozen applications without fear! I am not joking, 7 days ago I pushed us forward and touched 112 files across 18 CI tracked projects with +1,336/−1,882 changes, and everything is still perfectly functional.

A Note on Hiring

Hiring in Alaska is much different than tech hub cities. While there are a lot of people that would give their soul to hack on Haskell, we haven’t convinced any of them to move to Alaska. When we hire for a new position it’s usually “does this small stack of resumes have someone who actually knows how to develop software period?”, and not, “let’s pick the cream of the crop and take our time interviewing them.” It is at times tempting to hire a sub-par candidate, but that would be a disservice to the team.

A person might argue that this is the perfect reason to not pick Haskell as a core development tool, because it is supposedly hard to hire for. However, for us developing quality software for our customers is the goal and finding the right people to achieve that goal is more important than having someone be 100% productive on their first day.

I’d rather hire a developer that has solid development experience and is a genuinely pleasant individual, regardless of the language or the tool stack they are experienced with, because they have the capacity to be trained. Building software is about more than just the language and frameworks you use.

  • Do you have solid soft skills for communicating with customers and peers?
  • Can you write clear and concise documentation?
  • Do you understand how to test software, and why it is important?
  • Do you understand how to use tooling like git and GitHub?
  • Are you capable of tracking tasks inside issues, kanban, using milestones?
  • Do you understand what it’s like to work on software at scale?
  • Do you have solid devops experience as well? Or at least understand the philosophy?

The list goes on. And sure, it takes us 1-2 months to get someone productive with our tooling, but the payoff when you look 5 years out is considerable when you keep in mind that 60%-80% of a developer’s time is actually spent on software maintenance, something Haskell has a great story for doing well.

Conclusion

We’re a small team. We work on a lot of projects, which means we have to maintain a growing number of active projects. The Haskell type system and GHC compiler are like platinum cards for paying down on technical debt.

Show stopping production bugs: 0