The not invented here syndrom and technical debt

Probably the single largest source for unnecessary technical debt is the “not invented here syndrom”. The not invented here syndrome is a humorous way to explain a system development problem, which is the natural resistance many developers have towards using 3rd party tools, and would rather reinvent the “wheel” themselves when confronted with a problem requiring “wheels”.

Once you realise that code actually is technical debt, you also understand that you want to minimise your amount of code, to increase your future flexibility, and free up resources in your development department. The simplest way to achieve this is actually to use third party libraries and components. Either Open Source components, or proprietary components. From a technical debt point of view, it doesn’t matter if your component is open source or not – Since every single line of code you can avoid creating yourself, becomes one line of code less you need to maintain in the future. Ask yourself the following question.

What is most expensive; Having a developer spend six months coding, or purchasing a 3rd party tool for €500?

Then realise that even if the developer succeeds, he creates technical debt for you, that you’ll have to maintain in the future. Purchasing a component from a 3rd party, implies that the 3rd party will (probably) give you new versions, and bug fixes in the future, included in their price.

There is a reason why we don’t create our own cars, and that reason is because huge car manufacturers are able to optimise the process of creating a car, to the extent that their product is better, less expensive, and more secure than whatever we could have built ourselves home in our garage. The same is true for all fields that require specialisation of some sort. No company can master all fields required to assemble their entire product – In fact, not even a car manufacturer. Most car manufacturers will purchase a whole range of the components that makes up the final product from 3rd parties. This is a large reason of why they are more efficient at assembling cars than we are ourselves.

Whenever your software development department has a problem, the first question should always be; Have somebody else already solved this problem in an adequate way? If the answer to that question is yes, you can significantly reduce your technical debt by somehow obtaining these pre-existing components instead of reinventing the wheel within your own organisation.

Advertisements

Giving away your technical debt

This might be perceived as counter-intuitive for many managers, but sometimes you can actually save money by giving away parts of your product for free as Open Source.

As I discussed in my previous article about technical debt, you should always keep what is of strategical importance for you and your company a closely guarded secret. For instance, if your business is automated trading, based upon artificial intelligence and deep learning, you’d probably want to keep your algorithms and trading robots a highly proprietary piece of code, and lock it into your safe as you leave your office for the night. However, most companies tends to also create a whole range of additional software solutions that are not of strategical importance to the company, mostly out of necessity. Our example with trading robots for instance, could include the necessity to communicate with 3 different financial database systems simultaneously – Examples here are MT4, the FIX protocol and CTrader, for then to aggregate data from all these different systems together somehow, as input to your trading robots. If such components does not exist from before, the trading company becomes forced into creating such systems, out of necessity. Hence, creating parts of your software, becomes a necessary evil, you must do, in order to facilitate for your strategic advantage.

In the above example, the database adapter has zero strategic advantage for the trading company, but is still something that must be done, because no off the shelf components exists that does this from before. Hence, a database adapter becomes a necessary requirement to make the trading robots function. The problem is that this increases the amount of proprietary code the organisation is dependent upon to function adequately. Hence, it adds to your organisation’s “technical debt”, without really contributing much, besides being a necessary stepping stone, on the way to reaching your strategic advantage.

If you however publicly release your database adapter as Open Source after initially having created it, chances are that others will recognize its beauty, and start using it. As they do, they might find errors in it, and might come up with good ideas for improving it. In addition, they might also fix these errors, improve upon your system, and send these improvements back to you as “patches”. Congratulations, you have now shared your debt, and you no longer need to carry (alone) that which you shouldn’t have carried alone in the first place anyway.

At this point, your organisation have given away zero of its unique advantage or its proprietary secrets, while at the same time is sharing the burden of maintaining crucial components with a whole range of other companies. So even though this is easily perceived as giving away something for free, what you are ipso facto giving away, is actually debt – Nothing more.

In fact, I would argue that all “non-crucial strategical components” within a company should be shared across multiple organisations this way, since we happen to know for a fact, due to game theory, that such strategies results in literally creating value out of thin air, since this is not a zero sum game. What I mean by that, is that the value you gain by giving away that which has no strategical value for your organisation, becomes more than the sum if its participants’ contributions to the system. To speak in laymen terms, one might imagine a world where the rules of math literally dictates that “2+2=5”.

To understand why, imagine a world where every single individual had to build their own roads. No public roads existed, and nobody were allows to use roads built by others. This humorous description is paradoxically a very accurate description of the state of software development today. It doesn’t take a rocket scientist to understand that this would be a highly impractical world. Non-strategic software for your organisation holds the same value for you as roads do in this “parallel world”. Allowing others to use your roads, makes little or no change to your ability to move on these same roads, to sell your services and goods, whatever they may be. While at the same time the burden to maintain these roads, keeping them clean, and fix potholes in these roads over time, can be shared among the rest of society. Facts are, a large portion of your software portfolio today is probably infrastructure software, having little or no strategical value to your company per se, and could easily be “given away for free”, without you actually loosing anything – Except debt of course.

In fact, any financial scientist can tell you the exact same thing, since according to the rule of finance, value is created literally out of thin air, as a transaction between two people occurs – Since afterwards, they both feel happier than what they felt like before, and they have both been given something they want more than whatever they had to give away to acquire that which they were given.

So the ability to create value out of thin air, by giving away something you don’t want (debt), and share it with the rest of the world, is not something new in fact – It’s as old as the idea of trading is in fact. It’s just not commonly applied to the creation of software. In case you doubt it’s impossible to create things out of thin air, then do the following …

Ask an expert in fractional reserve banking if value can be “magically” created out of nothing …

Then realise, this isn’t anything new – We’ve done this for thousands of years in fact. It only implies applying the principles of game theory and finance to the subject of software development …

Strategies for reducing technical debt

This is my third article about technical debt. You can find my previous two articles here and here. In this one I will focus on how you can reduce technical debt as a middle manager or upper manager in an organisation.

Obviously, reducing technical debt to zero is easy, by simply making sure you don’t have any proprietary code at all in your company. If KLOC is the ability to quantify technical debt, then zero KLOC obviously equals zero technical debt. This however also eliminates your company’s edge and uniqueness, and makes it impossible for your organisation to stand out from the rest of the crowd, since everything you build your company from is “off the shelf components”, making it easy for other companies to copy whatever your company is doing – And if your company is successful, others will copy your strategy.

The best historical example of this is how IBM used exclusively “off the shelf components” to build their PC in the early 1980s. Which resulted in that Compaq and other computer vendors could easily copy their product, and such take market shares from IBM. Today IBM barely have market shares at all in the PC industry because of this decision. This is considered one of the classical historical strategical failures in the history of computing today. So obviously, keeping some parts proprietary as “your own” technology, is key here.

Identify your strategy

The first thing you need to do in order to balance this, is to identify your strategy. Your strategy should be summed up with a simple sentence, easily understood by all members of your organisation. I currently work in ForEx, and a good example here might be illustrated with some insights into how my industry works.

In ForEx trading, 80% of revenue is acquired indirectly through “Introducing Brokers”. These are key movers and shakers, often with a massive social profile, with thousands of followers. Making an IB evangelise your trading company, might sometimes result in thousands of traders willing to use your services. Hence if a ForEx company’s vision is to “be the preferred broker for ForEx traders”, a good example of a sound strategy might be as follows.

Provide superior tools for Introducing Brokers

Simply since you realise that if you can convince introducing brokers (IBs), you don’t need to convince your end customer (traders). The IB will effectively make the decision on behalf of his clients.

At this point you know what you don’t need to create yourself. For instance, superior tools for IBs does not require you to create your own CRM. Hence, you can use off the shelf components for your CRM. You can also use third party components while building your software, such as database adapters, other libraries and frameworks, that allows your developers to more effectively do their job.

However, the one thing you really want to build yourself, is the “partner management system”. This is the system your IBs are using on a daily basis to do their jobs, check their profits, and create sign-on links for their clients again. This system is arguably the only system in this example you’d want to create yourself, while everything else are things you could easily use pre-fabricated components and tools to accomplish. This allows you to focus all your energy into creating one staggeringly good system, while buying or obtaining everything else in “the market” somehow – Either by using Open Source components (Sugar CRM), or by purchasing services and libraries from other 3rd party vendors somehow.

In my next blog I will look at additional strategies for “outsourcing” the creation and maintenance of components and supporting tools, by leveraging Open Source as an asset in your organisation. But for now, let this article sink in, and try to identify your company’s main strategy, which is never the same as your company’s vision, but rather an “activity” that facilitates for your company’s ability to reach its vision.

Measuring technical debt

Once you realise you have a problem, understanding how much of a problem becomes imperative. Hence, some sort of model to quantify your problem becomes urgent. This is actually surprisingly easy though, since science shows us that the cost to maintain one line of code, is fixed over time, regardless of programming language. Hence, to quantify your technical debt, is actually as easy as counting your KLOC, or thousands of lines of code to speak in non-technical terms.

As you do though, please realise that simply because you have a lot of lines of code, doesn’t necessarily imply that you should simply throw it all out, to “reduce your debt”. However, since the cost of maintaining a single line of code is fixed, and a single developer can maintain somewhere between 50KLOC to 250KLOC, depending upon his skillset, and whether or not he was instrumental to its original architecture, etc – At least now you can start quantifying how much resources you are at the minimum in need of, in case tighter times are to come in the future.

This results in that you will know more about the potential future requirements of your company, and also be able to project what amount of freedom you have, in regards to resource spendings in the future, in order to maintain your current operational level of your organisation.

In my next article, I will visit strategies for reducing technical debt, and such create more freedom for you, and your organisation – But for now, feel free to run a count script through your Git repository, if you’d like to understand your organisation from a high level perspective. In fact, if you haven’t done this already, I would argue it’s the equivalent of driving your car blindfolded. KLOCs might not necessary be the perfect instrument for measuring quality, and measuring innovation – But it’s definitely the by far simplest way to measure your technical debt, allowing you to understand your own organisation, with a simple script that you could probably have one of your sys-admins execute in a couple of minutes. It’s the equivalent of windshield wipers on your car, making sure you’re able to see the road ahead of you – At least if you have the creation of software as some form of primary or secondary function within your organisation. In fact, KLOC should be one of the primary KPIs in any organisation serious about producing software of some sort.

Unless you can give me your KLOC count, you’re arguably blind to what your own organisation is capable of

Technical debt

Every line of code you create, must be maintained at some point in the future

Technical debt is one of those things that is difficult to explain to middle and upper management. Especially in companies that are creating software as a secondary function. However, it’s really quite simple to understand; It is the cannibalisation of future freedom and manoeuvrability in order to reach short term goals. Logically it’s the exact same thing as borrowing money today, without any thought of that you’ll need to repay your loans later. Therefor it becomes crucial for a company to understand this concept. Without an understanding of technical debt, it becomes impossible to create a sustainable business – At least a business that employs software developers.

The primary cause of technical debt is “code”. Every time you need a piece of software, you should ask yourself if you can get away with avoiding creating it yourself, and use off the shelf components instead. Every single line of code you don’t create, is one line of code that can never create technical debt for you. This is arguably the sole reasons why companies choose to create products out of code, since it allows one company to take on technical debt on behalf of other companies, in return for some monetary gain of some sort. This is why most companies chooses to purchase a word processor instead of trying to create one themselves. At the same time, owning (some) software can have strategic value for a company. A company that exclusively chooses to purchase its software as third party components, doesn’t have anything special that separates it from the herd. The key as always is to balance things.

Basically, you want to create as few lines of code as possible, while still retaining your competitive edge, making you unique in your market.

Due to that it is so difficult to understand this concept, it becomes crucial to make sure you hire people that do understand it. Hiring only junior developers, results in unimaginable amounts of technical debt, since juniors don’t have the experience required to understand the consequences of their actions, and that their “brilliant solutions” are things that needs to be maintained in the future. And the result becomes that the company increases its technical debt, until it can no longer sustain its operations – The same way a loan of money is defaulted upon due to not being able to pay the installations leads to bankruptcy. By all means, hire junior developers, but make sure you’ve got at least one developer in your organisation that understands the consequences of what is pushed into your GIT repository, and that can explain it to you, such that you can understand how your company is actually performing – Otherwise you might as wells start flipping burgers immediately, since that’s where you’ll inevitably end up at some point anyway.

A key component to avoiding technical debt is to ask yourself the following question every time you’re tempted into creating a piece of software; “Is this a part of my competitive edge, or is this something where I could get away with purchasing a third party component to solve my problems?”

If the answer is that you could purchase a third party component to solve your needs, but you still choose to create it for yourself – Then you are accumulating technical debt, and at some point in the future, your debt will come back to haunt you.