Google Tag Manager: Tag Priorities Vs Tag Sequencing

Google Tag Manager: Tag Priorities Vs Tag Sequencing

 

As most GTM (Google Tag Manager) users will agree, this is a much discussed yet confusing topic! The documentation on these topics is very concise and to be honest precise in describing what these two options do and what to expect, yet some of the side effects of these two options, combined with asynchronous nature of JavaScript are left out to be inferred by the users. And this is where much of the confusion seems to come from. Even the many blogs out there on this very topic barely touch this context.
Hence we are going to discuss this very thing today.

Tag Priority

Tag priority as described by the documentation is a number associated with a tag which identifies the order of firing the tag. Firing, not completion. Secondly, firing is itself an asynchronous process. If you consider all simple HTML tags, firing them meaning adding them to the HTML of the page, which is what the GTM script is responsible for. What tag priority means is that all the html elements will begin to be added in order identified by the Tag Priority but the GTM script will not wait for the elements added before to load and execute before adding next. (also it cannot know if it is finished, more on that later) And hence this does not govern the load order of the tags.

Tag Sequencing

Tag sequencing as described is a setting that governs which tags will fire before and after a particular tag. One can imagine this as a setup-run-cleanup processes, established with GTM tags (which is also apparent in the documentation). Think of it like a unit test; there is a setup which will fire before the tag (@Before in JUnit, or beforeEach in mocha), then the tag itself (@Test in JUnit or it("", ()=>{}) in mocha) and then the after-tag/clean-up tag (@After in JUnit or afterEach in mocha). If you were trying to make sure that a given tag fires before another, you should be happy sequencing exists.

Read the document carefully again and you will see it does not speak about completion yet again! Much like priority, sequencing cannot guarantee the completion of setup tag before firing the middle (test) tag! It will only ensure that setup tag is ‘fired completely’ before moving on to fire the middle / clean up tag.

In the Tag Priority documentation, it correctly states: “Tags will still be fired asynchronously (tags will fire whether or not the previous tag has finished.)” and “Tag Sequencing allows you to specify exactly which tags fire before and after a given tag.”. But says nothing of this sort in the documentation of Tag Sequencing.

By nature of JavaScript, it is difficult to know when execution of a particular snippet completes without explicit notification from the snippet; it can be in the form of an event being fired or a callback being triggered (Promise will come under this too); but as GTM does not ask for either, it cannot really know if your tag is ‘completed processing’.

We can see this by doing a simple experiment. In a GTM container, let us create 3 custom HTML tags as:

1. SetupTag:


<script type="text/javascript" src="https://code.jquery.com/jquery-1.12.4.min.js"></script>
<script type="text/javascript">
  sayOnDoc("'jQuery' object defined on page: " + !!window.jQuery);
</script>

<script type="text/javascript">
  jQuery(document).ready(function() {
    sayOnDoc("'jQuery.ready' fired.");
  });
</script>
<script type="text/javascript">
  document.addEventListener('DOMContentLoaded', function() {
  	sayOnDoc("'DOMContentLoaded' fired");
  });
</script>
<script type="text/javascript">
  var gtmName = 'google_tag_manager';
  var insideGTMAndDomReady = window[gtmName]
            && window[gtmName].dataLayer
            && window[gtmName].dataLayer.gtmDom;
  if (insideGTMAndDomReady) {
    sayOnDoc("'GTM ready' done.");
  }
</script>

<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjs/3.16.5/math.min.js"></script>
<script type="text/javascript">
  sayOnDoc("'math' object defined on page: " + !!window.math);
</script>

<script type="text/javascript" async src="https://cdnjs.cloudflare.com/ajax/libs/d3/4.11.0/d3.min.js"></script>
<script type="text/javascript">
  sayOnDoc("'d3' object defined on page: " + !!window.d3);
</script>

<script type="text/javascript" async src="https://cdnjs.cloudflare.com/ajax/libs/angular.js/1.6.5/angular.min.js"></script>
<script type="text/javascript" async=true src="https://cdnjs.cloudflare.com/ajax/libs/ag-grid/14.0.0/ag-grid.min.js"></script>
<script type="text/javascript" async src="https://cdnjs.cloudflare.com/ajax/libs/underscore.js/1.8.3/underscore-min.js"></script>
<script type="text/javascript">
  sayOnDoc("'_' object defined on page: " + !!window._);
</script>
<script type="text/javascript" async src="https://cdnjs.cloudflare.com/ajax/libs/backbone.js/1.3.3/backbone-min.js"></script>

<script type="text/javascript">
  var someScr = document.createElement('script');
  someScr.onload = function(){
	sayOnDoc("Moment js added to page");
  };
  someScr.src = "https://cdnjs.cloudflare.com/ajax/libs/moment.js/2.19.1/moment.min.js";
  document.head.appendChild(someScr);
</script>

<script type="text/javascript">
  setTimeout(function() {
  	sayOnDoc("My timer timed out..");
  }, 500);
</script>

2. MiddleTag:

<script type="text/javascript">
  sayOnDoc("Middle tag fired");
</script>

3. CleanUpTag:

<script type="text/javascript">
  sayOnDoc("CleanUp tag fired");
</script>

And create an html file or jsFiddle like this.
Now we simulate both scenarios:

Priority Test

Setup: Have all the three tags triggered on ‘All Pages’. Set priority of SetupTag set to 20, MiddleTag to 10 and leave CleanUpTag empty or zero.

Observation: The Setup tag is the first to fire. Mid way during the execution MiddleTag and CleanUpTag fire, while the Setup tag is still to complete.

Sequence Test

Setup: Have only MiddleTag triggered on ‘All Pages’. Set CleanUpTag as the clean up tag and SetUpTag as the setup tag for the MiddleTag.

Observation: The SetupTag starts to fire, all the script additions from the setup tag are done first, then the MiddleTag fires and finally the CleanUpTag, almost as if the next tag waits for the previous to complete. Yet the asynchronous sections of the SetupTag fire way after the CleanUpTag!

Additional Observations to Note

Note that in both cases the ‘async’ scripts cannot be guaranteed to be executed in order. Also that the event ‘DOMContentLoaded’ is completely skipped, this is because the event happens before GTM starts firing at all. Also interesting to note that although jQuery is loaded before, ‘jQuery.ready’ triggers after the ‘GTM ready’ is written to document, meaning that although jQuery is loaded on the page, there is a delay before the ‘ready’ event is fired; only that the delay is not long enough for text to show up after Middle/CleanUp tags. In both cases the completely asynchronous snippets: adding a script to page and a timeout delay happen way after the CleanUp tag.

There is a whole lot to discuss on the script loading and execution in browser, and asynchronous nature of JavaScript itself, which we have not and cannot cover in this post. There are a whole lot of different things that can happen depending on how we write the code in the tags and the browser you load the tags on. But at least we know that we cannot rely on GTM blindly to sequence the tags, especially if there are any asynchronous components in them.

As a side note, there are a whole lot of different and interesting scenarios that arise when we combine the priorities and sequencing with Tag firing options.

Software development hygiene: Why do we brush our teeth?

Software development hygiene: Why do we brush our teeth?

Yes, why do ‘you’ brush your teeth?
Is it guaranteed that if we brush our teeth twice a day, floss once a day, gargle with an antiseptic, we will never have toothache or bad breath? And if we did not brush teeth say, for a week, would we be guaranteed to have toothache? For a few months, may be yes, we might, might just have to get some treatment done for a few teeth. So the question, why do we brush our teeth, daily?

And how did we start brushing the teeth? Were we born with a brush in one hand, toothpaste in other and with an utter, inexplicable desire to brush teeth every morning after waking up from sleep and before going back to bed? Assuming that no one would remember how they themselves were born, all parents at least will agree with me, that this is certainly not the case. So the question, how did we start brushing our teeth daily?

And now the question you might have in your mind: “What’s the point?”
Recently, a person on our team raised this question(s): Why do we have unit tests. I have been writing good code, good enough that QAs do not find any critical issues, nor has anything ever severely broken in production because of my changes, why should I write tests? If I could think of all scenarios to unit tests, why do we have dedicated QAs on our team? Why should I pass my code through a static code quality analysis tool? All these processes are slowing us down. I have worked without all these processes in the past and that has worked quite well, why do I need this overhead of processes?

I agree, I hate processes.
Yet we need to appreciate the importance of processes and acknowledge where they are required. Come to think of it, why does a process exist? Can we not work without processes and the overheads thereof? Short answer: No, we cannot. Long answer: We can, given that everyone on the team understands the core reasoning for the existence of the process being bypassed and takes the responsibility of upholding the goal of this process without strict adherence to the process itself.

Well, how did I start brushing my teeth daily? My mother would tell me: till I was a couple years old, she used to brush my teeth. When I became three, she taught me how to do it and would ask me to show how clean my teeth were. She would ask me: “Are they shining when you look in the mirror?”, I would go and check and say “Yes”. When I became four, she would just remind me to brush, and I think at five I had finally started brushing my teeth daily, without having to show her how clean they were. I do not believe your story would be very different than this. It took years of practice and perseverance of our parents to eventually get us to brush our teeth daily so that finally we could get rid of the ‘overhead of process’.

Yes, many processes can be chucked as long as the goal is achieved; but are we, as a team, responsible enough to make sure they are achieved every single time? Let us say we are, but are we ready to carry the burden of remembering every single code smell, every single potential bug and be mindful of it while writing code? Is that even humanly possible? If the answer to that question is yes, sure go ahead and chuck the quality analysis tools, unit tests, pull requests and code reviews; we don’t need them. But if the answer is no, wait till it becomes yes!
We can certainly bypass processes and get an apparent speed-up, but chucking a process before we are ready is sure to give us pain in the tooth (and in a few more places)!

Experience: Introducing JMockit To The Team

Experience: Introducing JMockit To The Team

Like many codebases out there, our codebase at work had a backlog on unit & integration tests and it was high time we covered it up. So one fine day, it was decided that we shall no longer accept code without tests. Then the question of ‘how do we write tests’ came up. As one of the architects on the team I introduced the methodology of unit/integration testing and a mocking library (JMockit) to aid in cases where testing could be difficult without one, conducted trainings and hands-on sessions for everyone on the team, set up a peer review process and we were ready. That is all there is to the history of the situation we are in. Well, we are talking about a large team here, some eighty people working on a large codebase as a whole, although divided in multiple microservices.

Today, almost every test we have has a mocked class, and an expectation set on some dependency. Many tests verify how many times a particular internal/private method or a method on a dependency was invoked, some have VOs mocked, or have assertions on return values inside verifications block. A few have gone to the lengths of mocking constructors of certain objects because there was no way to inject them into tested class. We are not even considering the private and static method mocks here.

When I look at the tests that we have today, and look back on the last few months, I wonder if I made mistake while introducing the mocking tool. In the trainings we had, we discussed the purpose of a mocking tool, issues due to overuse, indicators of overuse; heck there was even slide dedicated to this in the developer induction we have here. Architects were involved in many code reviews and tried avoiding these pitfalls, but clearly it was not enough.

Tests are supposed to improve the design of the code. Since the highly tangled, coupled classes, classes breaking SRP (Single Responsibility Principle) are difficult to test, we tend to fix them. As the size of the class increases, the functionality it has grows making it difficult to test, so we split it. As the dependencies being created makes it difficult to test, we change the class to allow injecting them. We end up splitting large methods, redesigning methods with side effects, removing unneeded code, decoupling from the libraries all for making the code testable and effectively get cleaner, maintainable, verifiable code. All of this, only if we wrote tests correctly.

If we started modelling our tests to match our code, we not only lose all the benefits, but the tests start becoming coupled to our code. That brings down the speed of refactoring or new development because every time we change code, we need to change the tests to match the implementation. That brings down our overall productivity. And finally blaming it on tests, we would stop writing them altogether. Back to square one! Mocking tools have a purpose, but if we mock everything, we get our tests tightly coupled to the implementation, adding to the problem. Mocking simple data-structures and VOs is not meaningful, we never test their methods separately, they are not supposed to be tested. (And yes, VOs are data structures, let us not get into that here.) Mocking external libraries is risky, because we then verify their actual behaviour only at runtime, which is exactly what we are trying to avoid by writing tests.

JMockit is a powerful tool, a little too powerful, and harmful if we are not careful. Despite the misuse that we have done, it could do all those things is itself marvelous. I am not convinced to blame it on JMockit, it is as Uncle Ben said to Peter Parker: ‘With great power comes great responsibility’. What we did, is our fault, we should have been more responsible with it. I wish I could go back and change the way we used it, or overused it, but our life is not a Git repository.

Luckily though, we have identified these issues and their severity before they start heavily weighing us down. What we need to do first is avoid more such tightly coupled tests from getting in. The course of action now seems like along with training the team on how to be more responsible with tests (which by the way needs to be a continuous process), we need to identify reviewers and train them on spotting such instances. Revising the review process to have reviews through identified reviewers and not just peer reviews.

Please Give It A REST!

Please Give It A REST!

A regular stressful day in the life of a software developer. I was communicating a module we needed to quickly put together. The team was not exactly new. We had a backend guy, and a front-end guy. Interface was designed and agreed upon, we needed to make it quick, we just defined the resources and said that we need the ‘standard CRUD operations done via REST’ on them. And we got down to work.

When everyone reverted that they were done, we sat down to quickly integrate and test, but the front-end and the backend refused to talk, we got the 404, not found. Well, it should have been simple, we had very clear instructions; the resource names and that they were REST.

Or so I thought. REST, the standard, arguably most popular of the web service protocols out there should have made it very easy for the backend and frontend to talk to each-other. But no. Turns out that there are (still) huge misconceptions around REST; there are so many among us who believe that abominations like ‘getCustomer’ and ‘createCustomer’ (yeah, you guessed it write, the HTTP method was ‘GET’ for this one too) are ‘resources’ and are valid REST.

Oh please, give it a rest.

Not the first time have I encountered this, and it would not be the last either. I thought we were over this. But no, I had fallen for it, assuming that common knowledge is common. It is not.
Well, these ‘divergents’ among us, are not totally wrong in assuming what they write is REST, it could probably be referred to as a form of REST. They are on HTTP at least, someone could certainly fit the level-0 in Richardson Maturity Model to it. But this type of API modelling does not really help integrations, has no standard understanding nor predictable way of implementation and yet we tend to stick with it.

The level-2 is where we need to be with REST, as a basic understanding and expectation from APIs: with Verbs and Resources. It conveys a very clear message as what to expect
from the interface. While Level-3 or HATEOAS with a hypermedia client, or if you are so security conscious making the URLs opaque, would be a dream implementation but for a team struggling with ‘resources’ it feels far-fetched. So level-2 it is.

I have seen that there seem to be a whole lot of people with this kind of confusion around, entire applications built with just GET calls for everything, even for inter-service and frontend-backend communication. I wonder at times what could have caused this confusion, and popularity of Spring’s for REST implementation comes to mind time and again. It is also an observation I have made over time, those who think ‘resources’ also tend to be Jersey (JAX-RS) users at some point in time and the other class seems to be comprised largely of Spring users for their REST implementation.

Probably it makes sense, Jersey as a framework leaves little room for the idea of method-wise resource names, you tend to define the resource at the top of the class (or resource!) and HTTP methods are just marked below. Although you have the option of marking each of your method differently with an additional @Path annotation, the trend is not seen as much and people are forced to think in terms of resources and verbs. Whereas the Spring’s implementation of REST is basically the Spring MVS’s controllers and request handling used to simulate how a REST service will work. Although Spring 4 introduced the @RestController it did little to enforce the resource behaviour as did @GetMapping, @PostMapping and the siblings defined in Spring boot; the method implementations are still standard (or implied) @RequestMapping from Spring MVC and coming from there, people may tend to think a need to override the handling and define the java method name in the path, for some reason. Well, I would not know for sure but this seems like a logical explanation for the trend observed.

Now how do we convey to everyone that they do not need to define different paths and let the HTTP verbs do their job is a challenge. This post is just an attempt.

Participating In A 24 Hour Hackathon

Participating In A 24 Hour Hackathon

Just returned from a 24 hour hackathon, sleepy, red-eyed, tired, exhausted and yet writing this post. You know why? Because I skipped it the last time, and the time before, thinking I will do it the next day and that sleep was more important, but never did it. Not going to make the same mistake again. So here I am.

For those who are unaware of what a hackathon is, it is an event where dreamy eyed people enter and leave with red eyes, it is a single night sprint where people come together to build something that they believe will make them a billionaire or like Silicon Valley series mentions time and again, ‘will make the world a better place’! Well, jokes aside, a hackathon is an event / competition where teams / individuals build software / hardware in a single sprint of 24 hours. Hackathons have a theme, ranging from generic things like improving community to specific themes like solving a specific delivery problem the company faces. Some hackathons are closed, conducted only for the members of an organization, some are open to all. Some hackathon focus on profitability of an idea and implementation, teams winning sponsorships from investors, and some focus only on ideas and imagination of the participants. All in all, a hackathon is a developer’s Disneyland!

If you feel like you have ideas, but no time to develop them, try them out or run them by other people? Hackathon is the place for you to build your dream concept into reality. You are an amazing developer, you can punch in code and get things working in no time, hackathon is the place for you to show your skills. You like to interact with people, share ideas and learn how people feel about them, hackathon is the place for you to validate your product. You have a concept, but are looking for skilled brains to develop it, hackathon is the place for you to spot skills and recruit them. You are a nerd, introvert, who loves to code (the stereotypical software developer), walk right in, there are many like you in there. You are a night-owl who believes that sleep is for ‘cats’, you will fit right into a hackathon. Imagine a place which provides electricity, wifi (I have your attention now, don’t I?), food, seating space and all else you need, and leaves you undisturbed for 24 hours with the freedom to build your dream into reality, now that is a hackathon. (Put like this, it sounds better than Disneyland!)

Now let us say you wish to go to a hackathon and ‘make the world a better place’, you need to have a plan, like everything else. There are things that you should and should not do. There are things that you should and should not carry. I have over time built a list of items I carry to a hackathon, and like other ideas validated it against others during the hackathon. So what we have here is a list of items people in general carry, not every item will apply to you though. It is like a camping list, but for geeks.

Preparing for the Hackathon

  • Prepare for your idea: Think, elaborate it, plan it. This is also a test of your agility, all your skills in iterative, agile, development are going to be tested. Know that a 24 hour hackathon is 3 working days time on your hand, it is a lot, plan how to utilize it best.
  • Choose the right team: Long hackathons tend to be team games. Choose your team wisely. You should have compatible yet slightly overlapping, targeted skills, and equal passion. You probably do not need a marketing executive or the so called ‘product owners’ on your team unless the idea is from their domain. It is not a conference, you do not get a booth there. And passion, yes, you do not want your team members jumping into sleeping bags at the chime of 10 only to get up at 6 next morning.
  • Identify what open source projects can help you, know how to use them. Play around with them. May be send a pull request for missing features. But know what is going to help you get it done faster and plan for it.
  • Set up your machines. You do not want to be downloading a database server or creating cloud service provider accounts at the hackathon.
  • Set up productivity tools suitable for your idea. For example, get accustomed to using a clipboard manager; write and keep handy scripts to automate simple tasks like starting db, launching db shell, clearing tables, generators for various frameworks etc.

Packing for the Hackathon

  • Laptop: Of course! Humans are yet to build a computer one can use to develop software on in thin air, unless that is what your idea is for the hackathon.
  • Laptop charger: You will be surprised how many people forget this. Although your machine has juice for one work day, it is not enough for a hackathon. If you think of it, it is actually 3 work days there. And you don’t want to be making ‘connections’ by asking people for charging cables.
  • Phone & charger: You are going to interact with a whole lot of people, do carry your phone to note down numbers, not everyone brings the business cards, it is not a conference. Hackathon is for thinkers and doers not talkers, yet the excessive use of phone screens drains them, so do carry your chargers. Some hackathon venues do provide charging stations, you can check to confirm if there is one.
  • Peripheral devices: If you prefer to use an external mouse, drawing pad, a VR headset or whatever you need, carry them. Pack the devices you need to build your idea, like a raspberry-pi, a hovercraft kit or whatever. Keep pen-drives handy. You can check or request if the organizers provide an external screen, it will be too huge to carry anyway. It is all developers and geeks there, I would not blame you if you do not trust the security of the wifi there. In that case, carry your own portable hotspot.
  • Identity proof: You have registered online, but the organizers need to know who you are before you can enter.
  • Toiletries: Do I need to explain? Just don’t stink, you do not want to be remembered for that.
  • Medicines: If you are on medicines, do not forget to carry them. If you get acidity by staying up late, carry antacids. You get headaches, carry a mild pain killer. Have allergies? Carry an antiallergic. Afterall it is a competition, you want to be your best throughout.

Wearing for the Hackathon

  • Wear something casual and comfortable: If suits are your thing so be it, but remember you are going to be scratching your head over a lot of things in the next few sleepless hours, be comfortable. You do not want to be scratching other parts of your body due to uncomfortable clothes. You do not need to look pretty/handsome, make-ups and hair-sprays are not required, your personal comfort is.
  • If you are representing a startup, wear them on! Hackathons are good for creating awareness and hype.
  • It is okay to wear your lucky accessories, but limit it to that, avoid the temptation to wear superman under your clothes.

Hacking at the Hackathon

  • Not literally. Do not hack others’ devices, you can get banned from the premises or worse.
  • Divide your hours like they were days. 4 hours represent your half day of work in there. Have regular discussions.
  • Divide your tasks and identify interfaces where your tasks meet.
  • Use version control systems extensively, If it is a hardware product, keep taking photos from all angles. At hackathon, I prefer to change the way I commit; usually I commit often but push after cleaning. At a hackathon I commit and push extremely frequently, on every unit completion. It would not be an exaggeration to say that make the version control system your undo log. This also help prepare for an unfortunate event if your machine decides to take a nap while you are banging the keys.
  • Ban headphones on the team. Unless there is interaction, there is no team.
  • Plan your long breaks, like lunch, dinners and snacks to be in sync with your discussions.
  • Take frequent breaks apart from the discussions, individual or team breaks, but get up and walk around. I drink a lot of water, that forces me to take frequent breaks and helps avoid health issues as well. It is also a good idea in your day to day work.
  • Do not sit it through, walk around, jump around, interact, stay active and awake.
  • If you can, take naps in between. Just make sure you allow your partners to pour a water bottle on your head to wake you up, just in case. You do not want to miss all the fun by sleeping through.
  • There tend to be side events every few hours in long hackathons, participate in them, get to know people.
  • Meet people. You will find a whole lot of them are working on something amazing. You might end up meeting your next employer, co-founder, your living idol, or even your soulmate, if you are looking for that. You never know. All those coming to the hackathon tend to be there for their passion.
  • Have fun. Win or lose, unless you have fun at the event, it is pointless. Have what fun means to you. No one forces you to take part in any of the events or to talk to people, if you just wish to code, so be it. But enjoy the 24 hours, you do not get platforms like this every day.

After the Hackathon

  • Wrap up everything, pack the items and belongings, do not forget your chargers and peripherals.
  • Have one final meeting, plan out what you would like to do with the idea and the code / product built so far.
  • Divide the tasks for the future and decide timelines. If you leave here without a plan, and if you have not won the prize, it is highly likely that the idea will never be pursued further.
  • Know that what you are feeling, the mild body ache, sleepy red eyes, it is similar to a jet-lag and treat it as such, by sleeping only at your usual timing. Avoid taking an untimely nap in the day, set your routine back as soon as possible.
  • Write a blog. 😉

If there is something that should be added to the list to make it more usable, please suggest. See you at a hackathon some day.

Docker As Application Registry

Docker As Application Registry

Docker As Application Registry

Docker is great and solves a lot of problems with deployments. It taught VMs to share the resources, like how VMs taught hardware to share resources! Along with production, I have found that docker can work great as an application registry in a local development environment.
By applications I mean software that you install on your OS and launch them with shortcuts and they continue to live and retain state till you uninstall them; not exactly what containers are designed for but can work as. Something like snap or flatpak but with docker and for servers as well, not just for UI apps.

One advantage this has over using standard installers (like apt-get) is you are at complete liberty to start and stop the background processes, like mysql. If you installed mysql this way, you do not need to go and disable the autostart for it, it just does not matter! Similarly for your SonarQube server, you do not need to install it as a daemon, neither do you need to remember where you downloaded it to be able to restart it. Another advantage is most of such applications do have official docker images, it is the intended way to use them now!

One disadvantage of this method is that you always need to address them with their IP, you will not be able to bind them on host network then. But in my view, it is always better to have a dedicated IP, it emulates production scenario better and does not clutter your local machine ports.
So for apps what you need is containers to live long, be able to identify them with name, start and stop them easily and have a dedicated, static address to be able to reach to them. Most of these things are easy, except for a static IP. But once you create a network, you are set. That is it, it is that easy! Create a virtual network, and start your dockers with a name and static IP in that network. Simple!

To create a network:

docker network create -d bridge --subnet="172.30.0.0/24" --gateway="172.30.0.100" --ip-range="172.30.0.0/24" permanet

Now any app you need, just specify this network and a static IP of your choice; like this:

docker run --name mysql-server --network="permanet" --ip="172.30.0.1" -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql/mysql-server:5.7

More complex containers can be created like this:

docker run -it --name gocd-server --network="permanet" --ip="172.30.0.5" -v /yourhome/docker-volumes/gocd/godata:/godata -v /yourhome/docker-volumes/gocd/home:/home/go gocd/gocd-server

I keep a dedicated directory in my home for docker volumes, so I can back it up and use them as is when I change machines or OS. I also have a script where I add all the containers I need, so it is just a matter of copying the volume directory and running the script to create identical setups. Then even your .desktop files work as is!

Here is an interesting setup script for Jenkins (gist), it externalises all data directories from Jenkins, including the plugins and users and mounts your local m2 repository inside Jenkins so as to avoid downloading the libs again :

docker run -it --name jenkins --network="permanet" --ip="172.30.0.6" 
-v /yourhome/.m2:/var/jenkins_home/.m2
-v /yourhome/docker-volumes/jenkins/workspace:/var/jenkins_home/workspace
-v /yourhome/docker-volumes/jenkins/jobs:/var/jenkins_home/jobs
-v /yourhome/docker-volumes/jenkins/plugins:/var/jenkins_home/plugins
-v /yourhome/docker-volumes/jenkins/users:/var/jenkins_home/users
jenkins:2.32.3-alpine

You can create as many such networks as you wish for logical separation of groups of such apps; in my case this is the third network (172.30 for that reason), since first two were taken up by some compose scripts.

A list of few such containers I use: mysql (different versions), SonarQube, hystrix-dashboard, zipkin, swagger-ui, a redis-cluster for local use, gocd-server, Jenkins, portainer, postgres, pgadmin etc. I even have a couple Windows software running on wine in such containers, we shall talk about it some day.

Jetbrains messed up our license: Jetbrains still rocks!

Jetbrains messed up our license: Jetbrains still rocks!

Jetbrains messed up our license: Jetbrains still rocks!

It is story time today. We are a small startup, by small I mean like 4 people on the team and we established barely a couple months ago. It was time to start development and like all good Java developers depend on their IDE for their life, we did too. Too soon to go off-topic, but I wonder sometimes how large a program can I write without an IDE.

There is no better IDE for Java or Javascript than Intellij Idea from Jetbrains, and like all developers who know this, we went ahead and bought the Idea license. Lucky for us, being a startup, we were eligible for a 50% off offer. Jetbrains’s sales team was kind enough to approve it and we did get the license without much of a problem. My CTO asked me to get it set-up for myself, it was easy. There was a link in the email, I clicked, logged in and I got the license added in my Jetbrains account. It was almost smooth, except for a short to and fro on email to get the offer to reflect on the checkout page. All said and done, we loved the experience. Who would not love to get Idea at 50% off!??

So, done with the story? Why would I write it up if it was all so smooth? Read on..

A week went by without a problem and all of a sudden one morning my Idea closed on me complaining that I had no license! This was a shocker, we bought it, I had seen in it in my account and it was working for a week! I wondered if my CTO could have accidentally reallocated it, but of course he has better things to do than poke around in Jetbrains account configuration. I logged in to my account and turns out there was actually no license!

Since the CTO and I work in different time-zones, it was almost end of the day when we could chat and I could request him to check the matter. (A day saved by sublime-text.) Like I thought, he told me he had better things to do but we decided to have a look at the account anyway. Since it was a single link click for me to setup the account, he had not even created a Jetbrains account till then. We created the account and logged in, and were shocked to see some 31 licenses in our name. Something was certainly wrong.

Clicking through a couple of menus revealed that we were actually seeing the licenses/account of a different company! And our CTO, had become an admin for them. We had complete control over all the licenses, our as well as their licenses, and we could even decide who becomes an admin. We could remove their admin. While it was all fascinating (the devil!), we needed to know what happened to our license and we found it. This company’s admin had revoked my license and allocated to someone on their team! This company had a name similar to ours (Ours is one word, their was two), it was obvious that the person creating our license mistook us as them and instead of creating a new customer account amended theirs.

What do we do now? I contemplated our options and risks, contacting Jetbrains sale and support team and asking them to fix this was the best and the obvious option, but I had no idea how long it would take. What would AI do till then, being locked out of my own license, and when Idea keeps shutting down spontaneously. Can they even track it? Will they do it? Or would they ask us to buy a new one? There were a lot of unknowns, even when I knew Jetbrains would not leave us in the lurch. What if we wrote an email to this company’s admin and explain what has happened and request them to free up our license in good faith. But then they had their chance, they chose to to re-allocate a license they had not paid for to someone in their team, their admin would of course know this! They had their chance to see where the license came from, why it was allocated to someone outside, but they just revoked and used it, can we trust them to act on good faith? What if they just kicked our admin out, we would have no visibility into what was happening. Scary!

We quickly wrote an email to jetbrains, basically responded on the previous mail chain explaining this and requesting a resolution. Also attached a couple of screenshots showing what exactly had gone wrong, one showing the list of unrelated licenses and one showing our license being reallocated to someone else. Also tweated at Jetbrains to help us out, a good fellow working with Jetbrains responded and asked us to write to support email as well. We forwarded the email to support address and waited.

It was not long before we heard back from the sales team, some 4 hours or so. They quickly separated our company account, moved our license, assigned our admin and responded with a new link to claim license. We could now see a single un-allocated license in our account and allocate it. What they did great was that they also looped in this second company’s admin and wrote to them explaining the situation and offering a discount for one license. In my view, it was a good gesture at amending a mistake.

Mistakes happen, how you fix them is what decides if you retain your customers. And Jetbrains you certainly have retained us.

Creating a Coding Interview Problem

Creating a Coding Interview Problem

Creating a Coding Interview Problem

At work, I have been responsible for conducting the coding tests during the interview process, and have been doing that for over a few years now. Over the time I have made some mistakes, learnt some things and this post is a summary of what one should consider when creating a coding test.
Coding tests are generally the first round of interview; it is either online or on campus, depends, but a coding round is considered as the first gate a candidate has to cross. How you conduct it has some implications, for example, if you asked the candidate to submit code online, you may want to verify that the code was written by the candidate, they understand it and that they did it in the stipulated time. You can do that with an additional pair programming round on campus. If you conduct the test on-campus there is little room for such doubt. But then you need to ensure things like a dedicated desk or meeting room for the test, a machine, and food and coffee may be. But there are some considerations integral to coding tests themselves, these are the ones we shall be looking at today.
Before we begin, let me call out that there are those who believe that interview is not the best process to understand the suitability of a candidate, and I would not disagree. But I believe that if done right, coding tests are the best way to understand a person’s approach to problem solving, their expertise and command on programming. How to go from there is your choice.
We shall see the points I have learnt to consider when creating a test problem, we shall also discuss the the thoughts and reasoning behind them.

What to consider

Skills: What are you looking for

First is to identify the skill set you need, the practices that matter to you for the role you are looking to fill. People consider knowledge of language C as a basic expectation in computer science, but if a developer is not expected to work on C it is pointless to ask a question with C in mind. Similarly, the most attractive questions for an interviewer: questions on shortest path algorithms, sorting and searching algorithms seem pointless. If there was ever a need to use them, I would not expect any employee to implement them anyway, why expect them to implement in interview. All we need is that they understand how those algorithms work. In an interview, we need to better target the skills that matter to us. Yet, it is better to skip any frameworks that you might use, frameworks can be taught and learnt, teaching problem solving approaches takes longer. I am also of the opinion that the programming language used to solve the problem should be of the interviewee’s choice, but that is not always feasible, since we should have enough skill in that language to review the code they submit!

Prioritize: What is the value of each practice/skill to you

Prioritize the skills in these categories: ‘must have’, ‘should have’, ‘good to have’ and ‘cool to have’. Model the problem in a way that targets the ‘must have’ skills and probably touches on some ‘should have’ skills. Skip anything under ‘cool’ category. This is not to say that we should hire a person who can do only the job we require them to do at the time of hiring. It is to say that the person should be able to do at-least that much. You have rest of the interview process to assess the person’s ability to learn or apply creativity, coding test is the gating criteria.

Time: How much time can you allot for the test

This matters a little in an online test, but is a huge consideration in an on-campus test. For on campus tests we need to have provisions for machines, food and meeting room or desk. If it is an interview drive, we need plan for as many number of machines as there are candidates or divide the candidates in batches, with delay in batches equivalent to time for test. If it is an online test, how much time to provide the candidate to revert and if the candidate can get a weekend to revert are considerations as well. Either way, how much time do we demand from the candidate is a question and we need to decide on a problem that needs an estimated amount of time to solve.
It is also important to accommodate for the pressure the candidate may be under during the test. This matters even more for on-campus tests; it can put your time calculations off considerably. I try to make the problem statement very clear and as definitive as possible. Being explicit about what is expected and what is not helps, because an unclear question leaves room for random implementations or shortcuts where I least expect them. That makes it difficult to evaluate the solutions on equal grounds. On the other hand, unclear questions leave room for creativity, but I have seen people missing the point with these.

Branding: Showcase how you are

Interview is a window, window for the company into the candidate’s capabilities and window for candidate into company’s culture. Just as the company needs to like the candidate, the candidate needs to like the company. And the problem statement is the first impression of the company! It is important to make the problem ‘fun’ to work on. I prefer to use casual way to describe the problem and try to make it enjoyable to read and to solve. It should have the feel of a fun place. You should choose a style that best describes your organization. As a side note, nerd jokes or references to Hitchhiker’s Guide to the Galaxy or Star Wars do not always work. 😉

Creating the problem statement

Over the period I established a process for creating the problem statement, enabling a predictable estimation of time and solution. Over time you can choose to skip some of the steps, but to begin with here is the process: 
  1. Once the problem is identified, create the write-up describing it. Instructions, guidelines and rules about time, how to submit the code, languages allowed etc should be mentioned at the top.
  2. Once the problem statement is formulated, ask a peer to read and explain it to you. Ensure it means what you expect it to mean.
  3. Solve the problem. Record the time.
  4. Ask a peer to solve the problem, record the time taken. The peer here should be from the target experience range and skillset. Ask for their experience.
  5. Anything you learn during these discussions, or while solving the problem, convert them into instructions and add them to the top.
  6. Fix the scope of the problem to fit the time.
  7. Solve again, when you are in a different mindset and record the time.
This process, although tedious, sets the expectations right. Ponder on how good is your solution in that time and how acceptable are your mistakes to yourself. Consider that you already knew the problem and had (unknowingly/knowingly) designed a solution in your mind. That should help set expectations from an interviewee ‘under pressure, under time limit, likely less experienced than yourself, groomed in different culture with different practices than yours who was just presented the question.’ Sounds difficult, right? It is not to mean that the evaluation should be lenient though, it is only to identify what is acceptable.
While evaluating, one should remember that you are not looking for candidates who solve the question the same way as you do. In other words, you are not looking for candidates who think just like you. The different the better, it is always surprising how many different ways you find the problem solved.
That’s all from me, all the best for finding the right candidates!
Yet another packager for node

Yet another packager for node

Yet another packager for node

There are so many packaging systems for node already, or maybe not as many, so here I am presenting another way to package your applications into an self extracting executable that has no dependencies. Ah well, a few dependencies, like the processor architecture, and Linux operating system may be, but that is all.

What is it?

It is a modified shell script originally used to create self-extracting and installing applications for Linux platforms. What it does is, it creates a tarball which includes your code, the modules it depends on, the specific node binary it uses, and appends it to a script with the command to execute your code. It is essentially a binary merge of the files, the shell script and the tar.This is not something new, people have used such a system in the past to deliver applications for Linux, every time you see an obscenely large ‘.sh’ file (for being that, a shell file) that can install or execute an application without requiring any other files, know that this is the packaging system being used.This script is merely an adaptation of it for delivering node.js programs. And to give where credit is due, is pulled and compiled from a few sources.

What all can it do?

  1. I have been hoping you would ask that, it is interesting:
  2. Creates a single file that starts your code when executed.
  3. Does so without requiring even node or node_modules installed on the target system.
  4. No knowledge of any framework required, develop your code just as you normally would.
  5. Allows you to name the process it starts. Well, it at least helps you to do so.
  6. Allows you to have environment specific overrides for any configuration you might want.

What can it not do?

  1. It requires to be bundled for the target platform, but this is expected, is it not?
  2. Does not work well when if module has binary/native dependencies, for when things like node-gyp or build-essential come into picture.
  3. Cannot make you fly (but it can make you look smart!)

Where is it? How do I use it?

Here. It is a simple command. To package, run:
./selfXpackager.sh -s node-bin/launcher.sh -n selfExeSample -b node-bin/node -m mymodule/ -o dist/
And to run the package:
../dist/selfExeSample_launcher.sh
That easy. The repository also has a sample project to try it out.

Where should I use it?

Well, how can I comment on that, it would be for you to decide! But I can tell how we use it. The company I work for, is primarily a java shop. Our system is quite distributed, composed of many services (I dare not say microservices, it is easy to start flame wars these days) that talk to each other. But ever since we realized the power of node especially in quick new developments that we do, we have leveraged it. We have much code in the form of monitoring and mock servers, automation and code generation tools and fault injection systems built in node. These systems are delivered, they do their job and are removed when no longer required.This is where the script comes in, a no dependency delivery of a tool wherever we need it. Instead of requiring node installed on all servers, we bundle our tool with this script and deliver to the servers we need them on, when the job is done they disappear without a trace. Well almost without a trace, it’s not some stealth tool anyway.
Opinionless Comparison of Spring And Guice as DI frameworks

Opinionless Comparison of Spring And Guice as DI frameworks

Recently I had to delve into the play framework for a particular microservice at work. Now it is not exactly new, nor is Guice, nor DI, but coming from Spring world it was still a big shift in approach. There is a lot of documentation comparing Spring with Guice, stating which is better, why and how. In general these articles discuss specific points where these two frameworks differ in their approaches and which approach seems better to the author. I am not sure these articles really help someone trying to take a dip in the other framework. We know the differing opinions, as they are stated by the authors of the respective frameworks in their own documentation as well, another person (article’s author) reiterating it with an incomplete comparison of these frameworks does not sound helpful. What would work much better is a direct mapping of features, without author’s opinion (Didn’t this sound like an opinion). That should help someone getting into Spring from Guice world or vice a versa.


Now let me warn you, since these are different frameworks for the same purpose, DI (Dependency Injection), they exist for their differences. Hence, there cannot be one-to-one mapping of features/differences in these frameworks. What we can get instead is mapping of similar features and that is what we will have. If nothing else, the comparison below should help someone find the right documentation for what they are trying to do, instead of wondering what to look for.


Another point, we are here discussing Spring and Guice only on their dependency injection approaches and not as web frameworks, AOP, JPA abilities, their ecosystem or any other features they provide. That is for another time maybe, but not today.


Spring
Guice
Application level @Configuration
Extend AbstractModule, comes closest to that. It defines a part of your application; multiple modules can depend on each other in an application. (Unless your service is too small)
@ComponentScan
There is no classpath scanning in Guice. (keep reading…)
@Component
@Singleton with “bind() with/without .to()” in Module
@Scope(“”); singleton (DEFAULT), prototype, request, session, global-session
Default is unscoped, similar to prototype in Spring, @Singleton, @SessionScoped, @RequestScoped, custom.
eager/lazy differ for production and development.
@Autowired, @Inject
@Inject (from javax or from guice package)
@Qualifier(<name>)
@Qualifier / @Named, Names.named,

annotatedWith and @BindingAnnotation
@Bean
@Provides or implement Provider<T>
@Bean with @Autowired field in it
Explicit constructor binding: .toConstructor(A.class.getConstructor())
@Value
@Named with Names.bindProperties() in your module
Injecting static fields can be achieved with @Autowired on non-static setter method.
For static fields, use .requestStaticInjection() in your Module
ApplicationContext (BeanFactory to be precise)
Injector
@Autowired with context.getBean(Clazz, Object…)
@AssistedInject. Allows for using params, with injected beans to instantiate objects.
@Lookup
Provider<T> with FactoryProvider; FactoryModuleBuilder
@PostConstruct, @PreDestroy
No Support for lifecycle events. (extensions)


Let’s also see a few more points which would not fit well in a tabular form:
  • One can add more capabilities to Guice with plugins and there are a few actively maintained like Governator from Netflix. Spring can be extended using BeanPostProcessor or BeanFactoryPostProcessor in your application, but I was unable to find a plugin for extending Spring’s core DI abilities.
  • Unlike Spring, wiring in Guice (called binding) is plain Java, and so Guice has compile time verification of any wiring we do. Spring depends on metadata through annotations, which are not checked during compilation, does not have this feature and exceptions are at runtime.
  • Classpath scanning can be achieved in Guice by extending it. (Some plugins provide this, but governator for one has deprecated it.)
  • Lack of classpath scanning in Guice, most likely, considerably reduces the application startup time in comparison to Spring.
  • In Guice an Interface can declare the default implementation class (is it odd, Spring people?), @ImplementedBy annotation, which can be overridden by .bind() if found in a module. Similarly the interface can declare the configuration class which generates the instance: @ProvidedBy
  • I know I said we are not going to discuss any other abilities, but this one is a little interesting; Guice has built-in support for AOP, in Spring we need an additional dependency.
  • Not a difference, but a point to note: both frameworks have similar injection types, Constructor, method and field.


I have tried to be as opinionless as possible when writing the above piece; although there are a few things that I find important to note.
  • Guice is very much a non-magical (in the words of Guice authors) dependency injection framework, you can literally see DI happen, with the code that you write and can read.
  • Thankfully, Guice has no beans.. NO BEANS! How many beans do we have to remember and disambiguate before it is too much? Javabeans, Enterprise Javabeans, Spring Beans, Coffee Beans, Mr. Bean and I might still have missed a few others!
  • Guice still feels like java, you see, it does believe in extending classes, Spring nowadays seems to believe only in annotations, so much so that a few folks I asked around can’t even remember what ‘extends’ keyword stands for! 😉

So which one is better? Now, that was not the question we were hoping to answer!