Browsed by
Tag: docker

AWS Cloudwatch log scroller

AWS Cloudwatch log scroller

This is a quick tip about a quick solution to a regular small problem! AWS CloudWatch logs are great, but the search feature on those logs is not. The search only lists the lines that are exact match to your search term. Usually you would want to see a few lines above or below the match as well! Not all logs are single line, unless you spend efforts in ensuring that there are no newlines in the content being logged out. And so, every so often you have resort to ‘scrolling’ through the logs.

The logs load dynamically on scroll of the mouse wheel, and it is tedious (read ‘irritating’) to keep spinning the mouse wheel to scroll through those logs, especially the ECS container logs. So here is a little script which does that for you. Run this in your browser console and specify it the number of times it should run, and done. It does the scroll action, waits for the page to load (fixed) and scrolls again till it has scrolled for the times you specified.

You can also stop the scroll in between by calling clearTimeout on the timer.

Here it is:

Disclaimer: This is still not the best solution for long running tasks / huge log files, you are limited by the load times anyway. This only takes the need for manually scrolling through logs where you would have otherwise done so. For larger logs, it is probably best to export them.

CIDR Explained in Layman Terms and Decimal Numbers

CIDR Explained in Layman Terms and Decimal Numbers

If you work on cloud it is likely that you have used those numbers and slash that follow the IP addresses. The documentation points to something called CIDR. It is said to be super helpful, and awesome standard adopted by internet that extended life of IP4 . But have you tried searching ‘what is CIDR’? It is all jargon, all of it. There is hardly any lay-man friendly explanation of the term. Even Wikipedia has managed to find a complex way of explaining it. And yet it is something we use everyday, especially if you are working on cloud, containers or container orchestration frameworks.

We use it for defining networks when using docker. We use it when we specify services and networks in orchestration frameworks like swarm, compose, kubernetes, ECS or GKE. We use it when we specify ingress/egress rules in an AWS security group. We use it when we create a subnet in AWS EC2 specification, when we define VPCs, when we define clusters. Even single IP addresses (range size 1) at times are defined using CIDR notation.

I was wondering if CIDR could be explained without getting into binary number calculations, or more jargon of classes or routing, and found one.

CIDR is simply a way of specifying range of IP addresses. In cloud we mostly deal with IPV4 addresses and so let us see how we can think of CIDR in IPV4 context.

An IP address has 4 parts, joined together by dots. Each part can have 2 ^ 8 = 256 values, between 0 -255 both inclusive. In CIDR, we add a slash after the IP followed by a number between 1 to 32, both inclusive, these numbers are in fact a netmask specification. Now these 32 numbers can be divided in 4 groups of size 8, similar to the IP address (groups being 1-8, 9-16, 17-24 and 25-32). Each group has effect on the corresponding section of the IP address to generate a range. Like in the diagram below:

Now looking at the number in the group you can quickly tell what all IPs can come as part of the range. for example:

  1. 99.123.43.64/8 –> 99.0.0.0 to 99.255.255.255
  2. 99.123.43.64/16 –> 99.123.0.0 to 99.123.255.255
  3. 99.123.43.64/24 –> 99.123.43.0 to 99.123.43.255
  4. 99.123.43.64/32 –> 99.123.43.64!

The size of the range decreases as this number goes up, 1 being widest and 32 being strictest. Simple enough?

Now on to a little more complex part, what about number that are not multiples of 8? You can certainly define something like: 99.123.43.64/18 or 99.123.43.64/5 or 99.123.43.64/27 what would that mean? We have seen that each group of netmask governs IP values (0 – 255) in its group (and the groups that come after it). What if we divided these groups further? Larger groups were with multiples of 8, we will now divide the 255 numbers in 8 different ways, using 8 powers of 2, using a little 10th grade maths to do this:

  1. 7th power of 2, i.e. 128 creates two sub-groups: 0 – 127 and 128 – 255
  2. 6th power of 2 i.e. 64 creates four sub-groups: 0 – 63, 64 – 127, 128 – 191, 192 – 255
  3. 5th power of 2 i.e. 32 creates eight sub-groups: 0 – 31, 32 – 63, 64 – 95, 96 – 127, 128 – 159, 160 – 191, 192 – 223, 224 – 255
  4. 4th power of 2 i.e. 16 creates sixteen sub-groups: 0 – 15, 16 – 31, …. 240 – 255
  5. 3rd power of 2 i.e. 8 creates thirty two sub-groups: 0 – 7, 8 – 16, …. 240 – 247, 248 – 255
  6. 2nd power of 2 i.e. 4 creates sixty four sub-groups: 0 – 3, 4 – 7, …. 248 – 251, 252 – 255
  7. oneth power of 2 i.e. 2 creates one hundred and twenty eight sub-groups: 0 – 1, 2 – 3, 4- 5, … 252 – 253, 254 – 255
  8. zeroth power of 2 i.e. 1 creates two hundred and fifty six sub-groups: 0, 1, 2, 3, 4, 5, …. 253, 254, 255

With me so far? Now let us see how we can understand the meaning of intermediate numbers:

  1. Step 1: Identify the larger group your netmask belongs to using the diagram above, call it major group. Ex: /18 belongs to group 3 (17-24) and /30 belongs to group 4 (25-32)
  2. Step 2: Deduct your netmask number from the higher bound of the group. Ex: with /18, you get 24-18 = 6 and if /30 you get 32 – 30 = 2; this is your power of 2 (say n).
  3. Step 3: Now you can calculate the number of IPs that fall in this range by nth power of 2. Meaning when you specify /18, you have 2^6 = 64 and for /30 you have 2 ^ 2 = 4 possible values in the range.
  4. Now its just a job of identifying which sub-group your number in the place of major group fits in and you have the exact range of IPs that fit your CIDR. Ex: /18, means group 3 and sub-group with power 6. So the number in 3rd group in an ip like 99.123.43.64 is 43 and it fits in 1st sub-group.

Let us look at some examples:

  1. 99.123.43.64/27
    1. 27 is in group 4 (25 – 32)
    2. There are 32 – 27 = 5 and 2 ^ 5 = 32 addresses and we pick from the 5th power sub-group.
    3. and the 4th group number, 64 falls in 64 – 95 subgroup. so the range is: 99.123.43.64 to 99.123.43.95
  2. 99.123.43.64/30
    1. 30 is in group 4 (25 – 32)
    2. The power would be: 32 – 30 = 2 and there would be 2 ^ 2 = 4 addresses.
    3. and the range would be: 99.123.43.64 to  99.123.43.67
  3. 99.123.43.64/18
    1. 19 is in major group 3 (17 – 24).
    2. The power would be: 24 – 18 = 6 and there will be 2 ^ 6 = 64 possible values in group 3. Applying all possible values in group 4 for each in group 3, gets us: 2 ^ 6 * 2 ^ 8 = 16384 addresses
    3. and the range would be: 99.123.0.0 to 99.123.63.255
  4. 99.123.43.64/13
    1. 13 is in major group 2 (9 – 16)
    2. The power would be: 16 – 13 = 3 and there will be 2 ^ 3 = 8 possible values in group 2. Applying all possible values from group 4 and 3, we get 2 ^ 3 * 2 ^ 8 * 2 ^ 8 = 524288 addresses
    3. And the range would be: 99.120.0.0 to 99.127.255.255

I hope that clarifies it a bit!

[TechNggets] Episode 1: Intro to Containers and Self

[TechNggets] Episode 1: Intro to Containers and Self

Here is the first episode of “Tech Nuggets and Thoughts”

In this episode we talk about containers, what they are, how they work, what docker is and when / when not to use docker.

 

To get updates, you can subscribe to the podcast on: Apple iTunesplayer.fmRSS feed. If you have any suggestions, thoughts or recommendations, please feel free to comment below. You can also reach me on podcast’s twitter handle @TechNggets or my personal account @nikhilwanpal.

(If the fancy player above does not work, try the bare bones player below.)

 

Docker As Application Registry

Docker As Application Registry

Docker As Application Registry

Docker is great and solves a lot of problems with deployments. It taught VMs to share the resources, like how VMs taught hardware to share resources! Along with production, I have found that docker can work great as an application registry in a local development environment.
By applications I mean software that you install on your OS and launch them with shortcuts and they continue to live and retain state till you uninstall them; not exactly what containers are designed for but can work as. Something like snap or flatpak but with docker and for servers as well, not just for UI apps.

One advantage this has over using standard installers (like apt-get) is you are at complete liberty to start and stop the background processes, like mysql. If you installed mysql this way, you do not need to go and disable the autostart for it, it just does not matter! Similarly for your SonarQube server, you do not need to install it as a daemon, neither do you need to remember where you downloaded it to be able to restart it. Another advantage is most of such applications do have official docker images, it is the intended way to use them now!

One disadvantage of this method is that you always need to address them with their IP, you will not be able to bind them on host network then. But in my view, it is always better to have a dedicated IP, it emulates production scenario better and does not clutter your local machine ports.
So for apps what you need is containers to live long, be able to identify them with name, start and stop them easily and have a dedicated, static address to be able to reach to them. Most of these things are easy, except for a static IP. But once you create a network, you are set. That is it, it is that easy! Create a virtual network, and start your dockers with a name and static IP in that network. Simple!

To create a network:

docker network create -d bridge --subnet="172.30.0.0/24" --gateway="172.30.0.100" --ip-range="172.30.0.0/24" permanet

Now any app you need, just specify this network and a static IP of your choice; like this:

docker run --name mysql-server --network="permanet" --ip="172.30.0.1" -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql/mysql-server:5.7

More complex containers can be created like this:

docker run -it --name gocd-server --network="permanet" --ip="172.30.0.5" -v /yourhome/docker-volumes/gocd/godata:/godata -v /yourhome/docker-volumes/gocd/home:/home/go gocd/gocd-server

I keep a dedicated directory in my home for docker volumes, so I can back it up and use them as is when I change machines or OS. I also have a script where I add all the containers I need, so it is just a matter of copying the volume directory and running the script to create identical setups. Then even your .desktop files work as is!

Here is an interesting setup script for Jenkins (gist), it externalises all data directories from Jenkins, including the plugins and users and mounts your local m2 repository inside Jenkins so as to avoid downloading the libs again :

docker run -it --name jenkins --network="permanet" --ip="172.30.0.6" 
-v /yourhome/.m2:/var/jenkins_home/.m2
-v /yourhome/docker-volumes/jenkins/workspace:/var/jenkins_home/workspace
-v /yourhome/docker-volumes/jenkins/jobs:/var/jenkins_home/jobs
-v /yourhome/docker-volumes/jenkins/plugins:/var/jenkins_home/plugins
-v /yourhome/docker-volumes/jenkins/users:/var/jenkins_home/users
jenkins:2.32.3-alpine

You can create as many such networks as you wish for logical separation of groups of such apps; in my case this is the third network (172.30 for that reason), since first two were taken up by some compose scripts.

A list of few such containers I use: mysql (different versions), SonarQube, hystrix-dashboard, zipkin, swagger-ui, a redis-cluster for local use, gocd-server, Jenkins, portainer, postgres, pgadmin etc. I even have a couple Windows software running on wine in such containers, we shall talk about it some day.

Using Docker and a Private Registry with VPN On Windows

Using Docker and a Private Registry with VPN On Windows

Wasn’t that a very specific title? Docker has a very good documentation and reading that alone is enough for most of the straightforward tasks we might want to do. But as always some practical tasks are not straightforward, hence this blog. What we are going to see here today is how to setup docker toolbox on a Windows machine, make it work even when VPN is connected, make it talk to a private, insecure docker registry (that is why VPN) and configure it so it can run docker compose and see how we can set this config as a one-time activity. That’s quite a mouthful, but yes this is what we are going to do. All ready? Let us begin then.

Install Docker Toolbox

Go and download the docker toolbox and install it. That should create a shortcut called “Docker Quickstart Terminal”. Run it. That should show you an error about virtualization.

Enable Virtualization

Restart your machine, enter the BIOS settings and enable virtualization. It may be under advanced settings. On this Laptop, it is under the advanced settings -> device configurations and is named as: “Virtualization Technology (VTx)”. Whatever be the name, enable it.
Docker requires a Linux kernel, and since Windows machines lack it (of course!), docker toolbox runs a lightweight Linux distro called boot2docker in a virtualbox, hence the virtualization setting.

A Handy Tip

This tutorial will require you to copy and paste quite some shell commands, it is better we make that easy. Exit the quickstart terminal. Right click the shortcut, click properties -> options and enable ‘Quick Edit’ mode and save. It might ask for permission. This should now enable paste just by right clicking the mouse, to copy just select the text with mouse. While we are at that, also consider increasing the buffer and window size to suite your taste.

Start Up the VM

Make sure you are not connected to VPN and use the Quickstart Terminal shortcut again, this time it should proceed to validate if the boot2docker image is latest, or it shall pull the latest image, then it shall create a VM, get an IP, setup some ssh keys and finally the whale should appear with a terminal. Run the following commands to get a hang of docker running on windows:
docker -v
docker version
then docker run hello-world
docker images
docker ps -a
(And do read the output of hello-world, it describes how docker works). 

The Disappointment

Feeling happy? Now for a little disappointment, connect VPN and try again. Errors errors everywhere. Disconnect VPN. What happened: Docker is running in a virtualbox on your machine, which gets an IP in local range (normally: 192.168.99.100), and you are talking to it over ssh. Once VPN is up, it sets the new routes and sends the 192.168.* range traffic out over VPN and your commands never reach your VM running docker. The most popular solution to this is setting a port forwarding and is documented on many blogs/githubissues. Let’s just do that.

A new Beginning

Ensure you are not on VPN and remove the default VM, not necessary, but reduces the confusion. So in the quickstart terminal:
docker-machine rm default

And confirm. We are now going to create a new VM, let us call it ‘custom’. So type in:

docker-machine create -d virtualbox custom
eval "$(docker-machine env custom)"

It might take a couple of minutes, it is almost the same process as the first time. What we did is created a VM named custom and setup the environment to talk to this VM instead of the default. Mark this step, cause if anything goes wrong in the following steps, this is the one you should get back to to start over. Just be sure to use a new name, docker currently does not allow reusing names for VMs, so next time you may not be able to create a VM called custom. A new name should work just fine.

Battling With VPN

Now we shall create the a port forwarding on the virtual machine, binding the default docker port (2376) on localhost/127.0.0.1 to forward to this VM, whatever the ip of it.
docker-machine stop custom
/c/Program Files/Oracle/VirtualBox/VBoxManage.exe modifyvm "custom" --natpf1 "docker,tcp,,2376,,2376"
docker-machine start custom
docker ps -a
If you changed the location of virtualbox installation, please use appropriate path to vboxmanage. Assuming it was successful, last command should show you a table with all containers. You can use UI to do that as well: Open VirtualBox, stop the VM, open settings -> network -> NAT adapter -> advanced -> Port forwarding. Click add rule and use the same values as above (comma separates columns). If the command was successful, you should see the rule listed at the same location. Also, this is the place to add an entry if you need any port exposed from a docker container and use it with VPN enabled; for example your application’s tomcat port.
We are not done yet, a few more commands:
export DOCKER_HOST="tcp://localhost:2376"
export DOCKER_TLS_VERIFY="0"
alias docker="docker --tlsverify=false"
Kudos to this smart guy for that alias. In other posts, you might find IP of the VM (which does not work), public IP of your machine, or even loopback IP (127.0.0.1) being used, which might work but I would advise against that. Use ‘localhost’ instead; this and the TLS setting has to do with running docker-compose.
 
Now enable VPN and enjoy docker. This is where your journey ends if you are not using a private registry; but if you are, then continue.

Configuring Private Insecure Registry

Ensure that VPN is down, and ssh into the docker-machine. We want to enable it to talk to an insecure registry. A private docker registry does not need a name, but docker images in a non-docker-hub registry require that they be tagged with the URL of the registry prefixed to the usual repository name. They say it is for transparency, helps in identifying where the image originates from. Hence, it would be advisable to have a host-name even if your registry is private and has a static IP. That way even if you change the IP of the registry for whatever reason, you do not have to update all images/tags/compose ymls, shell scripts and whatever else is using them. Let us say our registry is hosted at: dockerregistry.example.com, on port 5000 and this being insecure, of course, is accessible only over VPN.
This step is intentionally manual, to avoid risks of breaking something else:
docker-machine ssh custom
sudo vi /var/lib/boot2docker/profile
In the EXTRA_ARGS, before the closing quote, add this line: --insecure-registry=dockerregistry.example.com:5000 
(I would ensure a blank line before the quote, as there already was) Save the file and exit vi (:wq). We now need to restart the docker daemon for changes to take effect:

 

sudo /etc/init.d/docker stop
Ensure service is down:  sudo /etc/init.d/docker status
sudo /etc/init.d/docker start
Ensure service is up: sudo /etc/init.d/docker status 
Exit the VM by typing exit in terminal. (BTW, there is restart command too)

Using the registry

Now let us try pushing and pulling from this registry. In the quickstart terminal: 
docker tag hello-world dockerregistry.example.com:5000/hello-world
docker push dockerregistry.example.com:5000/hello-world
docker rmi dockerregistry.example.com:5000/hello-world
docker run dockerregistry.example.com:5000/hello-world
What we did is tagged an image with the registry, pushed it to the private registry, removed the local copy and run the image by pulling from this registry.

Docker Compose

Next step is to get docker compose up and running with this setup. Actually, we are already ready, everything that we need to run docker-compose is taken care of in the previous steps. Most importantly docker-host configuration. You see, the TLS certs allow only for docker-machine IP and localhost to be used even when we disable verification, but we have already taken that into account and we have already configured our private registry. All set. Just connect VPN, navigate to your directory with docker-compose.yml file and hit: docker-compose up. You should see the images in compose file getting pulled and executed. 

Starting the quickstart terminal second time

When you restart the quickstart terminal you might find that it recreates the ‘default’ VM and configures the environment to use it. That is okay, it does not bother us. But what does bother us is that none of the docker commands are working with VPN again. Please keep reading..

Consecutive starts of quickstart terminal

Well, we have to reconfigure the terminal every time to use our VM of choice. Here is how to do it:
Always make sure that you start the terminal when VPN is down. Starting with VPN up has never worked for me; and then run these commands:
eval "$(docker-machine env custom)"
export DOCKER_HOST="tcp://localhost:2376"
export DOCKER_TLS_VERIFY="0"
alias docker="docker --tlsverify=false"
Yes, every time you start the terminal. There is a way to avoid this, read on. 

One Time Setup: For The Brave Among Us

From this point on, you are entering undocumented territory and are on your own. If something breaks, do not come looking for me. 🙂 And before making any modifications, take a backup.
If you notice, the shortcut points to a shell script called ‘start.sh’. We are going to modify this script to auto-configure our environment every time it is called. Navigate to docker installation directory (directory that quickstart shortcut is pointing to) and open the start.sh (After creating a backup) file in a text editor.
Change 1: On line number 10 which looks like: VM=${DOCKER_MACHINE_NAME-default}
change that line to: VM=custom. Custom here is the name of our VM. This saves you from typing the eval line every time.
Change 2: On line 66/67, in “Setting Env” step, after the existing eval command add the following lines:
eval "DOCKER_HOST="tcp://localhost:2376""
eval "DOCKER_TLS_VERIFY="0""
eval "alias docker=docker --tlsverify=false"
These handle rest of the config. That is all, save and exit the file and we are ready to roll. This may break when an update the docker toolbox is installed which overwrites the file, may not work if the script changes in future, may break things I am not aware of, hence only for the brave. Besides, I do not use a Windows machine daily, so you guys would be first to know if it starts breaking ;). Let me know and we will figure it out.