I have been making some good progress on my second run through of The Docker Book. I am definitely learning different things this time around. I have just completed the chapter on using Docker to make the testing of a web page easier – spinning up containers with different web technologies to test a simple page. I am now running through the chapter on doing something similar with a web application.

Confession time, I lost a little motivation over the past couple of weeks.  Not sure why but possibly a bit of tiredness, work being tough and everyone going back to the school routine.  Feel like my mojo has come back a bit in recent days though.

Despite my fun with Docker, more than one person I know has been hinting that Kubernetes has been “winning the war”.  In general, I think it would be a good idea to get stuck into a few specific technology areas next.  For a while I have been planning to run through Mongodb tutorials and my Git knowledge is not great so that needs some TLC too.  I will add Kubernetes to the list but there does not seem to be as many decent books and tutorials as for Docker.

Finally, I have produced the following Dockerfile whilst running through basic Docker commands.

# comment
# practising Docker commands
FROM ubuntu:16.04
MAINTAINER richardx14 "richard@blueharvest.co.uk"
#RUN apt-get update; apt-get install -y nginx
#RUN echo 'container!' > /var/www/html/index.html
#EXPOSE 80
#
# CMD when container has been launched. If you use a command after docker run, it overrides this
CMD ["/bin/bash"]
#
# ENTRYPOINT - any params used after docker run get passed as params to the ENTRYPOINT. Similar to CMD
# WORKDIR - cd
WORKDIR /var/log
#
# ENV env variables. Persisted into containers that are launched from image
ENV RICHARD richard
#
# USER account, UID, group that containers run from the image will be run by
#
# VOLUME - adds volumes to containers created from image. Allows access to the containers volumes
VOLUME ["/var/log"]
#
# ADD adds from build env into image
ADD testaddfile /var/log/testaddfile
#
# COPY add without without decompression. Will create missing directories
COPY testaddfile /var/log/copyoftestaddfile
COPY testaddfile /test/add/file/testaddfile
#
# LABEL - add metadata
LABEL name="Richard's test label"
#
# ARG - pass build time variables, can set default. use --build-arg build=buildnumber
ARG build
ARG builduser=user
#
# SHELL can override default shell
#
# HEALTHCHECK
HEALTHCHECK --interval=60s --timeout=1m --retries=5 CMD curl http://www.google.co.uk || exit 1
#
# ONBUILD triggered when image is used as the basis for another image
ONBUILD LABEL name="Richard's test label ONBUILD"

I have been adding Riemann alerts for high CPU/disk usage/memory and encountered two issues that need fixing.  That’s the downside.  The upside is my Clojure is improving.

Anyway, error 1:

ERROR [2017-09-16 17:09:59,617] main - riemann.bin - Couldn't start
clojure.lang.Compiler$CompilerException: java.lang.RuntimeException: Invalid token: /percent_bytes-used, compiling:(/etc/riemann/riemann.config:58:52)

No idea what is going on there.  Can be hard to google this stuff.  Might be a typo somewhere.

And error 2:

WARN [2017-09-16 17:56:17,775] defaultEventExecutorGroup-2-1 - riemann.config - riemann.email$mailer$make_stream__9273$stream__9275@552dcd79 threw
com.sun.mail.util.MailConnectException: Couldn't connect to host, port: smtp.gmail.com:587, 25; timeout -1

Wonder what is going on there?  Same stuff works on other two instances.   Another argument for Dockerising the whole thing.  Did wonder whether this due to one of my nodes spamming gmail.  Tomorrow’s problem.  Made some progress but now time for a beer.

I have decided to blog about some of the books I have been reading.   Reading is a vital part of my mission.

I am not going to “review” books, I am going to comment on them and how they have related to my education.

I am about to finish Release It! – author’s exclamation mark, not mine.  When the same book is referenced in a few other books you have been reading, it is probably worth a look.  I think that maybe I have tackled this one a bit too early.  It definitely taught me a few things, but it went into some technical depths that I am equipped to deal with yet.  It is also one of the few tech books that has made me laugh.  More than once.  It has helped me make the case for failing fast, timeouts, bulkheads and circuit breakers on a current assignment too.

Next up, I may continue with Site Reliability Engineering.  Other options are re-reading The DevOps Handbook.  I finished it in April but given it was such an easy read, I am going to try it again to see if it offers any insights I missed last time around.  I am also considering re-reading The Docker Book for similar reasons.

My full disks and Riemann logging issues continued over the past few days but appear to have been calmed.  Sadly I am not sure why.  I have a couple of theories though.

Firstly, after running through section 6.2 of The Art of Monitoring (checking processes are running), I pasted in new riemann.config files.  Not 100% sure but wonder if that has corrected a previous error – all the more reason for automation/Puppet/Docker eh?

Which brings me onto theory two.  I wonder if I stopped midway between sections and needed do further work to stop this happening.  This has happened to me before with The Docker Book when I exposed the Docker API publicly.   That’s a subject for another blog.

Of course, I may not have fixed this issue at all.  If I have, I would like to know what the fix was.  Chapters coming up will graph disk usage I believe.

On another note, it has occurred to me that there is probably huge value in revisiting the books I have gone through recently now that I know more.  That kind of makes my heart sink given my mission’s target date.  I have a continuing sense that I am learning different things to what the books intend.  Still learning in this space is surely going to be useful.

Finally, this pulled up outside my house this week.  Someone is trying to tell me something.  Insert your own Unikernel joke here.

Over the past two weeks I have become a bit like Richard Burton in  The Medusa Touch.  In this film he plays a character who has visions of disasters before they happen.

It seems that I only have to read about how unwise it is to share a database between customer and reporting traffic in  Sam Newman’s Microservices  before a slow running reporting query creates issues for customers.

On another occasion, I read  about circuit breakers and fail-fast timeouts in Release It! and almost immediately afterwards hit an issue that would have been avoidable if circuit breakers and fail-fast timeouts were in place.

And then, shortly after listening to three principles of CI on this podcast (whilst dog-walking, naturally), I run into issues with devs checking in code whilst the build pipeline is down.

For the time being, my team have asked me to stop reading about things that can go wrong or at least warn them in advance.

Realising that I am learning a lot about Unix and infra, but could learn this without ever learning anything about DevOps.  I can see where DevOps could make my life easier but the books I am following remove automation, duplication in order to explain config and the like.

Anyway my Riemann Mission Control is still in a state even when I free up disk space.  In today’s lesson I was setting up the collectd write-riemann plugin and had errors as follows on my problem-child host:

/etc/collectd.d$ sudo service collectd start
 * Starting statistics collection and monitoring daemon 
collectd ERROR: lt_dlopen ("/usr/lib/collectd/write_riemann.
so") failed: file not found. The most common cause for this problem is missing dependencies. Use ldd(1) to check the dependencies of the plugin / shared object.

This lead to my first contact with the ldd command which revealed:

libprotobuf-c.so.0 => not found

Finally with a bit of research the below fixed my issue:

sudo apt-get install protobuf-c-compiler protobuf-compiler libprotobuf-c0 libprotobuf-c0-dev

Having configured this plugin on four ubuntu hosts, I wonder why this dependency was missing on only one.  Two theories.  One – whatever is busting my disk space may be related to the absence of a working dependency.  Two – I may have missed a step on one of the four hosts.

Either way, if had configuration management tools or containerised, automated builds I suspect this error may not have occurred at all or at the very least have been fixable before building four hosts using the same image/container/scripts.

One thought I had was that it would be great to rebuild riemannmc.  Of course this would take a while to retrace my steps unless I had a Docker image to help me out (for example).

The other issue I had today is that collectd on my Red Hat hosts is not logging.  One for tomorrow.

Dog-walk learning

Quiet-ish couple of days on the mission front.  Rugby and work filling a lot of time.  However, in recent months I have learnt to regain time lost on dog-walking by listening to the following wonderful podcasts:

• http://devopscafe.org/
• https://aws.amazon.com/podcasts/aws-podcast/
• https://itunes.apple.com/us/podcast/aws-re-invent-2016/id1189917448

All brilliant in their own ways.  My introduction to James Turnbull and his books came from the DevOps cafe.  The Kelsey Hightower episode was a classic too.  The reminders I add to my phone to look at post-walk grows every time,

If anyone can recommend any more podcasts, please let me know.

Riemann mystery continues

Not had time to look at this in any detail, but issue remains.  I clear out the huge files and get an email the following day when the next log files grows to consume all of my space.

It appears the issue occurs, in Art Of Monitoring speak, on my Riemann Mission Control host.  So maybe other Riemann hosts are spamming it.  We will see.   The other two Riemann hosts and all Carbon hosts are performing well.

LCD Soundsystem are back

Finally, it is always a good week when this lot put out new music.  Not related to my mission but I am sure it will soundtrack much of my studying in the coming weeks.

I am a big fan of James Turnbull (https://jamesturnbull.net/) mainly because his Docker and Art of Monitoring books have been incredibly relevant to one of my day jobs.  I really knew nothing about Docker until I went through his book step by step.  It has since occurred to me that since I am starting from such a low base, many of the benefits of Docker over non-container technologies are probably lost on me.

That being said, I am about a third of the way through The Art of Monitoring (https://artofmonitoring.com/) and having launched six AWS EC2 instances and then configuring each with Riemann, Grafana, Carbon and Collectd, I think I get it better than I did before.  In fact I have added being able to containerise these tools to my learning and to do list – which is getting longer every day (and I may include it on this blog at some point).

In fact I have added all of the James Turnbull books to my to do list as  I see them as being critical to achieving my mission.

In today’s updates, I have been getting emails like this:

This is quite exciting.  Turns out Riemann logging has been going barmy on one of my six EC2 instances:

No idea why yet but has further added to my to do list:

• Why is this happening and why on only one host?
• How do I set up a log rotation policy (although it would have to be severe to control this scenario)?
• Why are the email notifications bouncing with “address not found”.  (I forgot to mention that.)

Back soon.

I have a mission.  I currently work in technology.  You will know my current role as one of the following:  Delivery Manager, Project Manager, Team Lead, even Scrum Master perhaps.

I had an epiphany recently when a team member horizontally scaled web-servers in our production environment without any downtime.  I was even more impressed when same team member downscaled the web-servers later, again without downtime.

Having worked on projects that suffered lengthy lead times when new infra was needed, I was blown away.

This started a chain of events that have lead to this mission:

I am going to try and become a professional DevOps practitioner with two years.

Professional for two reasons.  One – I would rather work in this field than my current one and two – I want to get to a point where I am strong enough to get a paid DevOps role.  The latter is a good measure of my success.

My clock started ticking in April 2017, so my target is April 2019.

A few final comments for now.  Firstly, I am coming for a very low base.  Only last year someone had to explain what an “AMI” was to me.  I barely knew any Unix commands when I started.  I thought “Docker” was someone who working in a harbour (which I guess is also true).

Secondly, I am intending this to be akin to a diary to keep track of my progress and document notes that I will find useful to look back on.  I am late starting this, as I wanted to get the blog going, I have not been overly bothered about look and feel of this page.  Expect to see some iterations through WordPress themes.

Thirdly, this site runs on AWS LightSail for two reasons.  I wanted to try it out and not spend too long setting something up.  It is also cheap.

Finally, the name of the site.  You will not need to be too much of a geek to know where it comes from.  I could create a tenuous link to DevOps to the name but in truth it is a domain name I have had for a while.  I wanted to put it to work.