DevOpsDays Berlin 2017

It's a good tradition to attend DevOpsDays Berlin. 2 days with many talks, ignite talks, hands-on workshops and open space sessions. But this year I was a bit disappointed: less attendees than the years before, and also less interesting talks - at least to me ... I know, this is my personal point of view ... others might see it differently.

The following were the top talks to me:

• Long-Lived Devops And The Pit Of Success by Jason Diller. Jason talked about the evolution of teams and technologies dealing with ongoing constant changes. He introduced how to measure the bus factor to gain visibility on orphaned technologies in business. My favorite quote: "Freedom until it hurts, standards until it works."
• It's Not Continuous Delivery If You Can't Deploy Right Now by Ken Mugrage. He mentioned integration of performance & security testing within your build pipeline, canary deployments, feature toggles etc. among others. One of the points - every commit should be pushed to master - was quite controversial. But in the discussion afterwards, Ken put this into another perspective: merging your feature branch to master in short intervals is ok, too.
• Encouraging A Culture Of Learning Across Your Organisation by Jenny Duckett. Jenny presented her motivation of preparing and running a security workshop although not being a security expert. She also showed how this has changed the culture within her organsiation.
• In his ignite talk Ken Mugrage showed value stream mapping by example with getting a teenager doing the chores - absolutely entertaining.

DevOpsDays Berlin 2016

Aah, Berlin again, DevOpsDays again - love it!

Here are some notes:

• Day 1:
• Looking back at 7 years of DevOpsDays, a keynote by Kris Buytaert.
  Kris showed a compact review from when Patrick Debois and he initiated the DevOps movement in 2009.
• Infrastructure as a Microservice by Konrad Hosemann.
  It was a very good talk about their journey from a monolithic architecture run in a data center to microservices running in a public cloud. They implemented some infrastructure services and used several distinct accounts for different business units like search and data science. The advantages they recognized were benefits from external innovation, autonomous teams with a "you build it, you run it" attitude, less to no side effects, cost transparency incl. cost awareness. On the other hand, they were facing problems like clarifying the ownership of the infrastructure services, distributed updates across several accounts, avoiding fragmentation and gaining a unified view of their landscape. Konrad also showed interesting numbers: The Graphite setup consists of four 24 core machines with 72 GB RAM and 360 GB HDD each and handles 1.5 million metric updates per second; the ELK landscape utilizes ten 20 core machines with 128 GB RAM, 1.5 TB SSDs and 15 TB HDD each handling 150,000 write requests per second.
• Communities of practice in large scale DevOps transformations by Martin Thalmann.
  A talk about the transformation towards a DevOps culture by establishing communities of practice in an organisation of about 4,300 IT people (give enough room/time to the members ... and provide beer and sausages). They also founded a local meetup group to open up the exchange with external contributors. Based on the number of questions after Martins talk, this seems to be a topic a lot of people deal and struggle with.
• Talking to People: The forgotten DevOps tool by Peter Varhol.
  Instead of (over-)using electronic communication (e.g. email, IM) we should communicate more face to face, as we build trust and solve problems more quickly. Everyone, but especially management should lead by example, e.g. by putting the phone away during meetings. A regular "beer thirty" could help to establish communication and trust.
• Ignites:
• The 20 Shades of Grey by Michael Hüttermann.
  Michael showed 20 ways to make your project and DevOps efforts fail, e.g. DevOps teams as an anti-pattern.
• Git and Github Tricks by Michal Bryxí.
  Things I didn't know before:
  - Ignore whitespace changes in diff: git diff -w -b
  - Better diffs by leveraging heuristics: git diff --compaction-heuristic
• Come to Germany, Pepe! by Anthony Stanton.
  Anthony presented his development and learnings from a classical IT department employee to a DevOps.
• DevOps? That's not my job! by Nathen Harvey.
  An entertaining ignite about the walls between dev and ops motivated by this blog post about serverless computing. Here is the longer version of that ignite as a full-length talk.
• Felix Frank shortly introduced the motivation behind mgmt.
• Hands-On Workshops (in parallel to Open Spaces):
• Next Generation Config Mgmt by James Shubin.
• How to read code and understand bugs by David Schmitt.
• Habitat Workshop by Eric Maxwell.
• I skipped the workshops, but joined the following Open Spaces - the resulting schedule is available here, and someone had the great idea to add further information directly to each card:
• Secret Management
• Chef Vault
• HashiCorp Vault
• AWS Key Management Service
• Confidant
• hiera-eyaml-gpg
• Audit breaches using auditd or go-audit.
• Use Teleport to control SSH access.
• Monitoring needs more LEDs
• Panels, LEDs, Arduino, Raspberry Pi etc.: here
• Scanning containers for CVEs
• JFrog Xray
• Clair
• Vuls
• Overview of the container vulnerability scanners.

Yes, we're all certified now! ;-)

• Day 2:
• Information Overload and the Real Costs of Interruptions, a keynote by Hannah Foxwell.
  This was an awesome talk about how interruptions of any kind influence our effectiveness, the quality of our output and particular our health. This is a related blog post by her. Example quotes from her talk:
  - "It takes an average of 23 minutes for the average person to regain focus after being interrupted."
  - "Cognitive losses from multi-tasking are greater than the cognitive losses from smoking weed."
  Some ways to fight against interruptions:
• Manage your notifications.
• Do a brain dump (i.e. taking notes).
• Sort tasks and plan to complete similar tasks at the same time (e.g. email).
• Don't multi-task.
• Have a routine that works for you.
• Decide what NOT to do.
• The Art of Supportability by TP Honey.
  Quote: "I don't work for the management, I work for the leaders in my team." TP talked about why it is important to produce code that is maintainable. Topics covered: clean code with good naming, good documentation incl. FAQs, meaningful error messages, ease of setup and use.
• The Mathematics of Reliability by Avishai Ish-Shalom.
  This was a very interesting talk about calculating reliability e.g. taking dependencies into account. The repo contains the presentation and a Jupyter notebook.

For the rest of the day I missed the talks, ignites, open spaces etc., as I met a lot of friends and former colleagues exchanging lots of news, gossip and "war stories".

DevOpsDays Kiel

So, DevOpsDays again - but this time in Kiel.

Some notes from the first day:

• From 0 to DevOps - Coping with Continuous Delivery to the Cloud by Oliver Siebenmarck, IBM:
• In the old days it was "Dev versus Ops". Now the culture is changing towards a shared ownership of test automation, continuous integration, continuous delivery and continuous deployment.
• Lessons learned from a project he worked on:
• It's about communication.
• It's not about the tools.
• DevOps is not No-Ops.
• Automation is dangerous. My first reaction was that he is completely wrong with that, but he restricted his statement to continuous deployment. But still I think the statement on its own is wrong ... the tests gating the deployment should have been improved.
• Automation is your friend.
• Write good documentation. Hmmm ... at least partial disagreement from my side, as I think outdated documentation is even worse than no documentation - e.g. you should establish a good self-explanatory coding style instead.
• Keep on learning.
• ITIL and DevOps can be friends by Jan-Joost Bouwman, ING:
• Transformation of the IT department towards agile and shared responsibility among Dev and Ops.
• Make ITIL processes as lean as possible.
• In order to still be able to respond to incidents, approximately 30% of each sprint is reserved for operational tasks.
• Problem management tasks are managed as user stories within the product backlog.
• What agile/Scrum can learn from ITIL:
• Customer focus and reliability.
• Feedback loop (mostly concerning incident management).
• Uniform process.
• Discipline.
• What ITIL can learn from agile/Scrum:
• Need for speed.
• Customer focus by adding value.
• Limiting work in progress.
• Feedback loop (concerning what customers really want).
• Continuous Testing for DevOps: Evolving Beyond Automation by Wayne Ariola, Parasoft:
  A business-focused talk covering some numbers about impact of faulty software, the evolution of the software development lifecycle and the need for test automation and continuous testing.
• Learning to fall by Rafael Ördög, Emarsys:
  Immediately after lunch I'd have needed an exciting talk, but I got a high level talk on scaling issues and distributed transactions. It turned to become interesting, when the saga pattern as a way to guarantee distributed transactions was introduced: Split a long running transaction into several independent tasks and associate compensating actions for rollback to each of them. If the compensating actions are idempotent, they are repeatable, which is nice if you did not receive any acknowledgement.
• DevOps and sharing (slides) by Marta Paciorkowska, Acrolinx:
• DevOps is not only about technology.
• Obstacles when adopting DevOps at her company:
• Mindset. Rejecting changes.
• Overwhelming amount of tools.
• No one wants to do the boring tasks.
• Using outdated software/technologies and the lack of documentation for that.
• Lack of resources.
• No bigger picture.
• Other people.
• Possible fixes:
• Respect people's preferences and their strength. Leverage that when distributing work.
• Fight the hesitation (empathy). Develop the mindset.
• Build a long-term plan.
• Listen to complains.
• Motivate colleagues.
• Ignites:
• Keep it simple, stupid! (A reminder to myself and a surprisingly accurate analogy to how (not) to build sustainable systems) by Moritz Rogalli, Init AB:
  He showed his way towards KISS.
• The (Un)Surprising Truth About DevOps Culture by Manuel Pais:
• Introduced different types of organizations (power-oriented, rule-oriented and performance-oriented).
• A different context requires different DevOps means, e.g. depending on the type of organization you are working in. You need to identify that.
• Ops requirements have to appear within the product backlog.
• Devs should monitor apps.
• Devs should deploy apps.
• CAMS.
• 3 Open Space sessions with 3 parallel tracks for each session.

And my notes from the second day:

• The Challenges of Adopting DevOps by Bianca Heinemann, IBM:
  Culture, cloud, network, budget, tool and app ... I neither fully agree with the selection of challenges nor with the statement that you have to define everything upfront before you "implement" DevOps.
• Docker Container Lifecycles - Problem or Opportunity? by Baruch Sadogursky, JFrog:
  An interactive poll among the audience showed that just a few companies do Docker in production. The talk advertised the good Docker repository support in JFrog Artifactory and how it supports promotion pipelines from image builds in development to image deployments in production.
• Promote Docker images across stages. Do not promote Dockerfiles and run "docker build" in each stage.
• Labels in Docker repositories are not immutable by itself, e.g. the 14.04 label of the official Ubuntu Docker image will point to different images over time (due to security patches). The image hashes are immutable.
• With "docker tag" you can just address a single repository. JFrog Atrifactory offers virtual repositories to address different repositories across stages. You can promote images between virtual repositories.
• The Artifactory Query Language supports searching images by their metadata.
• Collect all the things with Beats by Philipp Krenn, Elastic:
• Short intro into the ELK stack.
• Beats is written in Go and is meant as a complement to Logstash. Beats offers different shippers to collect and ship data either directly to Elasticsearch or let Logstash enrich the collected data.
• It's all based on a framework, and here is an interactive generator to create your own shippers.
• Here is a quickstart setup to start playing around with ELK and Beats.
• War Games - flight training for DevOps by Jorge Salamero Sanz, Server Density:
• Humans in DevOps: System issues often become human issues due to alerts, incidents, pressure of time etc. See also here.
• Expect downtimes: Prepare and practice, respond, do postmortems.
• They established a general process to respond to incidents, which includes logging the incident, joining certain chat rooms and following checklists. Especially with the last I totally agree: At my former employer we had good experience using checklists - we called them standard operating procedures.
• They introduced war games, in which they actually walk through the incident response processes on realistic replica systems or by utilizing infrastructure mocks. So, they practice, and they check the process for its validity. They can go even further by combining failure scenarios to get used to such quite rare, but fatal situations.
• Results for the humans:
• Increased confidence.
• Reduced panic.
• Improved coordination.
• Established trust relationships.
• Improved time to resolution.
• They also worked on the following:
• Optimizing the on-call design.
• Prioritization of alerts.
• Optimizing notifications (concerning amount and content).
• Rethinking Performance Engineering in the DevOps World by Felix Willnecker and André van Hoorn, fortiss/University of Stuttgart:
• Performance = timing behavior + resource utilization.
• Use performance models to simulate changes in infrastructure, code or user behavior.
• Opportunities from DevOps:
• Automation.
• Tooling adoption.
• Modern technologies (e.g. using cloud services).
• Opportunities from software performance engineering (SPE):
• Mature approaches and tools.
• Challenges for the integration of SPE and DevOps:
• Fast feedback expected, but performance evaluation takes time (e.g. stress tests).
• Keep up with more frequent changes.
• Modern technologies can be challenging by itself.
• Optimistic approaches like A/B testing might not be able to be performed on business-critical systems.
• There is also a DevOps Performance Working Group.
• Working in and with Open Source Communities by Bernd Erk, NETWAYS:
• Communities are controlled by people, not by leaders.
• Communities look differently from the outside and from the inside.
• Four stages of community building:
• Pseudo-community of people sharing a common value to get something started.
• Chaos by growing demands.
• Emptiness, as you have to deal with weakness of yourself and others.
• Respect and empathy.
• Disagree with ideas, not with people.
• Bring in new members.
• Be as flexible as possible, but take over responsibility.
• Take care of the internal culture and respect it.
• Don't be a control freak. Be transparent.
• Measure progress.
• Value and mention the community members.
• Meet in person as much as possible. Communicate a lot.
• Establish a code of conduct, take care of it and comply with it.
• Be polite to everyone, but don't accept harassment.
• Ignites:
• Scalable and clean build environments with Jenkins and Docker by Stefan Kahlhöfer:
• Increased number of build jobs.
• Less dependencies between build jobs.
• Integration methods tested so far:
• Docker Plugin.
• CloudBees Docker Custom Build Environment Plugin.
• Custom bash script developed by themselves.
• He also mentioned Yet Another Docker Plugin, which looks promising, but he has not tested it yet.
• Praise the "overwrite" - Why boring approaches are awesome! by Florian Sellmayr, ThoughtWorks:
• Build something simple.
• Reduce moving parts.
• An Open Space session.

Here are some pictures taken at the DevOpsDays Kiel.

DevOpsDays Berlin 2014

For the last two days, DevOpsDays Berlin 2014 attracted quite a lot of people.
The location (Kalkscheune) was great, the catering even better. The conference itself consisted of a good balance of classical talks, ignite talks and open spaces. Recordings of the talks are available here - my favorite talk was Ben Hughes' Ein Security Bitte.
The open spaces covered a lot of topics like ChatOps hands-on, automating security (this Chef cookbook seems to be a good inspiration), DevOps at scale and monitoring in the elastic cloud with lots interesting discussions.
All in all it was an extremely well organized event with tons of inspiring information ... and we jumped on the DevOps certification joke :-)

Currently a running gag in the DevOps community: DevOps certification