DevOpsDays Berlin 2017

It's a good tradition to attend DevOpsDays Berlin. 2 days with many talks, ignite talks, hands-on workshops and open space sessions. But this year I was a bit disappointed: less attendees than the years before, and also less interesting talks - at least to me ... I know, this is my personal point of view ... others might see it differently.

The following were the top talks to me:

• Long-Lived Devops And The Pit Of Success by Jason Diller. Jason talked about the evolution of teams and technologies dealing with ongoing constant changes. He introduced how to measure the bus factor to gain visibility on orphaned technologies in business. My favorite quote: "Freedom until it hurts, standards until it works."
• It's Not Continuous Delivery If You Can't Deploy Right Now by Ken Mugrage. He mentioned integration of performance & security testing within your build pipeline, canary deployments, feature toggles etc. among others. One of the points - every commit should be pushed to master - was quite controversial. But in the discussion afterwards, Ken put this into another perspective: merging your feature branch to master in short intervals is ok, too.
• Encouraging A Culture Of Learning Across Your Organisation by Jenny Duckett. Jenny presented her motivation of preparing and running a security workshop although not being a security expert. She also showed how this has changed the culture within her organsiation.
• In his ignite talk Ken Mugrage showed value stream mapping by example with getting a teenager doing the chores - absolutely entertaining.

DevOpsDays Berlin 2016

Aah, Berlin again, DevOpsDays again - love it!

Here are some notes:

• Day 1:
• Looking back at 7 years of DevOpsDays, a keynote by Kris Buytaert.
  Kris showed a compact review from when Patrick Debois and he initiated the DevOps movement in 2009.
• Infrastructure as a Microservice by Konrad Hosemann.
  It was a very good talk about their journey from a monolithic architecture run in a data center to microservices running in a public cloud. They implemented some infrastructure services and used several distinct accounts for different business units like search and data science. The advantages they recognized were benefits from external innovation, autonomous teams with a "you build it, you run it" attitude, less to no side effects, cost transparency incl. cost awareness. On the other hand, they were facing problems like clarifying the ownership of the infrastructure services, distributed updates across several accounts, avoiding fragmentation and gaining a unified view of their landscape. Konrad also showed interesting numbers: The Graphite setup consists of four 24 core machines with 72 GB RAM and 360 GB HDD each and handles 1.5 million metric updates per second; the ELK landscape utilizes ten 20 core machines with 128 GB RAM, 1.5 TB SSDs and 15 TB HDD each handling 150,000 write requests per second.
• Communities of practice in large scale DevOps transformations by Martin Thalmann.
  A talk about the transformation towards a DevOps culture by establishing communities of practice in an organisation of about 4,300 IT people (give enough room/time to the members ... and provide beer and sausages). They also founded a local meetup group to open up the exchange with external contributors. Based on the number of questions after Martins talk, this seems to be a topic a lot of people deal and struggle with.
• Talking to People: The forgotten DevOps tool by Peter Varhol.
  Instead of (over-)using electronic communication (e.g. email, IM) we should communicate more face to face, as we build trust and solve problems more quickly. Everyone, but especially management should lead by example, e.g. by putting the phone away during meetings. A regular "beer thirty" could help to establish communication and trust.
• Ignites:
• The 20 Shades of Grey by Michael Hüttermann.
  Michael showed 20 ways to make your project and DevOps efforts fail, e.g. DevOps teams as an anti-pattern.
• Git and Github Tricks by Michal Bryxí.
  Things I didn't know before:
  - Ignore whitespace changes in diff: git diff -w -b
  - Better diffs by leveraging heuristics: git diff --compaction-heuristic
• Come to Germany, Pepe! by Anthony Stanton.
  Anthony presented his development and learnings from a classical IT department employee to a DevOps.
• DevOps? That's not my job! by Nathen Harvey.
  An entertaining ignite about the walls between dev and ops motivated by this blog post about serverless computing. Here is the longer version of that ignite as a full-length talk.
• Felix Frank shortly introduced the motivation behind mgmt.
• Hands-On Workshops (in parallel to Open Spaces):
• Next Generation Config Mgmt by James Shubin.
• How to read code and understand bugs by David Schmitt.
• Habitat Workshop by Eric Maxwell.
• I skipped the workshops, but joined the following Open Spaces - the resulting schedule is available here, and someone had the great idea to add further information directly to each card:
• Secret Management
• Chef Vault
• HashiCorp Vault
• AWS Key Management Service
• Confidant
• hiera-eyaml-gpg
• Audit breaches using auditd or go-audit.
• Use Teleport to control SSH access.
• Monitoring needs more LEDs
• Panels, LEDs, Arduino, Raspberry Pi etc.: here
• Scanning containers for CVEs
• JFrog Xray
• Clair
• Vuls
• Overview of the container vulnerability scanners.

Yes, we're all certified now! ;-)

• Day 2:
• Information Overload and the Real Costs of Interruptions, a keynote by Hannah Foxwell.
  This was an awesome talk about how interruptions of any kind influence our effectiveness, the quality of our output and particular our health. This is a related blog post by her. Example quotes from her talk:
  - "It takes an average of 23 minutes for the average person to regain focus after being interrupted."
  - "Cognitive losses from multi-tasking are greater than the cognitive losses from smoking weed."
  Some ways to fight against interruptions:
• Manage your notifications.
• Do a brain dump (i.e. taking notes).
• Sort tasks and plan to complete similar tasks at the same time (e.g. email).
• Don't multi-task.
• Have a routine that works for you.
• Decide what NOT to do.
• The Art of Supportability by TP Honey.
  Quote: "I don't work for the management, I work for the leaders in my team." TP talked about why it is important to produce code that is maintainable. Topics covered: clean code with good naming, good documentation incl. FAQs, meaningful error messages, ease of setup and use.
• The Mathematics of Reliability by Avishai Ish-Shalom.
  This was a very interesting talk about calculating reliability e.g. taking dependencies into account. The repo contains the presentation and a Jupyter notebook.

For the rest of the day I missed the talks, ignites, open spaces etc., as I met a lot of friends and former colleagues exchanging lots of news, gossip and "war stories".

DevOpsDays Kiel

So, DevOpsDays again - but this time in Kiel.

Some notes from the first day:

• From 0 to DevOps - Coping with Continuous Delivery to the Cloud by Oliver Siebenmarck, IBM:
• In the old days it was "Dev versus Ops". Now the culture is changing towards a shared ownership of test automation, continuous integration, continuous delivery and continuous deployment.
• Lessons learned from a project he worked on:
• It's about communication.
• It's not about the tools.
• DevOps is not No-Ops.
• Automation is dangerous. My first reaction was that he is completely wrong with that, but he restricted his statement to continuous deployment. But still I think the statement on its own is wrong ... the tests gating the deployment should have been improved.
• Automation is your friend.
• Write good documentation. Hmmm ... at least partial disagreement from my side, as I think outdated documentation is even worse than no documentation - e.g. you should establish a good self-explanatory coding style instead.
• Keep on learning.
• ITIL and DevOps can be friends by Jan-Joost Bouwman, ING:
• Transformation of the IT department towards agile and shared responsibility among Dev and Ops.
• Make ITIL processes as lean as possible.
• In order to still be able to respond to incidents, approximately 30% of each sprint is reserved for operational tasks.
• Problem management tasks are managed as user stories within the product backlog.
• What agile/Scrum can learn from ITIL:
• Customer focus and reliability.
• Feedback loop (mostly concerning incident management).
• Uniform process.
• Discipline.
• What ITIL can learn from agile/Scrum:
• Need for speed.
• Customer focus by adding value.
• Limiting work in progress.
• Feedback loop (concerning what customers really want).
• Continuous Testing for DevOps: Evolving Beyond Automation by Wayne Ariola, Parasoft:
  A business-focused talk covering some numbers about impact of faulty software, the evolution of the software development lifecycle and the need for test automation and continuous testing.
• Learning to fall by Rafael Ördög, Emarsys:
  Immediately after lunch I'd have needed an exciting talk, but I got a high level talk on scaling issues and distributed transactions. It turned to become interesting, when the saga pattern as a way to guarantee distributed transactions was introduced: Split a long running transaction into several independent tasks and associate compensating actions for rollback to each of them. If the compensating actions are idempotent, they are repeatable, which is nice if you did not receive any acknowledgement.
• DevOps and sharing (slides) by Marta Paciorkowska, Acrolinx:
• DevOps is not only about technology.
• Obstacles when adopting DevOps at her company:
• Mindset. Rejecting changes.
• Overwhelming amount of tools.
• No one wants to do the boring tasks.
• Using outdated software/technologies and the lack of documentation for that.
• Lack of resources.
• No bigger picture.
• Other people.
• Possible fixes:
• Respect people's preferences and their strength. Leverage that when distributing work.
• Fight the hesitation (empathy). Develop the mindset.
• Build a long-term plan.
• Listen to complains.
• Motivate colleagues.
• Ignites:
• Keep it simple, stupid! (A reminder to myself and a surprisingly accurate analogy to how (not) to build sustainable systems) by Moritz Rogalli, Init AB:
  He showed his way towards KISS.
• The (Un)Surprising Truth About DevOps Culture by Manuel Pais:
• Introduced different types of organizations (power-oriented, rule-oriented and performance-oriented).
• A different context requires different DevOps means, e.g. depending on the type of organization you are working in. You need to identify that.
• Ops requirements have to appear within the product backlog.
• Devs should monitor apps.
• Devs should deploy apps.
• CAMS.
• 3 Open Space sessions with 3 parallel tracks for each session.

And my notes from the second day:

• The Challenges of Adopting DevOps by Bianca Heinemann, IBM:
  Culture, cloud, network, budget, tool and app ... I neither fully agree with the selection of challenges nor with the statement that you have to define everything upfront before you "implement" DevOps.
• Docker Container Lifecycles - Problem or Opportunity? by Baruch Sadogursky, JFrog:
  An interactive poll among the audience showed that just a few companies do Docker in production. The talk advertised the good Docker repository support in JFrog Artifactory and how it supports promotion pipelines from image builds in development to image deployments in production.
• Promote Docker images across stages. Do not promote Dockerfiles and run "docker build" in each stage.
• Labels in Docker repositories are not immutable by itself, e.g. the 14.04 label of the official Ubuntu Docker image will point to different images over time (due to security patches). The image hashes are immutable.
• With "docker tag" you can just address a single repository. JFrog Atrifactory offers virtual repositories to address different repositories across stages. You can promote images between virtual repositories.
• The Artifactory Query Language supports searching images by their metadata.
• Collect all the things with Beats by Philipp Krenn, Elastic:
• Short intro into the ELK stack.
• Beats is written in Go and is meant as a complement to Logstash. Beats offers different shippers to collect and ship data either directly to Elasticsearch or let Logstash enrich the collected data.
• It's all based on a framework, and here is an interactive generator to create your own shippers.
• Here is a quickstart setup to start playing around with ELK and Beats.
• War Games - flight training for DevOps by Jorge Salamero Sanz, Server Density:
• Humans in DevOps: System issues often become human issues due to alerts, incidents, pressure of time etc. See also here.
• Expect downtimes: Prepare and practice, respond, do postmortems.
• They established a general process to respond to incidents, which includes logging the incident, joining certain chat rooms and following checklists. Especially with the last I totally agree: At my former employer we had good experience using checklists - we called them standard operating procedures.
• They introduced war games, in which they actually walk through the incident response processes on realistic replica systems or by utilizing infrastructure mocks. So, they practice, and they check the process for its validity. They can go even further by combining failure scenarios to get used to such quite rare, but fatal situations.
• Results for the humans:
• Increased confidence.
• Reduced panic.
• Improved coordination.
• Established trust relationships.
• Improved time to resolution.
• They also worked on the following:
• Optimizing the on-call design.
• Prioritization of alerts.
• Optimizing notifications (concerning amount and content).
• Rethinking Performance Engineering in the DevOps World by Felix Willnecker and André van Hoorn, fortiss/University of Stuttgart:
• Performance = timing behavior + resource utilization.
• Use performance models to simulate changes in infrastructure, code or user behavior.
• Opportunities from DevOps:
• Automation.
• Tooling adoption.
• Modern technologies (e.g. using cloud services).
• Opportunities from software performance engineering (SPE):
• Mature approaches and tools.
• Challenges for the integration of SPE and DevOps:
• Fast feedback expected, but performance evaluation takes time (e.g. stress tests).
• Keep up with more frequent changes.
• Modern technologies can be challenging by itself.
• Optimistic approaches like A/B testing might not be able to be performed on business-critical systems.
• There is also a DevOps Performance Working Group.
• Working in and with Open Source Communities by Bernd Erk, NETWAYS:
• Communities are controlled by people, not by leaders.
• Communities look differently from the outside and from the inside.
• Four stages of community building:
• Pseudo-community of people sharing a common value to get something started.
• Chaos by growing demands.
• Emptiness, as you have to deal with weakness of yourself and others.
• Respect and empathy.
• Disagree with ideas, not with people.
• Bring in new members.
• Be as flexible as possible, but take over responsibility.
• Take care of the internal culture and respect it.
• Don't be a control freak. Be transparent.
• Measure progress.
• Value and mention the community members.
• Meet in person as much as possible. Communicate a lot.
• Establish a code of conduct, take care of it and comply with it.
• Be polite to everyone, but don't accept harassment.
• Ignites:
• Scalable and clean build environments with Jenkins and Docker by Stefan Kahlhöfer:
• Increased number of build jobs.
• Less dependencies between build jobs.
• Integration methods tested so far:
• Docker Plugin.
• CloudBees Docker Custom Build Environment Plugin.
• Custom bash script developed by themselves.
• He also mentioned Yet Another Docker Plugin, which looks promising, but he has not tested it yet.
• Praise the "overwrite" - Why boring approaches are awesome! by Florian Sellmayr, ThoughtWorks:
• Build something simple.
• Reduce moving parts.
• An Open Space session.

Here are some pictures taken at the DevOpsDays Kiel.

Delivery of Things World 2016 - Day 2

... continuing with my notes from the second day:

• From oil tankers to speedboats by Jonathan Smart, Head of Development Services at Barclays (keynote):
• They had 400 internal standards (lots of regulation) and used waterfall.
• They are currently in the process into an agile organization. 800+ teams working with agile practices, 3x more stories finished than a year before. He showed a graph indicating a strong correlation that apps being deployed more often lead to less production incidents.
• Cultural changes may take years.
• Don't scale agile. Descale the work first.
• Shu Ha Ri.
• Disciplined Agile used as scaling framework (here and here you can find more information about this framework).
• Implement communities of practice.
• Story telling, training and coaching are essential.
• Leadership training.
• Head for technical excellence.
• Agile architecture (no monolithic architecture).
• DevOps is a practice, not a product.
• They implemented a simple control tool: Answer 20 questions (before: hundreds), the number of control points were reduced from 7 to 2.
• Blending not clashing (one size does not fit all): Be agile in adoption, sometimes different speed in adoption is required.
• Embrace continual change.
• Transforming application delivery for Continuous Innovation by Jonny Wooldridge, CTO at The Cambridge Satchel Company (keynote):
• "It's all about the code."
• Heavily invested in increasing excellence & craftsmanship with their move towards agility.
• Legacy applications can impede moving towards agility.
• Define and decouple your "pace layers".
• Watch out for corporate equilibrium: Enterprise equilibrium (wrong technology, wrong people, wrong frameworks, wrong 3rd party suppliers) tends to drive DevOps adoption backwards.
• Build new initiatives the right way (reduce complexity, reduce legacy code).
• Kill dependencies.
• Focus on APIs.
• Don't create new legacy (i.e. don't use legacy approaches just because they are quickly available, you have to actively invest into agile). 
• Accelerating DevOps adoption: Patton and Gandhi by Stephen Fishman, Senior Director, Platforms & Services at Autotrader (keynote):
• Never be ashamed to ask for excellence.
• Religiously strive for collegiality.
• How to persuade people being skeptic about agile transformations:
• Step 0: Assume positive intent.
• Step 1: Empathize.
• Step 2: Point to higher ground.
• Step 3: Wait.
• Step 4: Repeat.
• World Café Sessions (smaller groups of about 20 people discussed a certain topic for about 30 minutes; we had some good discussions about the integration of developers and ops people, the introduction of continuous delivery and agile methods):
• Analyzing Continuous Delivery: What are the advantages - big and small - that you can look forward to once you have Continuous Delivery in place? by Lars Bendix, Professor of Software Engineering, Department of Computer Science at Lund Institute of Technology.
• Ericsson Eurolab: World of DevOpsCraft by Almudena Rodriguez Pardo, Senior Scrum Developer, ICT Development Center Eurolab Aachen at Ericsson and by Norma Acevedo, Product Deployment Manager at Ericsson.
• Don’t build a golden prison – Dos and Don’ts for working with DevOps by Udo Pigorsch, DevOps System Manager at Axel Springer SE.
• Continuous Quality and Continuous Delivery: How to have both by Tobias Kutzer, Senior Solution Architect at Worksoft.
• Data-Driven DevOps to Improve Velocity, Quality, and Impact / Using Continuous Delivery to build products customers actually use by Andi Mann, Chief Technology Advocate at Splunk.
• Screwing Up For Less by Stephen Hardisty, Director of Engineering at Rocket Internet (keynote):
• If people feeling comfortable, they get careless (see "risk homeostasis", also here).
• Embrace feature toggles.
• Make problems more obvious (i.e. visibility, wall screens).
• Automate all the things.
• Managers: No shouting! (I.e. no blaming.)
• Reduce risk by using technologies that are already established in your organization.
• Balancing the 3 pillars of DevOps - Managing people, process, and technology change to grow a DevOps culture with Stephen Hardisty, Director of Engineering at Rocket Internet, Jonny Wooldridge, CTO at The Cambridge Satchel Company and Dave Nolan, CTO at AppearHere (panel discussion):
• Topics addressed:
• Hire people with the right mindset.
• Establish culture and mindset.
• The right tools.
• Trust in the change.
• Gain speed and reduce time to market.

Panel discussion

Delivery of Things World 2016 - Day 1

Today, Delivery of Things World 2016 - a conference about DevOps and continuous development - started in Berlin. It spans two days, and about 300 people participate. I attended the following sessions on the first day:

• DevOps and Moving the Elephant by Chris Gargiulo, Head of DevOps & Enterprise Service Platforms at Maersk Group (keynote):
• You have to start communication and work against fears.
• Continuous integration and continuous delivery are starting points for the journey towards trust in DevOps. He showed numbers of an example concerning decreased build times due to automation.
• First rule of DevOps: There's no crying in DevOps! I.e. you have to stand throwbacks and resist switching back to the old manual processes, learn from failures and watch out for potential improvements.
• Define a mission goal, adopt changes incrementally, quantify the results and celebrate success accordingly.
• Leading organisations that don‘t need leading by Mark Rees, former COO at E-POST Development (keynote):
• Their development units consist of several DevOps teams, each DevOps team in turn consists of a number of developers, a QA engineer and a delivery lead - kind of a mixture of a scrum master and a product manager.
• Responsibility and leadership always came up in discussions.
• Leaders should act as enablers and provide an environment for teams to become confident about taking responsibility for yet unexplored fields.
• Leaders should coach their teams in the beginning and then step back, delegate and provide support.
• Don't manage your teams, lead them!
• SecDevOps: How can we create cultural change to bridge this gap between security and DevOps? by Ben Hughes, Network Security Manager at Etsy:
• A typical DevOps pyramid: 10 developers, 1 ops person.
• Typical SecDevOps pyramid: 100 developers, 10 ops persons, 1 security person.
• Establish security champions: Join security conferences, hang around in relevant chats etc.
• Etsy does security bootcamps: Join a couple of other different teams for some time, then return to your home team (you are now well connected and got the security word spread).
• Security team culture:
• "Don't hire a**holes."
• Be approachable.
• Be transparent.
• Be humble.
• Act blamelessly.
• Create a culture where people report strange things happening rather than pretending everything is fine.
• "If you make security hard, people won't use it."
• Get everybody in your company/department to be "part of" your security team. Security is not something someone else cares about.
• Buy vs Build by Nishan Subedi, Senior Engineer at Etsy:
•  A nice tour through the tools they built:
• Virtual Madness (set up and manage development VMs).
• deployinator (deploy code changes).
• supergrep (tail production server logs).
• StatsD (metrics collector).
• morgue (for blameless postmortems).
• The main ideas behind these tools:
• Encapsulate ideas.
• Provide reasonable defaults.
• Make users feel comfortable due to transparency.
• Technology is a product of the culture that builds it. The Etsy culture consists of:
• Just Ship! (I.e. enable frequent deployments of small changes leading to continuous deployment.)
• If it moves graph it!
• Optimize for developer happiness.
• Don't be an a**hole.
• Engineering with a capital "E" - "E" as in empathy.
• A new hire always does a "First Day Deploy":
• Learn from what you see (and do).
• Make implicit explicit. No knowledge hidden in a small number of brains.
• Etsy strongly considers itself a learning organization.
• Generali‘s journey to reduce release and maintenance downtimes by 90% by Torsten Rehfisch, Scrum Product Owner at Generali:
• They decided to use XL Deploy for deploying web applications into a IBM WebSphere landscape including every dependent artifact and configuration which is needed to run a web service.
• After a description of their evaluation process these results were shown:
• Deployment landscape: 6 XL Deploy instances (3x for development/integration, 3x for UAT/pre-production/production).
• Each XL Deploy instance is set up via Puppet.
• Jenkins is used for build.
• They gained speed, confidence, transparency and repeatability.
• From Need to Speed - How to successfully setup a DevOps transformation by Robert Michel, IBM Cloud Solutions Technical Sales at IBM:
• Levels of DevOps adoption to increase delivery speed:
• Agile development/continuous integration.
• Continuous delivery.
• Continuous deployment.
• Continuous operations.
• Start fresh with a new project, start small, and think of "developer first" and "cloud first".
• Evolving to Continuous Development: a skill set example by Sebastian Ehrich, Software QA Engineer at ebay Kleinanzeigen and Manuel Aldana, Senior Development Manager at ebay Kleinanzeigen:
• Iterating often leads to a smaller change set per single deployment/iteration, which leads to less risk, better traceability and faster feedback.
• Key success factors:
• Culture.
• Mindset.
• Communication & collaboration (e.g. create communication/collaboration spaces in order not to broadly disturb people, implement open meetings where people can decide on their own whether the meeting is of interest for them).
• Visibility and spreading the knowledge (e.g. metrics on wall screens).
• Automation.
• They are mainly looking for people fulfilling the t-shape model.
• Product teams are self-containing, i.e. each product team consists of a mix of required people like developers, QA people, designers, operations people etc.
• Use kanban (due to its inherent continuous improvement).
• DevOps for leaders by Adam Lukas Urban, Head of Order Management Development at 1&1 and Tim Schuppener, Head of Development Webhosting Infrastructure & Content at 1&1:
• Invention and adoption cycles get shorter and shorter (examples: steam machine, computer, mobile phone).
• Faster shipment is required.
• Collaboration (communication, transparency) and common responsibility are essential.
• CAMS.
• First day deployment (like they do at Etsy).
• Pair operations (like pair programming).
• Hackathons, team events, socializing/communications areas.
• For leaders/management:
• Define values & targets.
• Give time & space.
• Performance will go down. Don't start measuring too early.
• Retrospectives are the key to continuous improvement.
• Software is eating the world by Henk Kolk, Chief Architect at ING (keynote):
• The bank is IT, and the IT is the bank.
• ING's journey from an organization heavily driven by processes (e.g. there were times when 64(!) documents had to be written before getting the clearance to start coding) to an agile one.
• Their reorganization lead to 180 full DevOps teams (i.e. operations people and developers in one team).
• Later they integrated the business people, too, to build product teams being completely responsible for a product. With that they got rid of 2 management levels.
• Results of their journey:
• IT drives the commercial strategy.
• IT is a value driver.
• Hiring the best talents.
• Building as the way to understanding.
• Line drives the changes.
• Products and services.
• People.
• The Rationale for Continuous Delivery by Dave Farley, Author of Continuous Delivery (keynote):
• The scientific method:
• Characterization.
• Hypothesis.
• Deduction.
• Experiment.
• Repeat!
• Iterative development processes are more successful than traditional ones like waterfall.
• Lean thinking:
• Deliver fast.
• Build quality in.
• Optimize the whole.
• Eliminate waste (unnecessary variations, overburden, wasteful activities).
• Amplify learning.
• Decide late.
• Empower the team.
• Continuous delivery leads to shorter cycle times and thus enables iterative processes.
• Continuous delivery motivation:
• First principle of the agile manifesto.
• Logical extension of CI.
• Holistic approach to development.
• Every commit creates a release candidate.
• "Finished" means released into production.
• Continuous delivery gains:
• Repeatable and reliable process.
• Automation.
• Everything under version control.
• If it hurts, do it more often.
• Build quality in.
• Done means released.
• Everybody is responsible for the release process.
• Continuous improvement.
• The Google build process shows that CD scales:
• A single monolithic repository.
• All tests run on every commit.
• More than 100 million tests are run per day.
• Amazon does one deployment every 11.6 seconds, each deployment hits 10,000 servers.
• The DevOps transformation at the HP LaserJet firmware team e.g. resulted in 8x the capacity for innovation.

DevOps Conference 2015

DevOps Conference 2015 in Munich - three days consisting of a workshop day and two conference days.
I've chosen The Docker Basics Workshop (in German) by Peter Roßbach. This workshop was very well prepared and led us through some typical tasks of building and running Docker containers with a good mixture of tips and tricks around managing those containers.
On the first conference day I attended the following sessions:

1. A keynote entitled DevOps State of the Union by John Willis. He talked a lot about culture, CAMS and ICE. Here you can find a collection of his recent talks.
2. Define Delivery Pipelines in Jenkins easily with Workflow DSL (in German) by Bernhard Cygan. It was an introduction into the Jenkins workflow plugin which provides a nice DSL for workflow definitions within Jenkins - but there seem to be some substantial differences between the commercial and the open source version ... not so nice.
3. Running Docker on AWS by Jonathan Weiss - a good wrap-up of the application management possibilities within AWS including a live demo of the integration between AWS OpsWorks and Amazon ECS.
4. Another keynote: Flexibility, how to make data science in banking work by Bart Buter. This was about how to enable data scientists with an agile working environment in the field of Big Data analysis.
5. Microservices and DevOps Journey at Wix.com by Aviran Mordo. This talk was very entertaining! Aviran talked about their journey from a monolithic to a microservice architecture ... about what to do when, and even more important, about what not to do at the beginning. As the company has to deal with the clash of their company name in German, he also showed their funny German commercial.
6. Completely Without Any Server? Yes, with AWS Lambda and API Gateway! (in German) by Andreas Mohrhard. AWS Lambda lets you run code without the need to explicitly set up an execution environment - Andreas presented a nice use case for this.
7. Wake me up before you Go-Go: Lightweight, Fast and Maintainable Web Services in Go (in German) by Philipp Tarasiewicz. A nice intro into the key concepts of Go and how to easily implement a web service.
8. Panel session: Building happier engineering teams with John Willis, Erkan Yanar, Peter Roßbach and Sebastian Meyen. To summarize it in short words: DevOps people want freedom, respect and challenges; they hate limitations, boring work and a blaming culture.
9. Three Open Space sessions completed this first conference day.

The second conference day was shorter:

1. Eight Things that make Continuous Delivery go Nuts by Eduards Sizovs - filled with war stories about how organizations kill CD, DevOps and a good team spirit by their processes and business decisions - very entertaining.
2. High Throughput Logging with Kafka and Spark by Alexandru Dabija and Viktor Kubinec. They showed a kind of opinionated logging solution based on Kafka and Spark motivated by Elasticsearch not fulfilling their requirements.
3. Reliable and Flexible Build Resources with Mesos, Docker and CoreOS (in German) by Georg Öttl - a good presentation of how they used Docker containers as Jenkins slaves.
4. Keynote: Tools, Culture, and Aesthetics: The Art of DevOps by J. Paul Reed - the title says it all and perfectly chimes in with the discussion about the DevOps mindset and culture. 
5. rkt: the container runtime by Iago Lopez Galeiras. A nice presentation of the core concepts and the architecture behind rkt, which takes a different road to application containers compared to Docker.

I skipped the final panel session.
My personal summary:

• Docker seems to be everywhere. But we should be cautious and not just follow a hype - ask yourself: Does Docker help in my current use case? Do I understand the implications?
• If you want to learn a new programming language, learn Go.
• Culture and people are more important than technology - see CAMS.

DevOpsDays Berlin 2014

For the last two days, DevOpsDays Berlin 2014 attracted quite a lot of people.
The location (Kalkscheune) was great, the catering even better. The conference itself consisted of a good balance of classical talks, ignite talks and open spaces. Recordings of the talks are available here - my favorite talk was Ben Hughes' Ein Security Bitte.
The open spaces covered a lot of topics like ChatOps hands-on, automating security (this Chef cookbook seems to be a good inspiration), DevOps at scale and monitoring in the elastic cloud with lots interesting discussions.
All in all it was an extremely well organized event with tons of inspiring information ... and we jumped on the DevOps certification joke :-)

Currently a running gag in the DevOps community: DevOps certification

Puppet DevOps Conference 2013

Yesterday, the Puppet DevOps Conference 2013 took place in Berlin organized by the Puppet User Group Berlin. It's been a very interesting conference about Puppet with a lot of good talks.

The first two talks were held by Puppet Labs staff about community management and their online training material, of which they have quite a lot.

Data driven infrastructure by James Fryman was a very interesting talk about how to implement a feedback loop to increase the degree of automation. Amongst others, he mentioned an interesting tool: Hubot.

After that David Mytton showed in Puppet at the center of everything how Puppet is used at ServerDensity.

The worst code I ever wrote by Tomas Doran was really, really entertaining with lots of code examples about what not to do in code and how to improve it.

Workflow story - theory versus practice in Large Enterprises by Marcin Piebiak was about the problems large, old-school enterprises may have with using Puppet and how tackle them.

In Puppet and Hiera - managing external information Oliver Schad gave a good overview of Hiera and its possibilities.

Finally, I attended the workshop Managing Puppet using Foreman by Martin Alfke showing Foreman in a live test environment - very nice.

Enthusiasmus ...

Diese Woche hatte ich die Gelegenheit, einem Vortrag des Wooga-CEO Jens Begemann beizuwohnen. Ein unheimlich interessanter Vortrag eines Menschen, der Enthusiasmus versprüht - genau so einen CEO wünscht man sich.
Jens Begemann hat vieles angesprochen, was wie und warum so bei Wooga gemacht wird - z.B.:

• Kleine Teams (6 bis 20 Personen), die von der Idee bis zum Betrieb (inkl. Notdienst bzw. Rufbereitschaft) die Verantwortung übernehmen - keine separierten QA-Teams, kein "irgendwer", der den Betrieb macht. Das sorgt für Identifikation und Motivation, Fehler zu vermeiden (da man diese ja selbst ausbaden muss). Es gibt wenige Querschnittsabteilungen, von denen andere abhängen.
• Fokussierung: Tote Pferde werden nicht weitergeritten, sondern aufgegeben - auch wenn es anfänglich wehtut und Verlust bedeutet. Auf längere Sicht ist es ein Gewinn und sichert die Zukunft für das Unternehmen
• Keine Bonuselemente im Gehalt: Warum sollte ein Mitarbeiter seine Energie dafür aufwenden, vorab definierte Ziele zu erfüllen, an denen Geld für ihn hängt, obwohl sich die unternehmerischen Ziele bereits verändert bzw. die vorab definierten Ziele keine Relevanz mehr haben?
• Offenheit: Jeder hat Einblick in alle Zahlen. Begemanns Tipp: Wenn man Spionage bei Wooga betreiben möchte, sollte man sich einfach als Praktikant anstellen lassen ;-)

Das ist nur ein kleiner Teil des Vortragsinhaltes, an den ich mich spontan noch erinnere. Einen weiteren Eindruck zu Wooga findet man hier.

Das führt mich gerade auf eine andere Seite, auf der man sehen kann, wen die Berliner Startups aktuell suchen. Schön ist dort auch die Kartendarstellung.

FOSDEM 2012 ...

Last weekend FOSDEM 2012 took place at the area of the Université libre de Bruxelles (ULB) - and it's been the first time I've been there. All in all it's been a great and well-organized conference with lots of interesting talks - unfortunately with many interesting ones in parallel ... so, I've missed a lot of them. Here are the talks I've attended.

• On Saturday:
• Ensuring uptime in a small startup: A good talk about some principles on how to ensure uptime during schema updates - based on PostgreSQL, but applies to any relational database.
• An introduction to Ada 2005 and Ada 2012: A time shift back into my days at the university - at least concerning the presentation style: good didactics and very old-style slides. I've never coded in Ada. So, this talk gave me a rough introduction.
• Why Java for Linux Applications?: A very personal talk about the speaker's learnings when trying to develop a GUI for an existing set of programs - and why he failed with C and Gtk+ ending up with Java and Swing.
• Debian packaging for beginners: As I'm working a lot with RPM packaging and have never done packages for Debain-based systems so far, this talk was a good introduction for me.
• Linux Containers and OpenVZ: Nothing really new for me, but a good overview of the current state and the upcoming features.
• Garbage Collection Visualization, GCSpy and NetBeans in tandem: I expected a bit more hands-on, as I spent a lot of time on GC optimization for various JVM scenarios. But GCSpy seems to be a project worth looking at in the near future.

Winter impressions at the ULB area

• On Sunday:
• Sphinx Users stories and How to offload MySQL server with Sphinx: Very interesting for me, as I'm using Sphinx for enabling fast full-text search on email data stored in a MySQL database.
• MySQL HA reloaded - old tricks and cool new tools to guarantee high availability to your MySQL Servers: A good and surely quite complete overview about the possibilities to ensure MySQL high availability and their limitations.
• Works with persistent graphs using OrientDB: A short and hands-on introduction into OrientDB.
• Bringing monitoring into the 21st century - How to predict problems before they arise: A good talk about adaptive monitoring and why fixed thresholds can be misleading.
• DevOps is not an absolute. It's a range.: A really cool talk about the different nuances of DevOps inspired by real-world experiences.
• The Apache Cassandra storage engine: Not so interesting for me, as I've never used Cassandra before.
• Configuration management and deployment with RPM and YADT: The project I'm partially working on for my current employer. A colleague of mine held this talk and made a good job.

Besides the talks there were many booths from a lot of vendors and organizations.

Beastie, the mascot of BSD operating systems