ContainerDays

I skipped the workshop day and just joined the second day of ContainerDays 2016. My notes from the sessions I attended:

• From Borg To Kubernetes: The History And Future Of Container Orchestration by Mandy Waite (Google): Mandy explained Google's demand for scheduling different workloads (monitoring, production tasks, batch runs) with resource isolation concerning CPU, memory and I/O. Many learnings and experience from internal container management systems has been baked into Kubernetes (k8s). Some core features of k8s were presented (e.g. deployments update a service, which is made of pods, that in turn group closely related containers) concluding with an outlook of some upcoming features (e.g. improved monitoring).
• Shaping Applications for Docker, CoreOS, Kubernetes and Co by Thomas Fricke (Endocode AG) (here's another version of the slides from a different conference and a video of his talk at that conference): Key takeaway: When designing your application, separate "cattle" (i.e. systems you don't explicitly care about and that can be scaled massively and replaced easily) and "pets" (i.e. systems you care about), separate stateless layers (this is your "cattle") and persistence layers (these are your "pets"). Thomas also introduced the key differences between virtualization and containers and showed the existing ecosystem in the container world plus a case study with the lessons learned.
• Linuxkernel features building my $CONTAINER by Erkan Yanar: Erkan showed some details of the base technologies for containers that are available for years now (e.g. chroot, capabilities, cgroups, namespaces).
• The NoOps Movement by Marco Hutzsch (Otto): To me, what Marco named NoOps is actually DevOps done right or pushed further. It's about automation, infrastructure as code, full-stack responsibility incl. developers being on call, serverless architecture etc. He claimed to get rid of dedicated operations teams, but instead to build delivery teams with engineers being able to develop software and being interested in running what they've built. Reminds me of what we had at my former employer, and I can confirm that you can really gain speed with this approach.
• Rancher Docker - From zero to hero by Michael Vogeler (Nexinto GmbH): Actually, I've recently played around a bit with Rancher in the AWS cloud. It offers nice features like resource management, cross-host networking, service discovery, monitoring and support for several scheduling engines like Kubernetes or Swarm. And it's set up quite easily - so, worth a second look ...
• Plan B: Service to Service Authentication with Oauth by Henning Jacobs (Zalando): Each team has its own virtual data center (i.e. separate AWS cloud accounts) with its own domain. If one service wants to talk to another service, it has to contact the ELB bound to the domain of the other service. For the authorization on that communication they've chosen OAuth 2.0 with credentials stored on S3 (protected by IAM policies) to acquire auth tokens. The tokens are JSON Web Tokens which are self-contained for validation. The public keys are stored in a OAuth provider service, token revocations are handled by another service. All components are open source, docs are here.
• A lightning talk by Florian Leibert (Mesosphere) providing an overview of the history and the architecture of Mesos and DC/OS including a live demo.
• Efficient monitoring in modern environments by Tobias Schmidt (Soundcloud): Soundcloud created Prometheus. You should collect metrics on each and every layer of your systems (hosts, containers, applications) - detailed per component and not just aggregated. Aggregations are handy for alerting and dashboards. Four golden signals to monitor: Latency, traffic (e.g. requests per second), errors and saturation/utilization. Don't rely on medians, use percentiles instead (90%, 95%, 99%, 99.9% etc.). At Soundcloud every developer is on call. Use symptom based alerting instead of cause based, i.e. monitoring business processes instead of isolated technical components adds value - the four golden signals mentioned before can provide a good guidance. Only alert when human interaction is required - this reduces noise and prevents alert fatigue. Use alert grouping and service silencing e.g. for maintenance, avoid static thresholds in favor of relative thresholds e.g. based on historic data, use a ticketing system and treat warnings like new features. Provide concise playbooks/runbooks to enable fast problem mitigation. Practice outages! Recommended read: My Philosophy on Alerting.
• A web shop in containers - Building the microservice platform for otto.de by Florian Sellmayr (ThoughtWorks) & Felix Bechstein (Otto): Florian and Felix presented an overview of the architecture behind otto.de using microservices running on Docker containers controlled by Marathon. With this architecture they gained freedom and responsibility for the developers, the ability to deploy new services without interaction with operations, increased standardization, better scalability and faster delivery. Problems they faced: Poor adoption of Mesos in various frameworks, service discovery, poor support for multi-tenancy and authentication, resource and container isolation. To mitigate some of these problems they came up with separated clusters and refactored them a lot over time (automation helped here a lot). Secrets are managed by Vault.

Distributed Matters 2015

Distributed Matters 2015 in Berlin - Friday was workshop day. I've chosen to attend Data applications on Hadoop, live from the trenches held by Arnaud Cogoluègnes and A Hands-on Introduction to Ansible by Victor Volle.
Arnaud's workshop was well-prepared by supplying a VM with everything needed for the workshop. It was a beginner level workshop with a good amount of exercises using Java. Unfortunately, we ran out of time to do all exercises. Arnaud covered the following topics:

• Introduction into HDFS, YARN and MapReduce
• Cascading (incl. using Tez under the hood)
• File formats to store data in Hadoop (e.g. Avro, ORC, Parquet)
• Hive
• Spring Batch

Victor's workshop consisted of an introduction into Ansible accompanied by exercises performing installation and configuration tasks. Again, we ran out of time to complete all exercises. He covered the following topics:

• Motivation for automation and configuration management
• Ansible core concepts (incl. the very good documentation)
• Idempotency and target state detection
• Playbooks with some modules by example
• Modules and reuse of code
• The still poor multi-stage/multi-environment support of Ansible, if you want to group nodes in several dimensions (e.g. roles like web servers and development stages like test environment)

Saturday was conference day with a bunch of talks in two parallel tracks. Here are my notes from the talks I've attended:

• The keynote was held by Kyle Kingsbury about Jepsen, a tool to simulate network partition for different distributed databases, schedulers etc. Network partition is kind of a worst case scenario in distributed systems, and Kyle went through some NoSQL database products and how they cope with network partition and whether they keep their promise of data consistency.
• Clojure at Braintree: Real-time Data Pipeline with Kafka by Joe Nash: Braintree is a payment processor using real-time data pipelines. Their data warehouse is Amazon Redshift, and Joe described their journey to introduce Kafka and Clojure for feeding data into Elasticsearch for transaction search. Kafka features they found useful:
  - Implements the pub/sub pattern.
  - Topics are spread across partitions.
  - The retention period is configurable.
  - The message stream is strongly ordered.
• Upgrade your database: without losing your data, your perf or your mind by Charity Majors: What can go wrong? A lot! Data loss, queries being slowed down, broken replication, ... Risk assessment is important, e.g. about the type of data you risk to loose or the existence of a rollback path. Charity presented why MongoDB has the highest update risk in the context she is working - mainly because of the type of data they store in there and the fact that MongoDB is quite young and immature with a lot of changes between releases. Some learnings she presented:
  - One change at a time.
  - In an ideal world run tests with production traffic and production data with each change. They capture and replay each API request to hit the production database and a shadow database (which might already be upgraded) and compare the results.
  - Don't trust vendors, run your own benchmarks.
  - Know your workloads.
  - Measure and look for outliers e.g. by tracking the p99 percentile.
• Stream based textanalytics with Spark and Elasticsearch by Stefan Siprell and Hendrik Saly: They explained CRISP (Cross Industry Standard Process for Data Mining) using different domains. Tools mentioned were R (with some small code examples concerning data cleansing and clustering being presented), Spark (e.g. for stream processing of tweets and pushing data to Elasticsearch - incl. some code examples), Elasticsearch (for interactive data exploration using Kibana). Some machine learning fundamentals like supervised learning, e.g. based on support vectors, were covered.
• Running database containers using Marathon and Flocker by Kai Davenport: He talked about scheduling of containers across many hosts. Mesos manages resources in a cluster. It uses a master/slave architecture and stores its state in ZooKeeper. Marathon is a framework on top of Mesos and manages containers in a cluster. Flocker orchestrates storage across a cluster of containers providing a REST API. It requires an agent on each host and relies on a central control service. Kai closed his talk with a storage failover live demo.
• Containers! Containers! Containers! And Now? by Michael Hausenblas: No slides! Just a terminal session and a web browser. Michael lead us towards the motivation for using containers and a cluster management layer for containers - Marathon again. He went through this blog post in a live demo fashion.
• Microservices with Netflix OSS and Spring Cloud by Arnaud Cogoluègnes: Spring Cloud wraps the tools from Netflix OSS to enable them for Spring projects. Arnaud showed some Java code examples to demonstrate how easily this can be used.
• Conflict Resolution with Guns by Mark Nada: This talk was about problems of distributed systems like split brain, building a quorum and the CAP theorem. Gun DB, a distributed cache, tries to mitigate those problems by using local caching and conflict resolution algorithms. It challenges the need for strongly consistent systems.
• Disque: a detailed overview of the distributed implementation by Salvatore Sanfilippo, the guy behind Redis: This final talk was about Disque, a distributed message queue. Disque has been forked from Redis and uses the same protocol. Salvatore provided some technical insights into the API and details like ID generation, delivery guarantees, replication schemes and save restarts (restarting the system with persisting the messages before shutdown and loading them again after start). He concluded with a short live demo.