Cilium 1.0.0-rc2 - gRPC, Kafka and much more

The Cilium community has been hard at work over the past weeks to get us closer to what we consider is required for a 1.0 release. We have made a ton of progress and are happy to announce the release of 1.0.0-rc2 at this point.

New functionality that was MERGED RECENTLY:

  • Security policy enforcement at application protocol level for Kafka, and gRPC.
  • Lots of tooling around operating Cilium based clusters (cluster wide connectivity monitor, bug reporting tools, Prometheus metrics, security incident process, ...) 

  • Integration of the Envoy proxy into the Cilium datapath.

  • Lots and lots of documentation and guides.

Cilium Now Speaks gRPC!

Screen Shot 2017-12-06 at 3.18.04 AM.png

The Cilium team is happy to announce tech preview support for gRPC-aware filtering!

While the majority of existing API-based services leverage HTTP REST as their primary protocol for inter-service communication, among teams designing new platforms from scratch, gRPC is quickly gaining steam.  gRPC is based on Google's popular protobuf project, which provides a more compact and efficiently serializable RPC payload.   

Microservices written using gRPC typically include a large number of RPC "methods", all of which are exposed on a single TCP port belonging to the gRPC server.  As a result, a traditional network firewall would either open or close the port of the gRPC server, exposing either all or none of the gRPC methods for a service to each RPC client.  However, Cilium's API-aware filtering enables fine-grain security policies that selectively expose RPC methods to different remote callers, eliminating unnecessary attack surface.  

We have created a Cilium + gRPC "Getting Started Guide" so you can try it out yourself: http://docs.cilium.io/en/latest/gettingstarted/grpc/ .  Building on our tradition of Star Wars-themed demos, this guide explains how the lack of gRPC-aware network security helped the rebels escape from Cloud City during "The Empire Strikes Back".   Check out the video!

As always, we're very interested in your questions and feedback, so don't hesitate to reach out via Twitter (@ciliumproject) or Slack (http://www.cilium.io/slack).   And don't forget to check out the code and star us on Cilium Github .  Happy gRPC-ing!

What Cilium and BPF will bring to Istio

There is a lot of excitement around Istio this week at KubeCon. We are getting pinged multiple times a day now with questions on how exactly Cilium and Istio relate to each other. Istio abstracts away a lot of networking specific complexity and provides visibility and control to application teams. We couldn't agree more with the moving networking to Layer 7 and provide the necessary instruments for efficient operation at the layer where it makes sense: the application protocol.

This blog post serves to answer a simple question: How is Istio related to Cilium? Can I use both together? Will one benefit from the other?

BPF Updates 13

This is issue 13 of the regular newsletter around BPF written by Alexander Alemayhu. It summarizes ongoing development, presentations, videos and other information related to BPF and XDP. It is released roughly once a week.


The v4.15 merge window is open and LWN.net already has a summary on part 1 out. Which contains a BPF section listing some of the new things:

BPF

The user-space bpftool utility can be used to examine and manipulate BPF programs and maps; see this man page for more information.

Hooks have been added to allow security modules to control access to BPF objects; see this changelog for more information.

A new BPF-based device controller has been added; it uses the version-2 control-group interface. Documentation for this feature is entirely absent, but one can look at the sample program added in this commit that uses it.

The highlights since last time

  • New helper function bpf_getsockops to retrieve socket options. supports TCP_CONGESTION for now. The new BPF_SOCK_OPS_BASE_RTT feature significantly improves TCP-NV.
  • It is now possible to attach multiple programs to tracepoint / kprobes / uprobes. The programs will run in sequence. With the change for trace points one application does not exclude others from attaching to the same call.

More interesting topics

  • New helper function bpf_override_function under discussion to allow for error injection via kprobes.
  • BPF runtime finally gets a FAQ section in the kernel's documentation directory.
  • bpftool gets support for dumping JSON.

Presentations

Cilium - Kernel Native Security & DDOS Mitigation for Microservices with BPF

The slides of Cynthia's talk were already in the last issue. Docker has since published the recording as well, definitely worth watching the recording. Fun talk on Cilium, BPF, and Kafka.

Linux Networking Development

Focusing on development areas in the kernel. Also some advice in there for aspiring kernel developers. ;-)

XDP: The Future of Networks

Great introduction to BPF and XDP. With some myth busting and potential improvements.

A Gentle Introduction to [e]BPF - Michael Schubert, Kinvolk GmbH

Good introduction to BPF. Also nice that it shows the structures, links to some tools and verifier.

LISA 17 - Fast and Safe Production Monitoring of JVM Applications with BPF Magic

Focusing on the tracing case with Java but the approaches could still be applied to other environments.

LISA17 Container Performance Analysis

Goes through some of the tools used at Netflix and a lot of other smaller tools for tracing. The emphasis on identifying the bottlenecks sounds good.

LISA17 Linux Performance Monitoring With BPF

Lab session for tracing tools with BCC. This is useful for learning about tracing on Linux. It also answers basic question what is tracepoints, kprobes, uprobes, etc. and what are some of the limitations to dynamic tracing. Looks like a lot of fun.

XDP – eXpress Data Path An in-kernel network fast-path A technology overview

Great introduction to BPF and XDP. Also explains the problems and why it is needed.

In case you missed it

Reports from Netconf and Netdev

LWN.net coverage of the discussions from netconf and all the talks from netdev. All lot of interesting BPF topics in there. Check it out!

security things in Linux v4.14

The security summary contains a section eBPF JIT 32-bit ARM support and seccomp improvements.

SystemTap 3.2 release

SystemTap now has an experimental eBPF backend.

Steven Rostedt proposes different scheme where tracepoints are placed but no trace event. Then on userspace a kernel module have to be loaded and there would be no need to add this to the kernel ABI. Will moving the ABI to a module really solve this problem?

LWN.net coverage of Eric Leblond's talk from Kernel Recipes. The recording was already in the last issue.

Projects

awesome-ebpf

A curated list of awesome projects related to eBPF

k8s-snowflake

Configs and scripts for bootstrapping an opinionated Kubernetes cluster anywhere.

libseccomp

The libseccomp library provides an easy to use, platform independent, interface to the Linux Kernel's syscall filtering mechanism. The libseccomp API is designed to abstract away the underlying BPF based syscall filter language and present a more conventional function-call based filtering interface that should be familiar to, and easily adopted by, application developers.

cbpf-rust

Userspace cBPF interpreter and cBPF to eBPF converter

vltrace

vltrace is a syscall tracing tool which utilizes eBPF - an efficient tracing feature of the Linux kernel.

Random cool note

We blew way past 7Mpps with UDP+XDP. I’m sure you know that already though :)

Patches

Please note that netdev and llvm-commits receive a lot of patches and the list below is not meant to be comprehensive.

LLVM

netdev

Cilium v0.10 & v0.11 Released: Double the Fun - Two Updates in One!

We're happy to announce our 2 recent Cilium releases: v0.10 and v0.11!

This is a brief recap of noteworthy functionality, including the expansion of Network Policy, simplifying deployments, Kubernetes integration updates, and Mesos integration. For the full list of changes, please refer to the Release Notes.

BPF Updates 11

The highlights since last time are

- New helper functions `bpf_perf_read_counter_time` and `bpf_perf_prog_read_time`.
- Initial BPF assembly support in LLVM.
- LRU map lookup improvements.

Linux 4.13 was released last week and net-next closed around the same time. The
last `[GIT] Networking` pull request includes a couple of BPF fixes and so do
the two after the merge window opened up as well. See the dates for all the
details

- [01 Septemper 2017](https://www.spinics.net/lists/netdev/msg453325.html).
- [05 Septemper 2017](https://www.spinics.net/lists/netdev/msg453873.html).
- [09 Septemper 2017](https://marc.info/?l=linux-netdev&m=150493364601151&w=2).

LLVM [5.0.0](http://lists.llvm.org