envoy-cmake: Adapting YAStack build system for regular envoy
Building vanilla envoy using cmake
EnvoyCon and KubeCon in Seattle 2018
We used a Lyft to get to KubeCon/EnvoyCon from the airport in Seattle. Being closely associated with Envoy made us think of Lyft over other services. However every time we take a Lyft, we ask the driver why they choose to drive for Lyft. And every time, there is an interesting story/perspective behind it.
The conference is one of the best we have attended. So, one may ask, what makes a conference great? For us, it was the attendees. We found a healthy mix of vendors showcasing solutions and actual users using the technology.
Envoy’s popularity and roadmap
Even more exciting was how widespread acceptance Envoy has gained. What stood out is, that it just took two years for Envoy to get to this level of popularity. This is quiet impressive compared to other open source proxies.
What makes Envoy so successful is also the great team behind it. It‘s a very well managed and extremely well coordinated project. This was also apparent from the roadmap and several initiatives around Envoy going forward.
For those invested in Envoy or want to know more, we strongly recommend visiting the kubecon/envoycon website/YouTube channel to get a feel of the conference.
Using CMake for Envoy builds
While we did not present YAStack at EnvoyCon, it did come up in conversations. What also came up is the wish to use cmake to build envoy. There are people incorporating envoy in the projects they undertake. They wanted to use cmake with envoy for several reasons:
- Their existing build system used CMake
- A library they were trying to integrate with used CMake
- They were more familiar with CMake and less familiar with bazel.
Adapting YAStack build system
YAStack uses CMake to build the project. From a code perspective, YAStack first compiles dpdk and f-stack. It then compiles envoy and links it with dpdk and f-stack libraries.
YAStack uses CMake and it wasn’t difficult for us make changes to this build system. Since there were multiple requests for this, we thought of creating a repository for building vanilla envoy using cmake. It would help avoid effort duplication in the community.
We have added a project envoy-cmake on github that builds regular envoy using cmake. There are some rough edges (static binary for envoy, building containers etc.) which we’ll get to eventually, but the basic build works. Here is a small description of how the build works and different components.
CMake build hierarchy
The top level CMakeLists.txt is located in ‘envoy’ directory. It includes all the sub-directories that also have a CMakeLists.txt file.
Project naming for these sub-directories/sub-projects is automatic. This is achieved using CMakeSettings.txt file. Every CMakeLists.txt in sub-folders, includes a CMakeSettings.txt. This file is responsible for naming the project. It derives the project name using the sub-directory it belongs to. For example, project under envoy/source/extensions is named as envoy-source-extensions. All the include paths are maintained in CMakeIncludes.txt and are included in every CMakeLists.txt file.
The api (protos) is the first thing that gets compiled. The proto files have dependencies with a directory structure it expects. To achieve this, we first create directories and copy dependent proto files from different locations. The directories and proto files are setup and copied under the build/protos directory.
The generation of files is done using the envoy/scripts/generate_pb.py script. It takes proto files as input from build/protos directory and outputs the generated files under build/include directory. Next, these generated files are compiled into a library. The CMakeLists.txt under envoy/api achieves this.
After the protos/api library is built, the external dependencies (under envoy/external) are compiled. Next is the envoy source code (under envoy/source). And then the tests (envoy/test). The test compilation invokes another hierarchy of CMake files to compile test components.
Enhancements
We currently have the following enhancements in mind (in no particular order).
- Use cmake’s native support for protobuf
- Improve dependency tracking across projects
- Create static binary for envoy-source-exe
- Build container images
- Using repositories.bzl file for hint, download specific version of pre-created tar/archive files for dependent library (under envoy/external). Right now they are all being compiled.
- Start using pre-compiled libraries to improve build time.
- Use git sub-modules to fetch and update dependencies
Troubleshooting
- The build process assumes that protoc compiler is available. To use a compatible version, build/install the one under envoy/external/protobuf-3.5.0
- The build process also assumes that the protoc-gen-validate binary is installed. We have verified the build with protoc-gen-validate binary built from source tagged at tag v0.0.6
- If the build is stuck while compiling external/nghttp2–1.30.0, use ./configure — enable-lib-only to build only libraries for nghttp2
Conclusion
We have come to use cmake extensively for all our projects and find it as a very versatile tool for our build system. We hope this project will help others in the community too.
If you can think of a specific enhancement that would help, or want us to prioritize an enhancement over others, please reach out to us using the contact form on yastack.io. Or drop us a line and say hello, we enjoy hearing back from the community.
