TL;DR

1. Deploy / Remove buildpacks using CI and IaC. Aim for complete automation
2. Continually update the ‘standard’ buildpacks that come with your CF installation & roll the apps that run on them
3. For custom buildpacks: setup 2 automated cycles for Create and Warn/Remove
4. Define a buildpacks service statement that makes the buildpack cycle clear
5. Avoid app-specific buildpacks wherever possible

 

CF Buildpacks vs. traditional infrastructure

With traditional infrastructure, where server installations are centrally managed, IT Security folks have often put pressure on IT departments to roll out server fixes e.g. new WebSphere fixpacks. This is tricky because IT departments do not have the resources to re-test all the apps against the upgraded binaries.

With Cloud Foundry Buildpacks the scenario becomes simplified:

• Buildpacks are managed by the CF platform teams, each of which lays down specific language runtime binary versions (e.g. Java, Tomcat)
• Application teams can choose from the available buildpacks in CF
• An extension to the platform reports which applications use which buildpack
• IT Security can chase the application teams direct

Standard buildpacks

Some Cloud Foundry implementations such as Pivotal Cloud Foundry (PCF) come with a whole bunch installed covering all the major language runtimes (Java, PHP, Python, Go, Ruby, Node, etc).

You could choose to remove the unused ones to reduce potential security attack vectors, however you will probably have to keep at least Java, Ruby, and Go as these are typically required by CF platform management apps.

A better approach is to leave them in place, keep them updated and report on usage.

A diligent platform operator will want to keep the remaining standard buildpacks up to date, as each buildpack constitutes a potential attack vector. A reusable process for doing this is:

1. Detect upstream buildpack releases from your CF provider

Your CI server can do this for you. If you’re using PCF check network.pivotal.io (pivnet), which is real easy if your CI server is Concourse & you can access the pivnet-resource.

2. Download and update buildpacks

Download the pre-built buildpacks and publish locally to your network. Binary repositories such as Artifactory or Nexus are ideal for this

3. Test the buildpack

Create a new buildpack in a CF environment without any business apps in it (e.g. a lab environment) & push a sample app against it, to prove the pre-built zip file works.

4. Update buildpacks

Update the buildpack in your CF installations without changing the buildpack names, via CI.

5. Repush all the apps that depend on this buildpack

After repushing, run a suite of smoke tests to prove that the platform still functions as you would expect. This will ensure that your core platform features are securely deployed. Any business apps that deployed on it should also be included in this 'rolling' of apps onto the upgraded buildpack, as should any native CF Services that are implemented as application instances.

Buildpack Deployment

Deployment of buildpacks requires Administrator-type access in CF.

The best way to manage it is using a CI job that compares the buildpacks installed against a master list stored in a file or collection of files: adding those that are missing, removing those that are not listed & notifying the community of the changes. This kind of job is idempotent, based on Infrastructure-as-Code (IaC) so it can be re-run on a regular schedule to continuously assure your CF environment buildpacks are up to date. The files should be kept under version management.

As individual buildpack CI jobs complete, they should update their entry in this buildpack definition file. Each entry should state the buildpack name and zip file location.

I advocate updating the file(s) with statuses and deletion dates as older buildpack versions are superseded by new ones, rather than removing them altogether. That way you have a history of what buildpacks were intended for addition and deletion from your CF environments.

By updating the buildpack IaC definition separately from deploying it into environments, you decouple the testing of new buildpacks from their general purpose rollout, which can be handy if CF environments are under change control.

 

Custom Buildpack Creation

You are going to be adding custom buildpacks to your CF installation, for the reasons of: adding enterprise SSL CA certificates, adding custom features, disabling unwanted functionality, custom language binary version requirements, etc. Remember to use TDD for each of the new features you add e.g. adding rspec tests for the Java Buildpack.

Since these are the ones likely to be used by your enterprise applications, I recommend introducing a deployment / deletion cycle for these whereby new buildpack releases are deployed with a new name, and old ones deleted by name.

The key driver behind this cycle will be to upgrade the binary versions that buildpacks install e.g. fixing security vulnerabilities. This is typically driven by a IT Security team who want to push on application security.

Code merging will most likely be manual, so the deployment cycle should take two parts.

CI Part 1:

1. Detect upstream buildpack releases from Github

This is a CI process that checks the relevant github.com repository

2. Merge code down into yours & build

You will have forked the public Github repository before adding your changes to it. Use standard Git techniques to merge the public changes to your own Git repository.

If the merge fails on conflicts, fail this build.

Any red on the CI board caused by a failed merge will notify the CF team that there is action required. After manually code merging & committing back to your custom git repository, CI Part 2 will kick off:

1. Detect code changes in your buildpack source code

2. Build the buildpack

Build processes are specific to each. Remember to run the buildpack unit tests. Each buildpack has a way of declaring their version internally. Ensure your build updates the internal version definition (e.g. config/version.yml for Java)

3. Test the buildpack

Create a new buildpack in CF at the lowest priority & push a sample app against it to test the custom features you have added (e.g. SSL certs) work. Remove the app and buildpack at the end of this.

4. Install buildpacks

Create a new entry in your buildpacks definition file. The separate Deployment CI process will kick off and deploy the buildpack.

 

Custom Buildpack Warning / Deletion

The process for warning / deletion on buildpacks will depend on your prior agreement with app teams. Removing buildpacks on which running apps have been built will cause issues if they restage the app & the buildpack is missing. However, waiting for all app teams to be off old buildpack versions before deleting them can cause a long tail of old buildpacks used by one or two stubborn app teams too busy to upgrade their apps.

I recommend a halfway house: warn users of deletion in advance; delete automatically, taking care to warn specific users who didn’t move off the old buildpack at time of deletion.

Create one CI job to give app teams advanced warning. The job should check the buildpack definition file for new buildpack versions, updating the entries for older buildpacks with an intended deletion date.

Create a second CI job to delete buildpacks automatically from CF after that date, remembering first to query CF for a list of apps that are on the old buildpack & the list of CF users that own the Orgs and Spaces in which these apps run. These lists can be used to notify users of their dependency on a deleted buildpack.

I like this approach because the buildpack removal dates could be adjusted manually in the event of an exception to the normal process.

 

App-specific Buildpacks

In this scenario application teams take responsibility for a copy of buildpack code. Responsibility for keeping up to date with language runtime binary (e.g. JRE) versions transitions to the app team. I have seen them used for cases as trivial as providing secret key material that was forbidden to be stored with ‘application code’.

From experience these are a very bad idea for everything except temporary fixes. Application teams rarely keep the underlying binary versions up to date, so should be used as a last resort only.

Where app teams require custom features above normal buildpack functionality, these should be added into the core (ideally public Github) buildpack code by the CF platform team in a generic way e.g. by adding support for downloading secret keys from a HSM.

 

Buildpack Service Contract

Create and publish a Buildpack Service Contract so that app teams understand the requirement to continuously upgrade buildpacks. An example contract might go something like:

Standard buildpacks

• Standard buildpacks will be upgraded in place without notice
• All apps that depend on them will be rolled onto the new buildpack version
• In the case that a Deploy and Smoke Test CI job is provided, applications will be smoke tested before in a LAB environment before being upgraded

Custom buildpacks

• Old buildpacks will be removed 2 weeks after provision of a later version of the same buildpack
• In the case of a Zero Day or High Severity security vulnerability, removal will be within 1 week
• All PCF Space-level users will be notified via email immediately a new buildpack is available
• At time of removal, the PCF Space-level users of any apps still running on the buildpack will be notified via email
• A versioned definition buildpacks and deployment dates will be made available

Application buildpacks

• The app team owns the buildpack code and is responsible for the binary versions it installs and fixing the security defects that arise

 

Hopefully this explains the cycle of buildpacks through CF.

The only real hands-on part is the merging of upstream code, if any.