At REI, the Cybersecurity Engineering teams constantly work hand in hand with our application and infrastructure teams. The outdoor analogy that sometimes comes to mind is that the Cybersecurity Engineering team are the belayers, and the application and infrastructure teams are the climbers. When it comes to securing our applications at REI, we want to ensure that developers can continue to climb to new heights, with our Cybersecurity engineers guiding and assisting along the way.

The Journey to Content Security Policy (CSP)

In the beginning of 2022, both the Cybersecurity Engineering team and the Site Reliability Engineering (SRE) team saw that CSP could help solve several of our questions:

• Is there an inventory of 3rd party Javascript that is used on REI.com?
• How do we control what 3rd party Javascript gets loaded on the client browser when someone visits REI.com?
• How do we ensure that when Javascript loads on a client browser, it can’t communicate with inappropriate or nefarious domains?
• How can we better prevent potential data skimming attacks?

While there are other security controls that can help in detecting a compromise of our first party Javascript, the CSP provides an additional defense-in-depth layer, preventing software supply chain attacks where a possible script tag may be compromised.

How Does CSP Work?

Content Security Policy, or CSP for short, declares to the client browser a set of approved sources that can run on a web page. Another way to think about CSP is that it’s a way to make an allow-list of things that your client browser has permissions to load and run.

CSP is implemented using a Content-Security-Policy HTTP response header, and it can help answer questions like:

• Where can I get images from?
• Where can I get stylesheets from?
• Where can I get JavaScript from?
• Can JavaScript run inline in the page?
• Can the page be embedded in another site’s frame?

By enabling CSP, we are being explicit on where our website can embed and load content from. When you load a web page, there’s other things that get embedded in that page, such as stylesheets, JavaScript, images, media, so on a so forth. Very often, these assets come from various other locations – for example, media from YouTube.

CSP is about security, and one of the main reasons why you would want to implement CSP is that it’s meant to detect, reduce, and/or eliminate the risk of threats such as cross-site scripting, packet sniffing, and other data injection attacks.

At a high level, when a web browser goes to www.rei.com, they receive a response from our web application that includes a response header of Content-Security-Policy along with a value of directives. These directives have many functions and basically gives instructions to the client browser on what can be loaded. If something is not specified in a directive, it will be blocked by the client browser.

There are a lot more details into exactly what each directive does and, to make things more complicated, some browsers may interpret or respect directives differently. As we were writing the policy, we found that the Mozilla Developer Network’s CSP page very helpful.

Things We Learned Along the Way

We knew early on that we would have to an exhaustive discovery of all the third party Javascript being loaded on REI.com and verifying internally that someone had a relationship with that third party vendor. To gather all that data we relied on using the Content-Security-Policy-Report-Only header at first. This header basically places the policy in a “monitor” mode and will not block Javascript being loaded. However, it will still send the results of what it would have blocked. This helped us in continually iterating over the policy until we were confident that we would not block any legitimate scripts.

We also needed a way to collect and analyze the data. By using the report-uri directive we were able to send any CSP violations to a collection endpoint for further analysis. As you can imagine, there can be a lot of reports coming from client browsers going to our site so having a way to organize and visualize the data was really important. We ultimately accomplished this by using a third party vendor that specialized in organizing all the CSP data for us.

After all that, we finally had a workable CSP to push into our production environment. The next step was to find a process to maintain our CSP.

CSP Challenges

One of the challenges with how we did CSP initially was that our policy directives were not in a formal source control system. This meant that whenever we had to make any changes to our CSP directives, we had to do the following:

1. Copy the one-line CSP values directly from our CDN configuration
2. Paste that line onto a text editor
3. Make the changes while still maintaining the single-line layout
4. Paste the modified line back into the CDN configuration
5. Hope that we didn’t break CSP

Solutions

To introduce source control, we simply copied and pasted the single-line CSP header from our configuration and into a text file that we checked into git. This solution worked okay at first, but after a few times having to update the CSP we quickly understood that we need a better process — specifically around managing the single-lined CSP header.

So, we converted the one-line into YAML! YAML is a data serialization language, and we chose that over other alternatives mainly due to YAML’s readability factor. The format of a YAML file is more human-readable and easier to manage than something like JSON.

Let’s take this CSP header for example:

frame-ancestors 'self' https://www.coolwebsite.com; default-src 'self' https://*.rei.com; script-src 'self' 'unsafe-eval' blob: https://www.coolwebsite.com https://*.another-website.com; img-src data: *; report-uri https://cspissues.com

After giving it the YAML treatment, the line above would look like this:

frame-ancestors:
- 'self'
- 'https://www.coolwebsite.com'

default-src:
- 'self'
- 'https://*.rei.com'

script-src:
- 'self'
- 'unsafe-eval'
- 'blob:'
- 'https://www.coolwebsite.com'
- 'https://*.another-website.com'

img-src:
- 'data:'
- '*'

This version is easier on the eyes don’t you think? We converted our gigantic CSP header into YAML and developed a script that converts the YAML file to the single-lined CSP header, and Voila! Our new process with these improvements now looks like this:

1. Make a change to the CSP YAML file and create a pull request
2. Once approved, merge the pull request and generate the single-lined CSP header value using the script
3. Replace current CSP header defined in our CDN configuration with the newly created single-lined CSP header value
4. Go on a hike to celebrate!

Conclusion

We now have a more readable, source control friendly, easier to manage version of CSP that even non-technical personnel can make changes if they wish. This collaboration between the Cybersecurity Engineering team and SRE is a perfect example of people from different expertise coming together to do more than they can on their own, fufulling one of REI’s core values: “We go further together!”