Do Not Use On-Line Services to Encode or Encrypt Secrets

March 17, 2017

I received an excellent reminder about protecting secrets from a developer this morning. His advice included:

In the course of development work, many of us need to encode or encrypt strings.  He had just bumped into a situation where teams were using an Internet-available, public service to base 64 encode OAuth key/secret pairs.  These OAuth “secrets” are used all over the Internet to authenticate against web service interfaces.  Too often they are static/permanent strings — which means that once stolen they are useful to anyone, hostile or otherwise, for long periods of time.  This type of authentication credential must be very carefully protected throughout its entire life-cycle.
[Please stick with me even if you are not familiar with base 64 or OAuth, because this is broadly reusable advice]

The specific site is not really important as it could have been one of thousands of other free data encoding/encrypting sites.

The risk issue is associated with the fact that the “free” encoding service cloud site knows the client’s source IP address (plus other endpoint/user-identifying metadata) and the secrets that the user inputs. Using that information, they can infer (with some confidence) that a given company is using these secrets, and can sometimes also infer what the secrets are used for by the structure of the inputs. Nothing on the Internet is truly free. We need to assume that these sites earn revenue by monetizing what they learn. Cyber-crime is a business, and it is often less expensive to buy information about specific or classes of candidate targets than to independently perform the initial reconnaissance. So we should expect that some percentage of what free sites learn ends up as inputs to cyber-crime planning and activities. In that context, our secrets would not remain secret — and our risks would be elevated. In addition, extruding secrets in this way would also violate company policy at every global Financial Services enterprise.

Lucky for all of us, there are easy alternatives to using Internet-available public services to encode/encrypt our secrets.

Encoding can be as simple as a PowerShell or Python one-liner:

powershell "[convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes(\"mySecret\"))"


python -c "import base64; encoded=base64.b64encode(b'mySecret'); print encoded;"

Or you can use any other development language of choice to easily assemble a utility to encode secrets. This is not technically difficult or especially risky.

Encrypting safely is a greater challenge. Understand your goals first. Once you know what you need to achieve, you can work with a professional to select a cryptosystem and coding/operational processes that should have a chance of meeting those goals. Cryptography can go wrong. Do not attempt to invent your own.

Another Example of How Cloud eq Others Computers

March 2, 2017

I have a sticker on my laptop reminding me that “The cloud is just other people’s computers.” (from StickerMule)  There is no cloud magic.  If you extend your global Financial Services operations into the cloud, it needs to be clearly and verifiably aligned with your risk management practices, your compliance obligations, your contracts, and the assumptions of your various constituencies.  That is a tall order.  Scan the rest of this short outline and then remember to critically evaluate the claims of the hypesters & hucksters who sell “cloud” as the solution to virtually any of your challenges.

Amazon reminded all of us of that fact this week when maintenance on some of their cloud servers cascaded into a much larger 2 hour service outage.

No data breach.  No hack.  Nothing that suggests hostile intent.  Just a reminder that the cloud is a huge, distributed pile of “other people’s computers.”  They have all the hardware and software engineering, operations, and life-cycle management challenges that your staff find in their own data centers.  A key difference, though, is that they are also of fantastic scale, massively shared, and their architecture & operations may not align with global Financial Services norms and obligations.

Amazon reported that the following services were unavailable for up to two and half hours Tuesday Morning (28 Feb, 2017):

  • S3 storage
  • The S3 console
  • Amazon Elastic Compute Cloud (EC2) new instance launches
  • Amazon Elastic Block Store (EBS) volumes
  • AWS Lambda

This resulted in major customer outages.

Here is how Amazon described the outage:

  1. “…on the morning of February 28th. The Amazon Simple Storage Service (S3) team was debugging (a billing system) issue…”
  2. “At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process.”
  3. “Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.”
  4. “The servers that were inadvertently removed supported two other S3 subsystems.”
  5. “One of these subsystems, the index subsystem, manages the metadata and location information of all S3 objects in the region. This subsystem is necessary to serve all GET, LIST, PUT, and DELETE requests.”
  6. “The second subsystem, the placement subsystem, manages allocation of new storage and requires the index subsystem to be functioning properly to correctly operate. The placement subsystem is used during PUT requests to allocate storage for new objects.”
  7. “Removing a significant portion of the capacity caused each of these systems to require a full restart.”
  8. “While these subsystems were being restarted, S3 was unable to service requests.”
  9. “Other AWS services in the US-EAST-1 Region that rely on S3 for storage, including the S3 console, Amazon Elastic Compute Cloud (EC2) new instance launches, Amazon Elastic Block Store (EBS) volumes (when data was needed from a S3 snapshot), and AWS Lambda were also impacted while the S3 APIs were unavailable.”

There is no magic in the cloud. It is engineered and operated by people. Alignment between your corporate culture, your corporate compliance obligations, your contractual obligations, and those of your cloud providers is critical to your success in global Financial Services. If those cloud computers and the activities by armies of humans who manage them are not well aligned with your needs and obligations, then you are simply depending on “hope” — one of the most feeble risk management practices. You are warned — again.

What do you think?

“The embarrassing reason behind Amazon’s huge cloud computing outage this week.”
By Brian Fung, March 2

“Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region.”

%d bloggers like this: