SolarWinds-Enabled Hacks Widespread

Hostile actors associated with Russian cyber-security organizations used SolarWinds Orion technology to enable unauthorized long-running elevated rights access throughout the U.S. government and as many as hundreds of the Fortune 500 corporations. This access may have included the Office of the President of the United States.

There is no reason for me to copy the operational details here. There are some good write-ups in the REFERENCES section below.

I just wanted to add to their content with this abuse case:

These hostile actors are getting a lot of attention for data & secrets exfiltration. In global financial services enterprises, we move trillions of dollars a day. These hostile actors were able to acquire elevated rights credentials and move laterally for months. They had enough time to figure out the cash management, account management, portfolio management, and back room accounting processes as well as the chains of approvers required to authorize the maintenance of external target accounts and authorizations for the movement of funds/securities. If so motivated, it seems likely they could have moved large amounts of the financial assets for which we are responsible to target accounts of their choosing. If this did not happen, financial services organizations dodged a big one.

In that case, it was only ‘luck’ that protected the financial services industry. Luck is a terrible risk management tool/technique. This hack is a loud signal that our resistance to and detection of attacks needs to be a lot better than it is today. The FireEye and Krebs references below include the types of details that support changes that will help fill some of that gap.

REFERENCES:

“U.S. Treasury, Commerce Depts. Hacked Through SolarWinds Compromise.” By Brian Krebs, 14 Dec 2020.
https://krebsonsecurity.com/2020/12/u-s-treasury-commerce-depts-hacked-through-solarwinds-compromise/

“Highly Evasive Attacker Leverages SolarWinds Supply Chain to Compromise Multiple Global Victims With SUNBURST Backdoor.” By FireEye, 13 December 2020.
https://www.fireeye.com/blog/threat-research/2020/12/evasive-attacker-leverages-solarwinds-supply-chain-compromises-with-sunburst-backdoor.html

“Russian government hackers are behind a broad espionage campaign that has compromised U.S. agencies, including Treasury and Commerce.” By Ellen Nakashima and Craig Timberg, 13 Dec 2020.
https://www.washingtonpost.com/national-security/russian-government-spies-are-behind-a-broad-hacking-campaign-that-has-breached-us-agencies-and-a-top-cyber-firm/2020/12/13/d5a53b88-3d7d-11eb-9453-fc36ba051781_story.html

“Russian Hackers Broke Into Federal Agencies, U.S. Officials Suspect,” By David E. Sanger, 13 Dec 2020.
https://www.nytimes.com/2020/12/13/us/politics/russian-hackers-us-government-treasury-commerce.html

“Suspected Russian hackers spied on U.S. Treasury emails – sources.” By Christopher Bing, 13 Dec 2020.
https://www.reuters.com/article/us-usa-cyber-treasury-exclsuive/exclusive-u-s-treasury-breached-by-hackers-backed-by-foreign-government-sources-idUSKBN28N0PG

17 Dec 2020 Addition:
“Sunburst Backdoor: A Deeper Look Into The SolarWinds’ Supply Chain Malware.” By Sergei Shevchenko, 15 Dec 2020 https://blog.prevasio.com/2020/12/sunburst-backdoor-deeper-look-into.html

Google Images Distributing Malware

The way Google links to images in search results was being actively abused by criminals to distribute malware. In support of that activity, there were networks of Twitter bots to push the ranking of malware delivery sites.

With nearly every corporate workforce using infrastructure having an increased attack surface and workers away from the traditional workplace, it seems like the risks associated with this activity are elevated.

There is an interesting outline at: “Distributing malware with Google images, service workers and vegan Twitter bots.” by Tom Forbes, 2019-12-15. https://tomforb.es/distributing-malware-with-google-images-service-workers-and-vegan-twitter-bots/

Felony Charges for Uber Security Chief

Corporate lawyers & leaders may wish to define-away data breaches rather than deal with the reporting and impacts that may follow. They may also take more aggressive measures to avoid revealing data breaches — both proven and potential/probable. That has always been an elevated risk behavior.

Corporate teams continue to expose critically-important access keys and tokens on source code repositories like Github, BitBucket, GitLab and others. These keys can be used to access corporate virtual servers, storage, databases and more in AWS, GCS, Azure, and beyond. Dealing with this scenario can be tricky and may involve difficult ethical issues. Because it remains a regular occurrence across global corporate enterprises, one would expect that leaders would be better prepared for this situation.

Revisiting an Uber breach might be informing…

Uber tried to re-define one of their breach incidents in 2016 through a bug bounty program. Paying hackers $100K and asking them to nondisclosure agreements. Later Brandon Glover and Vasile Mereacre pleaded guilty to the hack in U.S. Federal Court.

Glover and Mereacre breached Uber’s security controls using keys they found available on GitHub — where Uber employees had left them unprotected. The keys allowed the attackers to access Uber’s Amazon web servers, where it stored more than 50 million customer and driver accounts, including a pile of driver’s license numbers. Uber told their workforce that it was temporarily shutting down access to Github in order to hunt for and fix any additional access key (or key access) issues.

Joe Sullivan, Uber’s fired (in 2017) security chief was charged in U.S. District Court in San Francisco today with attempting to conceal a data breach that exposed the email addresses and phone numbers of 57 million drivers and passengers as well as drivers license numbers for about 600,000 more individuals.

The felony charges were related to the failure to promptly disclose the breach to employee and consumer victims.

Joe Sullivan led cybersecurity efforts at Facebook before becoming Uber’s chief security officer in 2015, and is now CISO at Cloudflare.

These behaviors have hard costs. To date Uber has paid out $150 million dollars for their behavior surrounding the 2016 breach.

Is your leadership ready to deal effectively & ethically with scenarios like these?

REFERENCES:
“Former Uber Security Chief Charged With Concealing Hack.” By Kate Conger, NYT, 2020-08-20
https://www.nytimes.com/2020/08/20/technology/joe-sullivan-uber-charged-hack.html

“Inside Uber’s $100,000 Payment to a Hacker, and the Fallout.” By Nicole Perlroth and Mike Isaac, NYT, 2018-01-18
https://www.nytimes.com/2018/01/12/technology/uber-hacker-payment-100000.html

New Open Source Static Code Security Analysis Tool for Python

Last week Facebook released Pysa: A new open-sourced a static code security analysis tool for Python code.

While it does not detect common conditional logic like ‘if not user_has_permission()’ it follows data flows through Python programs with some rigor and may be a useful amendment to your current Python static code security analysis processes.

There is an excellent overview of Pysa at: https://engineering.fb.com/security/pysa/

Over-the-Air Mobile Hack

An iOS mobile device used by a Moroccan journalist appears to have been attacked using a ‘network-only’ approach to delivering NSO Group’s “Pegasus” spyware.

This is an attack vector that is difficult to defend against. High value personnel (can you transfer $150M U.S?) doing business in areas with an elevated risk of cyber attack should be factoring this attack vector into their defenses.

Illustration from The Star

Read the whole story by Marco Chown Oved at https://www.thestar.com/news/canada/2020/06/21/journalists-phone-hacked-by-new-invisible-technique-all-he-had-to-do-was-visit-one-website-any-website.html

REFERENCES:
“Journalist’s phone hacked by new ‘invisible’ technique: All he had to do was visit one website. Any website.”
By Marco Chown Oved, Sun., June 21, 202
https://www.thestar.com/news/canada/2020/06/21/journalists-phone-hacked-by-new-invisible-technique-all-he-had-to-do-was-visit-one-website-any-website.html

All Attack Vectors Remain Relevant

ESET Cybersecurity researchers from said yesterday that they have disabled parts of the “VictoryGate” botnet command & control infrastructure.  It had been directing the activity of a malware botnet of at least 35,000 compromised Windows systems.  The hostile agents were using the Windows endpoints to mine Monero cryptocurrency since May 2019.  Most of the exploited endpoints are located in Latin America, with Peru accounting for 90% of them.  These hosts were owned by both public and private sectors organizations, including financial institutions.

ESET researchers confirmed that USB drives were the main or only malware attack and propagation vector.

“The victim receives a USB drive that at some point was connected to an infected machine. It seemingly has all the files with the same names and icons that it contained originally. Because of this, the contents will look almost identical at first glance” …”When an unsuspecting user “opens” (i.e. executes) one of these files, an AutoIt script will open both the file that was intended, in addition to the initial module, both hidden by VictoryGate in a hidden directory”…

Replication through Removable Media has been an initial access path for attackers for as long as there has been removable media.

Financial services enterprises need to remain vigilant in their efforts to police their entire attack surface — including the boring, routine portions like USBs & USB port usage.

I think that the detailed ESET write-up is worth a read if you have any responsibility for attack surface management, or if you have a role in incident response.

REFERENCES:

“Following ESET’s discovery, a Monero mining botnet is disrupted.” By Alan Warburton, 23 Apr 2020. https://www.welivesecurity.com/2020/04/23/eset-discovery-monero-mining-botnet-disrupted/

Capital One Concerns Linked To Overconfidence Bias?

Earlier this week on July 29, FBI agents arrested Paige “erratic” Thompson related to downloading ~30 GB of Capital One credit application data from a rented cloud data server.

In a statement to the FBI, Capital One reported that an intruder executed a command that retrieved the security credentials for a web application firewall administrator account, used those credentials to list the names of data folders/buckets, and then to copy (sync) them to buckets controlled by the attacker.  The incident appears to have affected approximately 100 million people in the United States and six million in Canada.

If you just want to read a quick summary of the incident, try “For Big Banks, It’s an Endless Fight With Hackers.” or “Capital One says data breach affected 100 million credit card applications.”

I just can’t resist offering some observations, speculation, and opinions on this topic.

Since as early as 2015 the Capital One CIO has hyped their being first to cloud, their cloud journey, their cloud transformation and have asserted that their customers data was more secure in the cloud than in their private data centers.  Earlier this year the company argued that moving to AWS will “strengthen your security posture” and highlighted their ability to “reduce the impact of compliance on developers” (22:00) — using AWS security services and the network of AWS security partners — software engineers and security engineers “should be one in the same.”(9:34)

I assume that this wasn’t an IT experiment, but an expression of a broader Capital One corporate culture, their values and ethics.  I also assume that there was/is some breakdown in their engineering assumptions about how their cloud infrastructure and its operations worked.  How does this happen?  Given the information available to me today, I wonder about the role of malignant group-think & echo chamber at work or some shared madness gripping too many levels of Capital One management.  Capital One has to have hordes of talented engineers — some of whom had to be sounding alarms about the risks associated with their execution on this ‘cloud first‘ mission (I assume they attempted to communicate that it was leaving them open to accusations of ‘mismanaging customer data’, ‘inaccurate corporate communications,’ excessive risk appetite, and more).  There were lots of elevated risk decisions that managers (at various levels) needed to authorize…

Based on public information, it appears that:

• The sensitive data was stored in a way that it could be read from the “local instance” in clear text (ineffective or absent encryption).
• The sensitive data was stored on a cloud version of a file system, not a database (weaker controls, weaker monitoring options).
• The sensitive data was gathered by Capital One starting in 2005 — which suggests gaps in their data life-cycle management (ineffective or absent data life-cycle management controls)
• There were no effective alerts or alarms announcing unauthorized access to the sensitive data (ineffective or absent IAM monitoring/alerting/alarming).
• There were no effective alerts or alarms announcing ‘unexpected’ or out-of-specification traffic patterns (ineffective or absent data communications or data flow monitoring/alerting/alarming).
• There were no effective alerts or alarms announcing social media, forums, dark web, etc. chatter about threats to Capital One infrastructure/data/operations/etc. (ineffective or absent threat intelligence monitoring & analysis, and follow-on reporting/alerting/alarming).
• Capital One’s conscious program to “reduce the compliance burden that we put on our developers” (28:23) may have obscured architectural, design, and/or implementation weaknesses from Capital One developers (a lack of security transparency, possibly overconfidence that developers understood their risk management obligations, and possible weaknesses in their secure software program).
• Capital One ‘wrapped’ a gap in IAM vendor Sailpoint’s platform with custom integrations to AWS identity infrastructure (16:19) (potentially increasing the risk of misunderstanding or omission in this identity & access management ‘plumbing’).
• There may have been application vulnerabilities that permitted the execution of server side commands (ineffective input validation, scrubbing, etc. and possibly inappropriate application design, and possible weaknesses in their secure code review practices and secure software training).
• There may have been infrastructure configuration decisions that permitted elevated rights access to local instance meta-data (ineffective configuration engineering and/or implementation).
• There must be material gaps or weaknesses in Capital One’s architecture risk assessment practices or in how/where they are applied, and/or they must have been incomplete, ineffective, or worse for a long time.
• And if this was the result of ‘designed-in‘ or systemic weaknesses at Capital One, there seems to be room for questions about their SEC filings about the effectiveness of their controls supportable by the facts of their implementation and operational practices.

In almost any context this is a pretty damning list.  Most of these are areas where global financial services enterprises are supposed to be experts.

Aren’t there also supposed to be internal systems in place to ensure that each financial services enterprise achieves risk-reasonable levels of excellence in each of the areas mentioned in the bullets above?  And where were the regulations & regulators that play a role in assuring that it the case?

How does an enormous, heavily-regulated financial services enterprise get into a situation like this?  There is a lot of psychological research suggesting that overconfidence is a widespread cognitive bias and I’ve read, for example, that it underpins what is sometimes called ‘organizational hubris,’ which seems like a useful label here.   The McCombs School of Business Ethics Unwrapped program defines ‘overconfidence bias’ as “the tendency people have to be more confident in their own abilities than is objectively reasonable.”  That also seems like a theme applicable to this situation.  Given my incomplete view of the facts, it seems like this may have been primarily a people problem, and only secondarily a technology problem.  There is probably no simple answer…

Is the Capital One case unique?  Could other financial services enterprises be on analogous journeys?

REFERENCES:
“Capital One Data Theft Impacts 106M People.” By Brian Krebs. https://krebsonsecurity.com/2019/07/capital-one-data-theft-impacts-106m-people/
“Why did we pick AWS for Capital One? We believe we can operate more securely in their cloud than in our own data centers.” By Rob Alexander, CIO, Capital One, https://aws.amazon.com/campaigns/cloud-transformation/capital-one/ and https://youtu.be/0E90-ExySb8?t=212
“For Big Banks, It’s an Endless Fight With Hackers.” By Stacy Cowley and Nicole Perlroth, 30 July 2019. https://www.nytimes.com/2019/07/30/business/bank-hacks-capital-one.html
“Capital One says data breach affected 100 million credit card applications.” By Devlin Barrett. https://www.washingtonpost.com/national-security/capital-one-data-breach-compromises-tens-of-millions-of-credit-card-applications-fbi-says/2019/07/29/…
“AWS re:Inforce 2019: Capital One Case Study: Addressing Compliance and Security within AWS (FND219)” https://youtu.be/HJjhfmcrq1s
“Capital One Data Theft Impacts 106M People.” https://krebsonsecurity.com/2019/07/capital-one-data-theft-impacts-106m-people/
“Frequently Asked Questions.” https://www.capitalone.com/facts2019/2/
Overconfidence Bias defined: https://ethicsunwrapped.utexas.edu/glossary/overconfidence-bias
Scholarly articles for cognitive bias overconfidence: https://scholar.google.com/scholar?hl=en&as_sdt=1,16&as_vis=1&q=cognitive+bias+overconfidence&scisbd=1
“How to Recognize (and Cure) Your Own Hubris.” By John Baldoni. https://hbr.org/2010/09/how-to-recognize-and-cure-your

 

Spring Boot & jackson-databind Frustration

​If you use Spring Boot framework, you will likely bump into ‘Unsafe Deserialization’ issues highlighted in your static code security analysis results.  Dealing with this type of vulnerability is one of those issues that tends to be one of the more labor intensive for software development teams.

A remote, anonymous, arbitrary code execution vulnerability in the open source component ‘jackson-databind’ is the most common root cause for these issues because spring-boot-starter-actuator uses it as a dependency.  ‘Unsafe Deserialization’ is typically addressed by upgrading com.fasterxml.jackson.core:jackson-databind — but because it is tightly-coupled with spring-boot-starter-actuator, that is problematic.  If your application tolerates it, the quickest fix is to upgrade Spring-Boot… to a version that uses jackson-databind version 2.9.8 (or above [as of today]).  The ‘available’ versions of ‘safe-enough‘ Spring Boot keep shrinking.  The key challenge is jackson-databind’s use of a black-list to resist specific attack payloads.  Every time there is a new attack gadget released, jackson-databind goes from ‘safe-enough’ to CVSSv3 10 (whole-house-fire) overnight.  Because versioning Spring boot… components requires significant effort, there is a lag between any new jackson-databind release and new Spring-boot releases that incorporate it.

If the application (the whole Spring-enabled stack) under evaluation does not employ beans nor does it expose any listening TCP ports (& no RMI, JMSInvoker, HTTPInvoker, etc., and you find no use of readObject(), readObjectNoData(), readResolve(), or readExternal() either), and you can produce some evidence of that, then that application may not be vulnerable.  It can be hard to prove a negative, and because of the depth & complexity of some Spring-boot-enabled applications (outside of the code that you write) that threshold can be a high bar.  So,…circle back to “If your application tolerates it, the quickest fix is to upgrade spring-boot-starter-actuator to a version that uses Jackson-databind version 2.9.8 (or above).”

REFERENCES:

CVE-2017-17485: FasterXML jackson-databind through 2.8.10 and 2.9.x through 2.9.3 allows unauthenticated remote code execution because of an incomplete fix for the CVE-2017-7525 deserialization flaw. This is exploitable by sending maliciously crafted JSON input to the readValue method of the ObjectMapper, bypassing a blacklist that is ineffective if the Spring libraries are available in the classpath. FROM: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-17485

Variants of this issue have been appearing and reappearing since 2011 (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2011-2894).

Also:
https://snyk.io/vuln/SNYK-JAVA-COMFASTERXMLJACKSONCORE-32111
https://nvd.nist.gov/vuln/detail/CVE-2018-7489
https://nvd.nist.gov/vuln/detail/CVE-2017-7525
https://fortiguard.com/encyclopedia/ips/46004

Some Upgrades Are Not Optional – Engineer Them For The Reality of Your Operations

Kubernetes enables complexity at scale across cloud-enabled infrastructure.

Like any other software, it also is — from time to time — vulnerable to attack.

A couple days ago I initially read about a CVSS v3.0 ‘9.8’ (critical) Kubernetes vulnerability:

“In all Kubernetes versions prior to v1.10.11, v1.11.5, and v1.12.3, incorrect handling of error responses to proxied upgrade requests in the kube-apiserver allowed specially crafted requests to establish a connection through the Kubernetes API server to backend servers, then send arbitrary requests over the same connection directly to the backend, authenticated with the Kubernetes API server’s TLS credentials used to establish the backend connection.”

The default configuration for a Kubernetes API server’s Transport Layer Security (TLS) credentials grants all users (authenticated and unauthenticated) permission to perform discovery API calls that result in this escalation.  As a result, in too many implementations, anyone who knows about this hole can take command of a targeted Kubernetes cluster. Game over.

Red Hat is quoted as describing the situation as: “The privilege escalation flaw makes it possible for any user to gain full administrator privileges on any compute node being run in a Kubernetes pod. This is a big deal. Not only can this actor steal sensitive data or inject malicious code, but they can also bring down production applications and services from within an organization’s firewall.”

This level of criticality is a reminder about how important it is to engineer-in the ability to perform on-demand Kubernetes infrastructure upgrades during normal business operations. In situations like the one described above, these upgrades must occur without material impact on business operations.  In real business infrastructure that is a serious engineering challenge.

Today is not the day to be asking “how will we upgrade all this infrastructure plumbing in real-time, under real business loads?”

The recent Kubernetes vulnerability is a reminder about how complex is the attack surface of every global financial services enterprise. With that complexity, comes a material obligations to understand your implementations and their operations under expected conditions — one of which is the occasional critical vulnerability that must be fixed immediately.

Oh joy.

REFERENCES:
Kubernetes Security Announcement – v1.10.11, v1.11.5, v1.12.3 released to address CVE-2018-1002105
https://groups.google.com/forum/#!topic/kubernetes-announce/GVllWCg6L88
CVE-2018-1002105 Detail
https://nvd.nist.gov/vuln/detail/CVE-2018-1002105
Kubernetes’ first major security hole discovered
https://www.zdnet.com/article/kubernetes-first-major-security-hole-discovered/
CVE-2018-1002105
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-1002105

Facebook Identity Token Thefts Result in Breach

Facebook’s VP of Product Management, Guy Rosen said today that a vulnerability in the company’s  “View As” feature that enabled attackers to steal to user’s access tokens.  These tokens are presented to Facebook infrastructure in ways that allow users to remain authenticated and able to interact with their accounts over multiple sessions. Once in an attacker’s possession, these tokens would permit attackers to impersonate actual users and enable Facebook accounts takeover.  Facebook’s reported that the current risk mitigation is their invalidating at least 40 million authentication tokens and temporarily turning off the “View As” feature while they “conduct a thorough security review.”  He went on:

“This attack exploited the complex interaction of multiple issues in our code. It stemmed from a change we made to our video uploading feature in July 2017, which impacted “View As.” The attackers not only needed to find this vulnerability and use it to get an access token, they then had to pivot from that account to others to steal more tokens.”

Mr. Rosen wrote that Facebook engineers learned of this vulnerability three days ago on Tuesday, September 25 and that almost 50 million Facebook accounts were affected.

Writing safe-enough user-facing software that must interact with a complex ecosystem of applications and APIs is a serious challenge for all of us.  In that context, this latest Facebook vulnerability should be a cautionary tale for all organizations implementing apps and APIs incorporating persistent token-based authentication.  Could any of us market our way out of a situation where 40 or 50 million of customers, marketers, and other partners were vulnerable to account takeover?

Too many architects and developers seem risk-inappropriately infatuated by pundit pronouncements about authentication fatigue and the “best practice” of extending any given authentication across multiple sessions using persistent tokens.  At our scale (a trillion dollars U.S. AUM each, give or take a few hundred billion), global financial services enterprises ought to be able to reason our way through the fog of happy-path chatter to engineer session management practices that meet the risk appetites of our various constituencies.

REFERENCES:

“Security Update.” By Guy Rosen, VP of Product Management
https://newsroom.fb.com/news/2018/09/security-update/