ML/AI Brings Promises and Risks

May 16, 2019

I received a message recently where the leader observed that advances in ML/AI in a given field will “make us even more effective at our work.”  While there may be an opportunity there, it brings with it another target for mature and advanced hostile activity.  In global financial services enterprises, we are all using these technologies.  Depending on the nature of your ML/AI application, attack-influenced outputs could have serious negative consequences.  Abuse via ML/AI have been a thing for quite a while.  Machine learning as implemented is vulnerable to ‘wild patterns’ or ‘adversarial machine learning.’

Modern technologies based on pattern recognition, machine learning and data-driven artificial intelligence have been fooled by carefully-perturbed inputs into delivering misleading outputs.  Here is a useful set of illustrations.  Battista Biggio & Fabio Rolia published a more thorough history of this topic here.  If you still can’t picture how this might happen today, see a story in the news yesterday.

If you are using any of the ML/AI-enabled security related platforms or services, you may want to invest in a quick read of the two papers linked above.  If you are actively involved in engineering the use of such a platform, you should consider reviewing the resources from IBM below:

IBM published a suite of ML & classifier attacks and defensive methods, and some additional work on detection, and it remains an active project: &

There are useful links to supporting resources throughout these materials.

As these platforms & services become more important to your organization — and to the success of your customers — pay special attention to the details of your implementation.  We all know that hostile agents are good at manipulating humans so we have invested in a broad spectrum of efforts to resist those attacks and to reduce the impacts upon their success.  We all need to approach ML/AI-enabled technologies and services in an analogous way — and ensure that our vendors & partners are doing the same.


There is a good high-level primer on this topic at:
“Breaking neural networks with adversarial attacks — Are the machine learning models we use intrinsically flawed?”
By Anant Jain, 02-09-2019.

There is an excellent history of this topic at:
“Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning.”
By Battista Biggio & Fabio Rolia

This is not an abstract, niche concern.  See: “Police have used celebrity look-alikes, distorted images to boost facial-recognition results, research finds.”
By Drew Harwell , 05-16-2019


Your Home Tech is Eavesdropping on You

May 6, 2019

Microphone-equipped smart devices from Amazon, Apple, & Google keep copies of what they record — and they probably record more than you think.

Geoffrey Fowler writes about this topic after listening to four years of his Alexa archive. He found thousands of fragments of his home-life from the mundain daily life activities to sensitive conversations.

This can also reach into business activities — Mr. Fowler’s Alexa recorded a friend conducting a business deal.  And those recordings are discoverable in court.

Remember that Alexa’s, Seri, and Google’s Assistant’s ‘wake words’ are imperfectly detected…  You may be recording more than you intended.

You can listen to your own Alexa archive here:

Mr. Fowler also illustrates the the eight steps required to delete Amazon’s recordings from your Echo speaker in the Alexa app.

Other stuff around your home or workplace may also be recording your activities:

  • Your Nest thermostat (by Google) reports back to its servers in 15-minute increments about not only the climate in your house but also whether there’s anyone moving around (as determined by a presence sensor used to trigger the heat).
  • Your Philips Hue-connected lights track every time they’re switched on and off — data the company keeps forever if you connect to its cloud service (which is required to operate them with Alexa or Assistant).
  • Your Chamberlain MyQ garage opener lets the company keep — indefinitely — a record of every time your door opens or closes.
  • Your Sonos speakers track what albums, playlists or stations you’ve listened to, and when you press play, pause, skip or pump up the volume.
  • And the list goes on…


“”Alexa has been eavesdropping on you this whole time.”
By Geoffrey A. Fowler, 05-06-2019

You can tell:
Amazon to delete the data it already has collected at:
Google to change its Assistant recording settings at:
Adjust your Apple Siri recording retention: (...Apple doesn’t give you the ability to say not to store recordings of your audio.)

Facial-Identification Bias and Error – Lesson For AI/ML-Enabled Security Services

February 2, 2019

Six months ago I published a short rant about the potential for material bias and unknown error represented in the many AI/ML-driven security services being pitched to global financial services enterprises. Since that time, and in the most general terms my experiences in this field have been less-than positive. The central themes that I hear from proponents of these services seem to be — “you don’t get it, the very nature of AI/ML incorporates constant improvements,” and “you are just resistant to change.” There seems to be little appetite for investigating any of the design, implementation, and operational details needed to understand whether given services would deliver cost and risk-relevant protections — which should be in the foreground of our efforts to protect trillions of dollars worth of other people’s money. It is not. “Buy in, buster.” Then a quick return to what seems central to our industry’s global workforce — distraction. Ug.

Because of the scale of our operations and their interconnectedness with global economic activity, financial services risk management professionals need to do the work required to make ‘informed-enough’ decisions.

Recent assessments of leading facial-identification systems have shown that some incorporate material bias and error. In a manner analogous to facial recognition technologies, AI/ML-driven security analysis technology is coded, configured, and trained by humans, and must incorporate real potential for material bias and unknown error.

Expense pressures and an enduring faith in technology have delivered infrastructure complexity and attack surfaces that were unthinkable only a few years ago. Concurrently, hostile activity is more diverse and continues to grow in scale. We need to find new ways to deal with that complexity at scale in an operational environment that is in constant (often undocumented) flux. Saying “yes” to opaque AI/ML-enabled event/threat/vulnerability analysis services might be the right thing to do in some situations. Be prepared, though, for that day when your risk management operations are exposed to legal discovery… Will “I had faith in my vendor” be good-enough to protect your brand? That seems like a sizable risk. Bias and error find there way into much of what we do. Attempting to identify and deal with them and their potential impacts have part of global financial services risk management for decades. Don’t let AI/ML promoters eliminate that practice from your operations.

“Bias & Error In Security AI/ML.”

“Amazon facial-identification software used by police falls short on tests for accuracy and bias, new research finds.”
By By Drew Harwell, 01-25-2019

Protect Location Data

January 15, 2019

Location tracking is sensitive business — one loaded with real and perceived risks and injustices.  There are global financial services business use cases where location data can be used to help resist fraud — for example: ‘is the customer present for the requested financial transaction?‘  Do so only with explicit customer consent.

Mobile phones constantly communicate with nearby cell phone towers to help telecom providers know where to route given calls and texts. From this, telecom companies also work out the phone’s approximate location.

Telecom companies doing business in the United States sell access to their customers’ location data, primarily to location aggregators. These data aggregators then sell it to others. Despite what carriers want us to believe, location data is not tightly controlled or carefully regulated. It is not.

Telecoms preach that they protect customer data and that location-based services require user’s notice and consent. That is not their behavior.

Carefully vet the use of location data services in your organizations and then carefully limit access to the resulting data.  In the long run, to do otherwise is a needless risk of brand damage.

Joseph Cox wrote an article on this topic published last week on MotherBoard.  It is worth reviewing to better understand how location services can be misused.

If you already employ these services — review your practices.

I Gave a Bounty Hunter $300. Then He Located Our Phone.” by Joseph Cox

Some Upgrades Are Not Optional – Engineer Them For The Reality of Your Operations

December 5, 2018

Kubernetes enables complexity at scale across cloud-enabled infrastructure.

Like any other software, it also is — from time to time — vulnerable to attack.

A couple days ago I initially read about a CVSS v3.0 ‘9.8’ (critical) Kubernetes vulnerability:

“In all Kubernetes versions prior to v1.10.11, v1.11.5, and v1.12.3, incorrect handling of error responses to proxied upgrade requests in the kube-apiserver allowed specially crafted requests to establish a connection through the Kubernetes API server to backend servers, then send arbitrary requests over the same connection directly to the backend, authenticated with the Kubernetes API server’s TLS credentials used to establish the backend connection.”

The default configuration for a Kubernetes API server’s Transport Layer Security (TLS) credentials grants all users (authenticated and unauthenticated) permission to perform discovery API calls that result in this escalation.  As a result, in too many implementations, anyone who knows about this hole can take command of a targeted Kubernetes cluster. Game over.

Red Hat is quoted as describing the situation as: “The privilege escalation flaw makes it possible for any user to gain full administrator privileges on any compute node being run in a Kubernetes pod. This is a big deal. Not only can this actor steal sensitive data or inject malicious code, but they can also bring down production applications and services from within an organization’s firewall.”

This level of criticality is a reminder about how important it is to engineer-in the ability to perform on-demand Kubernetes infrastructure upgrades during normal business operations. In situations like the one described above, these upgrades must occur without material impact on business operations.  In real business infrastructure that is a serious engineering challenge.

Today is not the day to be asking “how will we upgrade all this infrastructure plumbing in real-time, under real business loads?”

The recent Kubernetes vulnerability is a reminder about how complex is the attack surface of every global financial services enterprise. With that complexity, comes a material obligations to understand your implementations and their operations under expected conditions — one of which is the occasional critical vulnerability that must be fixed immediately.

Oh joy.

Kubernetes Security Announcement – v1.10.11, v1.11.5, v1.12.3 released to address CVE-2018-1002105!topic/kubernetes-announce/GVllWCg6L88
CVE-2018-1002105 Detail
Kubernetes’ first major security hole discovered

MacOS Firewall Bypass Demos

August 9, 2018

“You’re are not going to buy a car and expect it to fly?” Patrick Wardle, Chief Research Officer at Digita Security and founder of Objective-See, describing why he presented some of his research on MacOS firewall bypasses.

That sort of makes sense.  Nobody buys a Mac and expects it to resist attack?

In any case, we all have members of our workforce using Macs for non-trivial business operations.  We need to clearly understand the attack surface and Mac’s resistance to attack.  P.Wardle provides a little help on that exercise in his BlackHat presentation: “Fire & Ice: Making and Breaking macOS Firewalls.”

Tom Spring has a useful summary of the presentation on ThreatPost: “Black Hat 2018: Patrick Wardle on Breaking and Bypassing MacOS Firewalls.”  It is worth a read.  There is no reason for me to echo its content here.


“Fire & Ice: Making and Breaking macOS Firewalls”

“Black Hat 2018: Patrick Wardle on Breaking and Bypassing MacOS Firewalls” By Tom Spring

Cloud Concentration Brings Operational Risks

July 18, 2018

The sales job included a theme that the Internet is so decentralized that anything named cloud simply inherited all the good ‘ilities’ that massive decentralization might deliver. One of the core tenets of that Internet story-telling is about the resiliency that decentralization guarantees — as in: no single outage can cause problems for your cloud-delivered services.

The reality is that story is too often just nonsense. Google, Amazon, and Microsoft control enormous amounts of the infrastructure and operations upon which “cloud” services depend. When any of these three companies has an outage, broad swaths of “cloud-delivered” business services experience outages as well. If you are going to depend on cloud-enabled operations for your success, use care in how you define that success, and how your communicate about service levels with your customers. This is especially problematic for industries where customers have been trained to expect extremely high service levels — global financial services enterprises fall into this category.  Factor this risk into your business & technical plans.

Yesterday, Tuesday, July 17th, Google’s services that provide computing storage and data management tools for companies failed. The customers who depended upon those foundational services also failed. Snapchat and Spotify were two high profile examples, but the outages were far more widespread. Google services that also depend upon the storage and data management tools also failed. It appears that Google Cloud Networking was knocked down as the company reported “We are investigating a problem with Google Cloud Global Load balancers returning 502s for many services including AppEngine, Stackdriver, Dialogflow, as well as customer Global Load Balancers.” This has broad impact because all customers attempting to deliver high quality services depend on load balancers. It appears that this networking issue caused downstream outages like those experienced by Breitbart and Drudge Report.

This was the 5th non-trivial outage this year for Google’s AppEngine. Google’s ComputeEngine has had 7 outages this year, 12 for their Cloud Networking, 8 for their Stackdriver, 3 for Google Cloud Console, 3 for Cloud Pub/Sub, 4 for Google Kubernetes Engine, 2 for Cloud Storage, 4 for BigQuery, 2 for their DataStore, 2 for their Cloud Developer Tools, and 1 reported for their Identity & Security services.
To add insult, it appears that the Google Enterprise Support page may have also not working during the outage on the 17th…

Amazon also experienced service failures that were widely reported the day before (although they are not documented on the company’s service health dashboard).
Microsoft had “internal routing” failures that resulted in widespread service outages for over an hour on the 16th as well.

Plan this reality into your cloud-enabled strategies, architectures, designs, implementations, testing, monitoring, reporting, and contracts.

July 31, 2018 UPDATE:

One of my peers asked about the ability of legacy financial services IT shops to deliver ‘cloud-like’ service levels, arguing that all of our global financial services enterprises have material entrenched IT infrastructure that has service level issues as well.  It seems like a fair reaction, so here is my response:
My motivation for the essay above was driven by several observations:
(1) There are still individuals in our industry who seem to believe there is some magical power available to ‘cloud’ things that will make existing problems and constraints disappear. These individuals make me tired. I tried to highlight a little of cloud vendor’s muggle nature.
(2) The three main cloud vendors I mention — Amazon, Google and Microsoft — provide what I perceive as selective and self-serving views into their service outages. They have outages, short and not-so-short, that are unreported on all of their outage-reporting interfaces available to me. That behavior seems to be designed into their development and infrastructure management practices as well as their sales/marketing practices. When they fail fast, it is on customer’s time and investments (ours), not only on their own (and, I believe, rapid recovery skills do not justify or compensate for any given outage…). I mentioned Amazon on this score, but I my reading helps me believe that all three exhibit this behavior.
(3) We all have ‘knobs’ or ‘levers’ available to us at our corporations to influence our service levels in ways that might help us distinguish ourselves from our competition. My working understanding is that service levels are just an expression of management will. I get it that those responsible for serious profit/loss decision-making have a fiendishly difficult role. That said, if it were important-enough, our leaders would adjust the available management ‘knobs’ in ways that do/can/would deliver whatever was needed — on our systems or those owned & managed by others. I understand that there are a hornet’s nest of competing priorities and trade-offs that make those decisions tricky. I also know that many of those ‘knobs’ are not available to our management teams in analogous ways for most of our ‘cloud’ vendor candidates.

Some argue that there are the right drivers to replace our systems with ‘cloud-enabled’ vendor services in combination with some amount of our code and configuration. To live out that desire, it seems like most of us would need to re-architect much of our business practices and in parallel the stacks of systems that enable them if we were expecting to deliver competitive sets of features and accompanying service levels in the global financial services enterprise arena. A material chunk of that re-architecting would need to compensate for the outage patterns exhibited by the ecosystem of major and minor players involved — and because this is my blog, to compensate for the new information security risk management challenges that path presents as well.

Google Cloud Status Dashboard

Amazon Service Health Dashboard

Azure Status History
Azure Status
Office365 Health

Google Cloud Has Disruption, Bringing Snapchat, Spotify Down
By Mark Bergen, July 17, 2018

Spotify, Snapchat, and more are down following Google Cloud incident (update: fixed)
Jeff Grubb, JULY 17, 2018

Google Cloud Platform fixes issues that took down Spotify, Snapchat and other popular sites
Chloe Aiello, 07-17-2018

[Update: Resolved] Google Cloud has been experiencing an outage, resulting in widespread problems with several services
By Ryne Hager, 07-17-2018

Google Enterprise Support page Outage Reference

%d bloggers like this: