I was researching ransomware recently, and one of the solutions that both academic researchers and people working in anti-malware product labs told me would help prevent such attacks was a way to prevent unknown software from launching on your computer.
Ransomware works in “userspace,” or the set of documents and applications that a user has permission to run and modify, rather than requiring special permission to dig down into the system level like more insidious hijacking malware. It doesn’t even typically require remote interaction. Once the payload is installed through some subterfuge—as simple as convincing someone to install a Trojan Horse—ransomware starts encrypting user files. Double-click an encrypted file, and you’re told how to pay a ransom to get the decryption key.
There are millions of variants of ransomware derived from many dozens of distinct code families, because the malware is so easy to obtain, modify, and distribute. Some changes merely tell someone to pay (via Bitcoin) to a different address, while others may contain different code.
But what everyone I spoke to suggested is that a desktop operating system shouldn’t simply run any application that launches without alerting the user, nor any additional behind-the-scene programs—Unix utilities and the like—that it might spawn. (Another way to discover ransomware at work is to monitor for specific unusual file behavior, like a single program opening massive numbers of files with disparate file types.)
Most people only routinely run certain applications on a regular basis. Something new only comes when you install new software or upgrade an operating system. Apple cryptographically signs its apps and lets developers sign theirs, distributed through the Mac App Store or directly to users. Those signatures both verify an app and identify it uniquely behind its visible name.
While OS X almost always requires that you enter an administrative password to use a software installer, apps that can be dragged from a disk image or download directly and launched only alert you that they haven’t been run before or if the app hasn’t been signed by Apple-provided developer certificate. (See this recent column for more details on signed apps.)
Without using parental controls on an account, there’s no way to keep any arbitrary app from launching, especially when the software separately launches command-line utilities or scripts invisibly. This is why Santa coming to town could be a big help in layering in additional protection.
The Santa clause
Google operates a lot of Macs for its employees, and its Macintosh Operations Team maintains a public repository of unsupported code that other Mac admins might find useful. A project called Santa has a lot of potential; it’s currently still in development, and the folks creating it suggest you really know what you’re doing.

Santa detects the blacklisted “yes” app (an old Unix utility) is trying to run.
While anti-malware software can blacklist software from running and identify certain kinds of malicious behavior, it’s usually limited to a blacklist of bad players identified by “signatures” that represent a unique lump of software code. Malware authors now release huge numbers of variants to help bypass signature checking.
Santa is a whitelist/blacklist program that will eventually be able to talk to a synchronization server on a network, or, presumably, an Internet-accessible master list. It can be run in two modes. In “monitor” mode, it records information about app launches, but only blocks specifically blacklisted apps. Santa relies on a fingerprint of the app (a mathematical operation calculated against its binary code) to prevent apps from running.

In “monitor” mode, Santa logs apps’ signing certificates and other data.
In “lockdown” mode, Santa only allows whitelisted apps to run, whether they’re system level or launched by users. In this mode, the utility can be set to allow binaries, or apps that are signed by an approved developer signature, which allows updated apps to run without requiring that they be separately whitelisted. You can also blacklist a directory, such as the /Users/
directory, to prevent software not installed in the Applications or System folders from running.
The team suggests running it in “monitor” mode for a while to see what apps are routinely used and then develop a whitelist from that before switching to “lockdown.” No tools exist yet to convert the log into a whitelist, but those will certainly come.
Santa isn’t ready for deployment, and it may be best for large organizations, but I can also see how publicly minded Mac users or an organization with trust behind it could assemble whitelists for synchronization or as starting points for organizations without IT departments.
Meet the Little Flocker
In addition to Santa, security researcher Jonathan Zdziarski has been working on Little Flocker, a more extensive system behavior monitor and blocker that contains elements I’m surprised Apple doesn’t offer, and maybe yet will. Little Flocker is in beta testing, and Zdziarski has been approved for a kext signing certificate from Apple, required to allow users to install kernel-level software without disabling System Integrity Protection (SIP), which was added in El Capitan. (The “flocker” part of the name is a play on “flock,” an ancient Unix characteristic used to note that a file is in use, or “locked.”)
Little Flocker looks more closely at files to prevent allowing all apps unfettered access to modify or delete them. It can effectively approve apps to access only subsets of your folder hierarchy, instead of all user-modified files. This prevents a known app from being compromised or going berserk, as well as new apps from suddenly gaining access to everything without your knowledge.
Let’s say you installed what you thought was a calculator app, and used Little Flocker to grant it permission to a folder in your Documents directory called Saved Calculations. You’d likely be surprised when it starts trying to access files all over your Home directory. Little Flocker can also extend SIP to prevent software from being deleted from your Mac by malware, and can block disk image and physical drive mounting and unmounting and access to external drives. I’d recommend against installing it at this stage on a production Mac. If you have a Mac you can test it on (El Capitan or macOS Sierra beta), that’s more sensible.
Most of the software that tries to protect Macs against malicious software currently relies on monitoring or blocking unapproved network activity, both inbound and outbound. That’s because a lot of malware is useful only if it can be remotely controlled or acquire data and ship it out. Little Snitch is terrific at being a firewall; Intego’s VirusBarrier keeps known malware from gaining a foothold (including Windows malware a Mac user might accidentally pass through).
But Santa and Little Flocker move into new territory, prompted by ransomware and other threats that have different vectors and can inflict grave damage. Apple isn’t out ahead on this so far, and Mac users have (as with many kinds of malware) so far remained largely safe. I don’t believe that condition will last forever. The more barriers put in the way of effective Mac malware reduces the desirability of criminals creating it.