Narf Investigates New Methods of Detecting Emergent Behavior

Software systems are ever increasing in complexity as multiple machine and human generated programs interact against vast, changing, streams of data. This dynamic arrangement quickly outruns even an expert human’s ability to understand, and sometimes these software systems behave in surprising, unintended, and dangerous ways at the most inopportune moment!

That is why we’re excited to announce that Narf and its partners, Margin Research and Special Circumstances, were awarded a $6.3M DARPA contract for the Hardening Development Toolchains Against Emergent Execution Engines (HARDEN) program to develop novel, formal, models for flagging and enabling corrections to dangerous emergent behavior that arises from these complex systems.

The HARDEN program is designed to address these unpredictable behaviors and emergent properties by “explor[ation of] novel theories and approaches… to anticipate, isolate, and mitigate emergent behaviors in computing systems throughout the entire software development lifecycle (SDLC).” Controlling emergent behavior is a longstanding problem in computer science, and what is needed is a revolution in software maintenance: a kind of software-defined-software management scheme to observe, catalog, and correct unwanted dynamic behavior that arises from our complex systems. The impact of advances born in the HARDEN program could “radically improve security outcomes in software for integrated systems by creating novel tools … [ that ] will efficiently mitigate exploitation of software abstractions and protect intended abstractions from adversarial reuse.¹”

Although software testing and static code analysis can identify some types of vulnerabilities, guard against reintroduction of bugs, and check that important functional requirements are met, the industry largely lacks tools and methods that describe how the rich mixture of internal behaviors can be externally influenced by an attacker, or combine to result in complex system failures and security vulnerabilities.

Increasing the predictability of such emergent behavior could lead to more secure systems, but this can only be done by achieving radically new methods of defining and controlling emergent properties. Narf and its partners look forward to rapidly generating new technical capabilities and accelerating them to apply to critical use cases to decrease downtime and reduce the vulnerability surface of modern computing infrastructure.

Disclaimer: The views, opinions, and/or findings expressed are those of the author(s) and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government.