In a 2001 paper, Ashton Carter, then a professor at Harvard, wrote about how commercial technology would play a decisive role in deterring and prevailing in conflict. Carter warned the U.S military must become adept at rapidly integrating commercial innovations into its operations — or a technological revolution that began in the Defense Department would quickly pass it by.
In the years to follow, Silicon Valley surged ahead, driving technological advancements across various sectors, while the military continued to fall further behind due to outdated procurement processes, bureaucratic hurdles and cultural resistance, among other factors.
“It was as if the military had resigned itself to becoming a display in the Museum of Computer History,” Raj Shah and Christopher Kirchhoff wrote in “Unit X.” “You could visit an Army, Air Force or [a] naval base to see what people used in the past, then go to Best Buy to see how far technology had evolved.”
Carter saw it in 2001 — the Defense Department needed to become “the world’s fastest adapter and adopter of commercial technology into defense systems”— and nearly 15 years later, shortly after becoming Defense secretary in 2015, he created the Defense Innovation Unit Experimental (DIUx).
“There’s a lot of people with a lot of visions and ideas. There’s very few that know how to execute — and Secretary Ash Carter knew both. He had a vision; he had the knowledge and the wherewithal to actually get anything done. It was his brainchild, and he made it come together and fought all the bureaucrats and pivoted the department,” Shah, who was one of the DIUx’s early leaders, told Federal News Network.
Carter’s vision for DIUx was to create a bridge between the Pentagon and the commercial technology sector — using funding to quickly buy existing products, help companies develop new ones and leverage venture capital to grow a broader ecosystem of firms serving the Defense Department.
The organization had a rocky start, but when Carter asked Shah and Kirchhoff to lead the unit, the pair created a whole list of things they thought they needed to get the unit off the ground. They wanted the unit to report directly to the secretary, instead of being buried in layers of the Pentagon hierarchy. They wanted Carter to assign one of his special assistants to the unit. If the unit asked other parts of the military for help, they were expected to comply. The unit could request a policy waiver should it get in the way. And the unit would oversee its own budget and staffing.
The early days of DIUx were challenging — there were budget troubles, and the organization needed a way to buy technology faster.
“I think our strategy was just to prove that we could exist. We went from 8 people to 100 in two years. And one small office to four. We were just trying to stay alive. When I joined the organization, we had zero written contracts, so we had to build a whole contracting apparatus and system,” Shah said.
“I think the biggest battles in government outside the Pentagon were all budget battles with the Office of Management and Budget and Congress, trying to defend what we were doing and why we needed our budget. And because it was so high profile, being the Secretary cared about it, a lot of entities would try to use it as leverage. They would hold us hostage to try to extract something, some concession, from the department, and so we would get in the car in the middle of a lot of these fights. It’s very painful,” he added.
The Defense Department has traditionally operated as a monopoly buyer, but when it came to Silicon Valley, the Pentagon wasn’t even a minority buyer. DIUx knew it had to fundamentally rethink how it did business — it needed to become a better customer, be able to move quickly and get companies on contract fast. DIUx knew it had to find an alternative to the traditional acquisition process under the Federal Acquisition Regulations, which can take at least a year to negotiate.
At the same time, the 2016 defense policy bill drastically expanded the DoD’s ability to use Other Transaction Authority (OTA). OTAs have been around for decades, but agencies mostly used it for work with research labs, universities and small businesses to build prototypes.
But the 2016 defense legislation enabled the Defense Department to start using OTAs to not just buy prototypes, but to move from prototype into production.
Lauren Dailey, who was the director of acquisitions at the time, took it a step further by designing a “Commercial Solutions Opening” process, which would let DIUx work directly with commercial companies and buy technology at scale.
One of DIU’s early success stories was with Shield AI, a startup that makes autonomous quadcopters. The company wanted to work with the Defense Department but struggled to raise money since investors did not view DoD as a reliable customer. DIUx, however, liked the technology and brought the company through the CSO process — within about a month, Shield AI received an OTA contract to prototype its technology. The contract then helped unlock private investment in the company.
Michael Brown, who succeeded Shah as DIU director, came in to scale that model.
“We got the transition rate, or the rate from a vendor coming in to compete on a military contract to having a production contract, in half to 50%. It wasn’t surefire, because it wasn’t 100% but you had a very good shot now of getting a production contract if you came in through a DIU process. We introduced over 50 capabilities to DoD and 50 new vendors to DOD. We really expanded on the concept,” Brown told Federal News Network.
During his time as the director, Brown’s team, along with the office of the under secretary for acquisition and sustainment, created an OTA guide to make these authorities more accessible across the acquisition workforce. The guide helped clarify not only how OTAs work, but also what their limits are.
In addition, the organization launched the Immersive Commercial Acquisition Program, a yearlong rotational initiative that embedded acquisition professionals from each military service inside DIU. The goal was to cultivate a new cadre of acquisition experts equipped with practical experience in non-traditional contracting. Plus, the Defense Acquisition University introduced specialized training programs to teach the fundamentals of OTAs.
Brown said while the combination of those efforts yielded some positive results, it is nothing like Defense Secretary Pete Hegseth directing the Defense Department to use the Commercial Solutions Openings and Other Transactions as the default contracting approaches for buying capabilities under the software acquisition pathway.
“Now everyone is going to be interested in figuring out, ‘How do I do this?’ That pull effect is going to be very strong. I think that will be the start of a complete change in the order of magnitude of OTAs are being used,” Brown said.
“We’ll start to see a lot more change more quickly. Previously, I would say there was some concern about who all was endorsing the use of the OT versus FAR. If FAR is the way we buy most things and it’s already well understood by the General Councils, by the acquisition officials within the services, OTs would have remained on the fringe. But now there’s an emphasis to go commercial first. So let’s finally start implementing something Congress asked for 20 years ago called modular open system architecture,” he added.
Now, a decade later, DIU has evolved from a small experimental organization into a central player backed by a billion-dollar budget and expanded support from Congress. Plus, in 2016, around the time the CSO process was launched, there was approximately $8 billion in venture capital investment going into companies working on defense technologies. By 2024, the number went up to about $43 billion — a 525% increase.
“I’m disappointed that we still need DIU, that the whole department has not transformed itself into being a very innovative organization. So to that point, I think there’s still a lot of work to be done,” Shah said.
The post A look back at the origins of DIU first appeared on Federal News Network.