Trajectory #12: Optimizing Complex Systems
The magic of combinatorial optimization
Hey there,
It’s Mikal Khoso and welcome to Trajectory, a newsletter about where the world is heading, delivered straight to your inbox.
The High-Level This Week: Optimizing the systems that underpin the global economy - factories, powerplants, supply-chains and more - will be critical to combating climate change. Luckily the tools of combinatorial optimization combined with computing power make this possible at scale for the first time.
To learn more, read on.
If you haven’t already, subscribe here. If you liked this week’s edition and think someone else you know would find it interesting please take a couple seconds to share it via email or on social media.
Intellectual Sparks
Saving Democracy From the Managerial Elite: The central challenge of democracy in the 21st century.
Harvard Just Discovered That PowerPoint Is Worse Than Useless: Title speaks for itself I think.
Lab-grown food will soon destroy farming – and save the planet: Trajectory #4 talked about the same perspective.
Draft Executive Order Would Give Trump a New Target: Modern Design: A pretty interesting executive order around changing the style of US government buildings nationally
Why Don’t People Care That More Men Don’t Choose Caregiving Professions?: A long overdue discussion on the double standard around professions.
Key Issue: Optimizing Complex Systems
Image from here.
Efficiency has been a critical concept ever since the industrial revolution. Increasing productivity and efficiency have been core assumptions and goals of the field of economics since day one. Increasing efficiency is a complex challenge and there are numerous ways to do so.
Typically, efficiency can be approached at a unit-level or at a system-level. Most producers are part of a larger system which interacts to create additional value. For example, every beef farmer out there is part a larger system (the food supply chain) composed of slaughterhouses, meat packers, food suppliers, truckers, supermarkets and restaurants. This entire system interacts in numerous different ways to create end value in the form of goods and services.
Driving efficiency at a unit-level entails better technological innovation - faster trucks, better cold storage, better animal feed and so on. While unit-level gains driven by technological advancement can be tremendous, system-level optimization can drive even greater savings and therefore efficiencies.
Driving System-Level Efficiency
Driving system-level efficiency requires optimization. System-level efficiency involves asking questions like:
What is the cheapest delivery route a food supplier should take across the state?
Who are the best regional suppliers for a supermarket based on price and distance?
Which farms and suppliers offer the shortest time-to-shelf on meat?
The systems that drive the economy and which producers are part of are complex systems involving interactions between multiple agents. Optimizing complex systems to find the “ideal x” involves optimizing a large set of variables. The possibilities that result from combining these different variables can be immense. There might be 1,000,000 possible routes a food supplier can take across the state with different levels costs and speed involved.
Combinatorial Optimization
Optimizing for these complex problems are known as combinatorial optimization problems (COPs). Combinatorial optimization is a subset of mathematical optimization that intersects several fields including machine learning, software engineering, complexity theory and more. Combinatorial optimization problems are defined by several distinct features:
They are discrete
There are a finite set of possibilities in the problem set to pick from
The set of possibilities is defined by a number of restrictions (e.g. number of trucks, number of destinations, average speed of trucks etc)
The goal is to maximize or minimize a specific objective (cost, speed, price and many many others)
A good example of COPs is the knapsack problem. The knapsack problem is a challenge that involves maximizing the value of goods carried in a knapsack (backpack), while limiting the weight of the total goods. Another example is the traveling salesman problem where a salesman has to visit every client he has once while minimizing the distance travelled.
Combinatorial Optimization and Resource Use
These are relatively simple examples of COPs. However, the universe of COPs out there is vast:
What is the most ideal way to stack containers in a port?
What is the university-wide class schedule for the semester and which buildings and rooms will classes be taking place in?
What’s the fastest way to produce tomorrow’s orders in a factory (production planning)?
What is the ideal number of trains travelling across the subway system hourly to reduce congestion?
These system-level efficiency challenges have always existed and the world has used a mishmash of techniques in trying to solve them over time. However, the rise of computing power and machine learning is opening up entire complex systems in the economy to combinatorial optimization.
Note: Domestic Material Consumption (DMC) / cap is a measure of the material intensity of each unit of GDP. Source here.
Optimizing many of these complex systems will have significant and profound efficiency gains for the world economy in the year to come. As the book More From Less attests to, the United States continues to use less and less natural resources every year to produce a greater and greater proportion of wealth and GDP. The chart above illustrates how developed economies such as the US and EU are seeing falling material consumption and decreasing domestic material consumption per capita of GDP.
Combinatorial optimization will only accelerate this trend by making the systems that underpin the global economy dramatically more efficient. Combinatorial optimization is a powerful and underappreciated tool that will help the world deal with the challenges of climate change. Deploying this technology across the economy will profoundly reshape things in years to come.