Benjamin Tseng

”

“Before 2025 Taiwan’s defence exports consisted mainly of some old helicopters and small boats donated to African and Latin American countries. That is changing as Taiwan positions itself as a supplier of “non-red” unmanned aerial vehicles or UAVs—ie, without Chinese parts—to America, its allies and other countries worried about China’s espionage and control over industrial supply chains. Taiwan’s production of UAVs has increased from about 10,000 units in 2024 to more than 12 times that in 2025. Its exports rose more than 35-fold to about 123,000 units last year—almost all its output. The Czech Republic was its biggest buyer, followed by Poland (see chart). Industry insiders say their purchases mostly went to Ukraine, where many are used on the battlefield, giving valuable feedback on their performance. Another bumper year for exports is expected in 2026, after reaching 85,500 units in the first two months. Taiwan launched its drone programme after observing Ukraine’s use of the technology. The main aim was to equip Taiwan’s own armed forces, which had just a few hundred drones. China had tens of thousands. Taiwan has since pledged to build an entirely non-red UAV industry by the end of 2026, with annual output of 180,000 units by 2028. Last year it also promised to buy more than 200,000 drones for its own armed forces by 2033.”

Taiwan\u2019s bid to export drones free of Chinese parts is taking off

The Economist

The Difficulty of “Planting” Your Way Out of Climate Change

Tue, 17 Mar 2026 00:00:00 GMT

When I was an elementary school student and first heard of “global warming”, I thought the solution to climate change was simple: just plant more trees! 🌳

While I now know there’s a great deal more complexity to the problem than that, I have always wondered what would happen if someone had tried out that little boy’s instinct.

In a recent article in Earth’s Future, scientists took a look, taking advantage of China’s massive investment over decades in planting trees around the country, “which accounted for 25% of the global net increase in leaf area [between] 2000–2017”! In particular, they found that, while they were able to impact dust storms and desertification, the rapid planting of trees had major negative impacts on underground water supplies as the massive resulting evapotranspiration (literally plants pulling water from the ground and into the air) effectively transferred that groundwater into rainwater over the Tibetan Plateau!

China, especially its Northern/Northwestern regions, have major issues with water availability and drought and it is a cosmic irony that an attempt to stop desertification, while greening much of the land, may have resulted in worsening the water condition.

It is a humbling reminder that large-scale well-intention actions may have adverse consequences and a reminder that, while we should pursue indirect avenues for mitigating climate change, there is probably no substitute for ultimately cutting down on the emissions and activities that are most directly related.

Reading Excerpt • Econews

”

“Their model shows that land restoration increased evapotranspiration by around 1.7 millimeters per year and precipitation by about 1.2 millimeters per year, yet overall water availability still went down. The main driver was forest expansion in the humid east and large-scale restoration in the northwest and on the plateau. As deep-rooted trees pull water from the soil and release it into the air, they act a bit like giant pumps. The process cools the local surface but also ships moisture downwind, where it can fall as rain far from the place where it was originally stored.”

China planted trees like crazy to slow the advance of the Gobi Desert and ended up causing another big problem: water began to disappear from the soil and aquifers as rainfall patterns changed

Sonia Ramirez | Econews

CAR-T for Autoimmune Disorders

Mon, 08 Dec 2025 00:00:00 GMT

Immune cell therapies like CAR-T continue to provide some nearly miraculous therapeutic power. In addition to handling solid tumors, it turns out CAR-T can also be used to treat autoimmune disorders like lupus, rheumatoid arthritis, and ulcerative colitis.

By engineering CAR-T cells to attack B-cells, the body’s factories for the autoantibodies that do the damage in an autoimmune disorder, you not only eliminates the direct source of the problem (the autoantibodies), you potentially trigger an “immune reset” which seems to durably eliminate the problem (see diagram)!

How CAR-T for Autoimmune Disorders works (Source: Gemini Nano Banana)

Medical researchers rarely throw the word “cure” around, but this Nature News article does so 4 times! We live in interesting times!

Thoughts

The Declining Preference for Boys

Thu, 06 Nov 2025 00:00:00 GMT

What makes this social change — where parents, especially in East and Southeast Asia, previously heavily favored having male children and now much less so — extra surprising is that many of the technological factors that drove the gender selection preference (such as access to early gender testing and IVF) have dramatically improved over time.

Who’s Jobs are AI Taking Right Now?

Mon, 03 Nov 2025 00:00:00 GMT

Despite all the speculation over why Amazon is laying off 14,000 workers, as of this writing, and with a few exceptions I’ll note below, I don’t personally worry a great deal about AI taking away jobs currently.

The reason is that while new AI agent based services and products are becoming better at replacing humans at certain tasks:

Many tasks are not automatable — especially ones where the “product/service” is actual human interaction and judgement. And those tasks tend to be the ones that take the longest and the most people to do.
Technological improvements can lead to increased demand. The classic examples of this are how the advent of spreadsheet software increased the number of accountants (when calculation work became cheaper and easier, the demand for accounting-style thinking to be used broadly across the business increased) or how automated teller machines (ATMs) increased the demand for human tellers (by making it imperative for convenience to have more branches).
AI tools are great at answering questions and doing assigned tasks. But you still need a person with actual judgement and experience to ask the right questions and assign the right tasks

While the above three advantages may ultimately disappear as technology improves, in general, I am optimistic that, by making workers more productive overall, AI technology will make workers more valuable overall.

The one exception that I immediately saw, however, were entry-level knowledge workers. New (and, in most cases, young) knowledge workers (engineers, designers, analysts, writers, management consultants, etc) are uniquely not valuable when they first start a job. They lack context, judgement, and skill. They tend to only prove their value after they’ve had the chance to learn on the job. Historically, the bargain was that entry-level knowledge workers would start with relatively lower-value tasks that would, through time and exposure, help them learn the context, judgement, and skills they would need to become productive. This is, after all, the path I took as a novice management consultant and later investor.

But with new AI tools, the case for hiring these entry-level knowledge workers dramatically weakens. Claude Code might not be able to replace the judgement of a senior architect, but it can probably get up to speed on a codebase faster, write code more accurately, and all without needing rest or paid time off than a fresh-out-of-school developer. Gemini might not be able to have the same type of insights as someone with a deep rolodex in an industry, but it will certainly know more and can conduct & summarize internet research much faster than a freshly minted consultant. ChatGPT might not be able to capture the artistry or investigation skills of a Pulitzer Prize winning journalist, but it can definitely write up summaries of stock market movements or the press releases from a company better than a novice journalist.

This is ultimately self-defeating — as without new junior talent, where does one find good middle-level or senior talent — but it’s also something that I fear we are already beginning to see. This Goldman Sachs research report I just read has a great Exhibit 4 showing how while new AI tools have not significantly impacted employment in general or even employment in tech, it has meaningfully increased unemployment in 20-somethings who work in tech (see image below), exactly the demographic who’s value as entry-level workers has now been largely displaced by AI.

How the tech industry (and other knowledge work professions) ultimately choose to handle this will be the defining test of how we incorporate AI into our economic lives.

DeepSeek in Africa

Thu, 23 Oct 2025 00:00:00 GMT

Earlier this year, a Chinese company called DeepSeek took the AI world by surprise by releasing an incredibly cheap, performant reasoning model called DeepSeek R1. The result was an outpour of commentary on how Meta’s open sourcing of Llama led to its creation, whether or not the reported lower training cost for DeepSeeek’s model meant the end of NVIDIA’s business, and whether or not China was overtaking the US in AI technology.

While those discussions raged, Chinese companies like the telecom infrastructure giant Huawei took the low cost open source DeepSeek model and have turned it into a business targeting countries in Africa which have already been the beneficiary of substantial Chinese investment.

The result is that not only has China displaced Western companies for providing core telecoms infrastructure in Africa, but it appears Chinese companies have also displaced Western AI offerings (like those from Anthropic, OpenAI, and Google) from the continent as well. By offering lower per token prices and by having a technical backbone that uses fewer tokens per request (Chinese models employ tokenizers with larger vocabularies to handle multi-lingual data which results in fewer tokens for words in non-English languages) and being offered by partners who have already built much of their digital infrastructure, Chinese models (and especially DeepSeek) have become ascendant in Africa.

While this has led to some problems (for example, Chinese AI model providers disabled their image recognition systems during the Chinese 高考 gaokao, or annual undergraduate admissions exam), the token economics are difficult to resist for AI adopters in Africa.

This should be terrifying to Western companies (who are in a fierce competition for AI model supremacy) and especially Western governments concerned about China’s influence. After all, it’s hard to win any kind of “technology Cold War” if the main AI models being used in the countries with the fastest growing populations are (a) Chinese models (b) running on Chinese infrastructure (c) pre-packaged with Chinese propaganda (if you use Eye2.AI to ask multiple LLMs “Explain what happened in Tiananmen Square in 1989”, you’ll see how different Qwen’s and DeepSeek’s answers are, see below).

Screenshot from Eye2.AI on a “sensitive subject” for Chinese AI models

Reading Excerpt • Bloomberg

”

“Although much of the world’s attention has been focused on Western tech companies vying for lucrative corporate contracts in the US and Middle East, the meeting in Nairobi illustrates how their Chinese rivals are taking a different approach. OpenAI and its American competitors have focused almost exclusively on proprietary AI — models whose software, training data and algorithms are entirely controlled by their parent companies, with customers paying for access. Chinese firms like Huawei and Alibaba Group Holding Ltd., by contrast, are courting Africa’s startups and innovation hubs with open-source AI models — ones that can be accessed and modified for free, letting companies build products without expensive licenses. This strategy, with parallels to China’s Belt and Road Initiative for physical infrastructure, is not designed for immediate profit. Africa’s entire digital economy, valued at roughly $180 billion, pales in comparison to OpenAI’s $500 billion valuation in recent share sales. Instead, it’s a long-term bid for customers, soft power and the vast troves of data that will shape the future of artificial intelligence.”

China\u2019s DeepSeek Is Beating Out OpenAI and Google in Africa

Saritha Rai, Loni Prinsloo, Helen Nyambura | Bloomberg

Seven Lessons from My 30s

Sun, 19 Oct 2025 00:00:00 GMT

Source: Nano Banana

Turning 40 is a milestone that should come with a manual. It doesn’t, of course, but it does come with a decade’s worth of receipts for lessons learned the old fashioned way.

As I ponder the next decade ahead of me, I’ve been reflecting on the things I picked up in the last one that have shaped my understanding of people and the world around me. I share seven of these lessons in the hope they might be helpful to — or at least amusing to 😅 — the reader.

1. You Don’t Know Where You Rank in Someone’s Priorities Until You’ve Seen Them in a Crisis

It’s easy to be a good partner during the good times.

The real test of character comes during the bad times. It is only when the pressure is on and pleasantries feel like a wasteful luxury that people reveal their true priorities — and, especially, where you rank in those priorities. It’s in those moments of crisis that you can see who someone fundamentally is.

This advice applies to both love and business: if you can, don’t enter into a partnership — getting married, starting a company together, having a kid together — with someone until you understand how they handle crisis. How do they act when they’re scared, angry, or cornered? Do they collaborate with their partners, or do they retreat into self-preservation? Do they treat partners like equals or burdens? Sources of strength or resources to be exploited?

I’ve sadly learned several times in the past decade how people who are giving and collaborative when things are one way can quickly close off and prioritize themselves when things got tough.

To be clear, this isn’t to say people who react poorly under pressure are “bad” or deserve to be cut out of your life. Crises are difficult for even those with the strongest of moral character, and you can still have a friendly or productive working relationship with someone who doesn’t consider you one of their top priorities.

But understanding how someone acts in a crisis is a vital data point I would recommend thoroughly understanding before entering into any kind of serious partnership.

2. Profession can be an Oddly Good Read on a Person

I used to hate the question, “So, what do you do?”

It felt reductive, and almost insulting, especially when I was starting out in my career and my job was only a very small piece of what I felt I had to offer.

But I’ve come to appreciate that while a job is never the whole story about a person, it can be an incredibly revealing chapter. When you spend most of your waking hours on something that also shapes your finances, social status, and daily challenges, it inevitably molds you. Jobs define our incentives, hint at our training and skills, and frame our definitions of success and failure.

Knowing someone is a practicing doctor in the US, for example, gives you a hint about their:

age & education — given the amount of medical training necessary to practice medicine in the US
their income — likely relatively high, with more precision possible if you know their specialty
a sense of their views on the US healthcare system — I’ve never met a physician who likes the health insurance companies who impose rules and paperwork on them to get paid, their electronic medical record system, or medical malpractice lawsuits
their approach to risk — having being ingrained in the ethos of “first do no harm”
their views on credentials — given the amount of time (and money!) they spent on getting their Medical Degree as well as the additional licensing and certification necessary to practice

You won’t always get every detail right — and profession won’t give you a good indication on many aspects of their personality or personal lives that have little to do with their career — but it’s a more useful than not set of heuristics for understanding the people you meet.

This is why I’ve grown very wary of interfacing with politicians. To succeed there requires a level of self-obsession that’s hard to square with sincerity. Success is driven by saying what’s popular, not necessarily what’s true (or even what you believe). Politicians with any level of success are also surrounded by people paid to flatter them. It creates a backdrop where genuine connection is difficult, power dynamics are twisted, and relationships easily become transactional. Andrew Yang captured some of this in a piece he wrote for Politico Magazine in 2021 entitled “When I Ran for President, It Messed With My Head”.

Of course, there are exceptions. But in my experience, it is foolish to underestimate how influential the incentive structure of a career can be.

3. There’s Little Upside in Talking About Politics

To start with the obvious: policy matters. It shapes our lives, and in a democratic society, ignoring it is irresponsible.

That said, I suspect many of us engage with politics far too much for our own good. The problem is that modern political discussion and news isn’t about logic — it’s about evoking visceral, tribal outrage. This is bad for two reasons. First, it leads to bad policy — it makes us seek symbolic victories and “cultural” retribution rather than nuance and sustainable compromise. Secondly, it leads to bad behavior by training us to see the world in terms of “us” versus “them.”

In my 20s and early 30s, I loved a good political debate. I had a neat little box of “obvious things all reasonable people should believe.” That argumentativeness and certainty cost me. A long time friendship ended abruptly this past decade, likely over some position I took in one of those debates (it was never entirely clear to me). I also nearly became unfriendly with a former mentor over a political disagreement over that “neat little box of obviousness”.

As someone who values his relationships, these were not proud moments. And it led me to ask: what upside did I get from seeking these debates? A temporary satisfaction that “I am more righteous than the ‘other tribe’”? Scoring some kind of rhetorical point? Hardly seems worth it.

I now try to disengage from the daily outrage cycle of political news and who-said-what. After all, it rarely has real impact on policy, and the news cycle usually moves on after a few days. Instead, I try to focus on tracking substantive policy changes and analyses. It’s a lot more boring, but that’s a feature, not a bug.

I also am a bit embarrassed to admit but I now actively avoid discussing politics with people — both online and in real life. This isn’t to say that I don’t have my own opinions (and in moments of weakness, I’ll share some of them on social media), but in recognizing there’s little upside, I’ve come to realize nobody really cares what I think. They just want to know if they should get angry with me or at me, and I don’t see the point in playing that game any more.

4. Embrace the Unlikely

If my 30s taught me anything, it’s that the improbable is inevitable. In the last decade alone, I experienced things that I thought were highly unlikely to happen in my lifetime:

was fired
called as a witness in legal proceedings (twice!)
lived through a global pandemic
saw a vaccine developed in under a year
saw US inflation go above 9%
watched a massive political re-orientation on globalization in the West

Statistically speaking, over a long enough horizon, unlikely things will inevitably happen. But, beyond that, because unlikely things are highly correlated with one another, unlikely things will likely happen together. In my list above, for example, at least 3 of those things are directly related to COVID.

So, what do you do in a world where the improbable is not only inevitable but will likely co-occur?

First, prepare where you can. Get an estate plan, a rainy-day fund, a diversified portfolio, and make contingencies.

Second, and most importantly, try to embrace the unlikely. The chaos of 2020 was a period of great stress for me and my family. I remember wondering at points if this was the end of the global economy and of in-person interaction. These are especially unsettling thoughts when you have a toddler at home and weren’t sure when you would next see your aging parents.

But, in hindsight, 2020 was also one of the most transformative and productive years of my professional life. I helped build a telemedicine operation (that is, as of this writing, still doing pretty well); learned how to code AI models and had some of that work ultimately published in peer-reviewed journals; and tried my hand at policy work which led to new friendships and professional opportunities.

Out of that tumultuous year came something positive that, I have to admit, would not have happened otherwise. So while 2020 was a terrible year in many fundamental ways, I’m grateful I was in a position to embrace it as an opportunity.

5. Genius Doesn’t Generalize

Smart people tend to be smart about multiple things. So it makes sense to seek out the advice of people who know better.

However, we live in an age of unhealthy hero worship powered by influencer marketing and interview podcasts. We assume that because someone is brilliant in one domain, they’re worth listening to in others.

But, in my experience, nothing could be further from the truth. What it takes to achieve genius-level mastery in one area often comes at the expense of others.

I’m always baffled when people ask Elon Musk for life advice or Warren Buffett for his opinion on anything besides investing. A quick read of their biographies shows that their personal lives are a mess. Their genius is highly specialized. Their situations are extreme. At best, you get a different perspective. More likely, you hear something that is likely completely irrelevant to you.

Source: XKCD

It’s worth remembering this the next time you hear a billionaire investor weigh in on sociology, a Nobel Laureate scientist weigh in on politics, or an author of fiction weigh in on technology. It’s far more fruitful to consult a few smart friends or read about actual expert opinion than to mind the opinion of a famous (and/or rich) person on a topic outside their expertise.

6. Good Taste is Overrated

In our careers, “good taste” is something worth cultivating and investing in. It signals judgment and makes you hard to replace, especially in this new world of AI.

But in life? I’ve come to a radically different conclusion: it’s much better to have low standards.

My inability to distinguish a $20 bottle of wine from a $200 one means I can get the same pleasure from wine as a world-class sommelier for a fraction of the cost.

My love for bad action movies and rewatching sitcoms means there’s always something to watch on an airplane.

My willingness to enjoy fast food means I can find a decent meal almost anywhere in America.

I used to be slightly ashamed of all of these things as evidence of my “uncultured palate”. Now, I see it as a gift. Being easy to please is a superpower for everyday contentment. And, it frees up time and mental energy to focus on what I can discern and care more about.

7. Play Stupid (Status) Games, Win Stupid Prizes

Many things we pursue in life aren’t for a tangible reward, but to signal something about ourselves to others and (sadly, often) to ourselves. These status games are everywhere, and the trap is that you can never win. There’s always a new car to buy, a more exclusive vacation to take, a new trend to follow. Marketers are masters at playing on this to keep us on the never-ending spending treadmill.

I felt this most acutely in the past decade with my kids. The entire “new parent industry” is a masterclass in exploiting a new parent’s desire to be a good parent. It’s engineered to make you feel guilty for all the things you aren’t doing (and all the products you aren’t buying) while feeling proud of the things that you are (and all the products you are buying). I remember the shame and anxiety when our eldest son didn’t seem to be hitting his speech milestones as quickly as the books suggested (or *gasp* as my friends’ kids were).

Thankfully, my level-headed wife and pragmatic pediatrician helped me see the bigger picture: our son was happy, healthy, and developing just fine on his own timeline. And sure enough, he started talking up a storm at two-and-a-half and was reading full books not long after.

This sort of un-grounded status seeking is everywhere: people staying in “prestigious sounding” jobs they hate, spending money to keep up with their friends’ Instagram lives, or chasing exotic investments just so they have a more exciting story to tell than “I just buy index funds.”

The lesson isn’t to drop out of all status games: that’s probably impossible. I will sheepishly admit I buy high-end gadgets not because I need them but to signal that I’m the type of person who buys high-end gadgets. I’ve also maintained a ridiculous 425-day+ Reddit streak for reasons I can’t particularly defend.

But, the lesson is to be deliberate. Recognize a status game for what it is, and then consciously decide if it’s one you actually want to play.

Seeing Past the Surface

Looking back, if there’s a single thread connecting these seven lessons, it’s about learning to see past the convenient surfaces. It’s about understanding the hidden incentives that drive people, the true priorities that emerge under pressure, and the surprising freedom that comes from ignoring the ones that don’t matter.

This process of looking deeper is never really finished. The biggest thing I’ve come to appreciate is that the learning never stops. I’m sure my 40s will dismantle some of the certainties I’ve built in my 30s, and I can’t wait to see which ones.

That, I suspect, will be the real adventure.

Gene therapy for Huntington’s shows promising clinical trial data

Wed, 24 Sep 2025 00:00:00 GMT

Huntington’s Disease is one of the classic “uncurable” neurodegenerative diseases. A genetic mutation produces a mutant form of a protein (called Huntingtin) which leads to sudden onset of symptoms which gets progressively worse until death. It’s so bad that it’s considered a “textbook” example of the need for genetic counseling when people are considering getting their DNA sequenced: for some, it’s better to not know they’re a carrier or have the disease.

But some exciting news from biotech company uniQure today may change that! UniQure announced the results of a Phase I/II study showing how a revolutionary gene therapy + brain surgery can slow disease progression by 75% over 36 months.

What also stood out to me was the use of a synthetic control arm — basically using data collected on real world patients (rather than trial participants) to alleviate the time and cost burden of needing to recruit from a relatively rare disease population (as well as capture disease progression in a realistic setting). Overall, an amazing feat for everyone involved and hopefully a beacon of hope for those with Huntington’s Disease or carrying the gene.

LA child dies from SSPE — (yet) another reason to vaccinate

Sat, 13 Sep 2025 00:00:00 GMT

Let’s summarize the case for measles vaccination:

Measles is deadly — prior to the measles vaccine, of the 500,000 cases/yr that were reported, 400-500 died and 48,000 were hospitalized.
Measles clears immune memory, making people who recover more susceptible to infections afterwards — This phenomena is called “immune amnesia” and seems to be mediated by the measles virus being able to infect immune memory cells with some researchers estimating that potentially half of all childhood deaths from infectious disease prior to the measles vaccine may have been caused in part by this immune amnesia.
Children who recover from measles infection may develop SSPE, a neurological disorder with no cure and terrible prognosis. Subacute sclerosing panencephalitis (SSPE) occurs when the measles virus ends up “hiding” in neurons and glial cells as the body clears the initial measles infection. This “hiding” leads to a progressive neurological disease with no cure and few effective treatments beyond just trying to treat the seizures. While rare, the chance of developing SSPE is higher in infants and especially if unvaccinated.

Sadly, a child in LA county died on Thursday from SSPE — while we don’t have a ton of information about the child’s age, it’s likely the child came down with measles prior to vaccination. For us to protect our kids and each other, we should not only get them vaccinated but insure broad enough vaccination so as to insure herd immunity.

Why are ultra-processed foods bad for you?

Fri, 12 Sep 2025 00:00:00 GMT

Nutrition is a field that is fraught with studies that are not powered or designed well enough to make strong conclusions about what types of diets are the “best”.

So it was interesting to see this article dig into the evidence over whether or not ultra-processed foods, a category that itself is a little hard to pin down — somehow we’re supposed to find it “natural” that burgers and cookies and breakfast cereals are in the same group — are “bad” for you and why.

The short answer is: yes they are bad, but it doesn’t seem to be exactly clear why. Potentially some combination of the foods being energy dense, quick-to-eat, and potentially impact our gut microbiome.

What was most impressive to me, however, were the number of randomized controlled trials that were attempted with confined study participants (the only way to know who’s eating what and how much) — those are hard to do (and consequently relatively small) but are probably the most convincing evidence.

Well worth a read if you’re even remotely interested in what the best science has to tell us about how and why processed foods impact our health.

The first RCT on UPFs [Ultra-Processed Foods] was carried out by a team led by integrative physiologist Kevin Hall, who until April was at the US National Institutes of Health (NIH), part of the HHS. For more than a decade, Hall has sought to overcome some of the limitations of nutrition research by confining study participants to a research hospital, so that their diets can be precisely recorded.

In 2015, he was approached by two Brazilian researchers at a meeting after he presented the results of one such study, comparing a low-carbohydrate, animal-product-based diet with a low-fat vegan diet. “They told me they had enjoyed my talk, but that my focus on nutrients was very twentieth century and that really, I should be thinking about the extent and purpose of processing food,” says Hall, who had not then heard of UPFs. He was sceptical but intrigued enough that he devised a trial in which 20 adults stayed at the NIH Clinical Center in Bethesda, Maryland, and consumed a diet high in either UPFs or unprocessed food for two weeks before swapping diets.

Like many other researchers, Hall defined UPFs using a system developed by Monteiro. Known as NOVA, it classifies food into four groups according to the extent, type and purpose of processing8. Unprocessed or minimally processed foods, such as vegetables and pasta, are at one end of the scale. At the other end are UPFs, which undergo multiple industrial processes and usually contain substances rarely used in home kitchens, such as high-fructose corn syrup, hydrogenated oils and additives, including colouring and emulsifiers.

Participants in Hall’s RCT could eat as much as they liked, but the food presented to them was matched for nutrient groups. To Hall’s surprise, the results, published in 2019, showed that participants consumed around 500 calories more per day on average while on the UPF diet — and gained an average of 0.9 kg over the two weeks9. They lost that same amount of weight while on the unprocessed diet.

Are ultra-processed foods really so unhealthy? What the science says
Nic Fleming | Nature News

The Surrealism of Authoritarianism

Sun, 07 Sep 2025 00:00:00 GMT

Among the obvious hallmarks of authoritarianism are:

Arbitrary paranoia from on high — where students singing patriotic songs on bike rides are suddenly viewed to be a threat
Contradiction between reality and what the state wants you to believe — where state media simultaneously can champion “Labubu blind boxes” as signs of cultural & economic success but also frivolous “tools to exploit children’s wallets”

While every society has its own forms of this, it’s very clear China under Xi Jinping has advanced in troubling directions on both, an alarming sign for anyone who lives in China but also a reminder to free societies that it’s easier than you think to step down this path.

Reading Excerpt • The Atlantic

”

“Labubu appears to be yet another sign of China’s global success. Figurines of the grinning, pointy-eared elf, marketed by a Chinese company called Pop Mart, are so wildly popular that fans around the world go to great lengths to get their hands on them. Many of them come in “blind boxes,” meaning that the consumer gets to see the contents only after purchase. The Chinese state news agency Xinhua boasted in mid-June that the Labubu craze “signals a broader shift in China’s role on the global stage”: The country is becoming a cultural center. At home, however, the Chinese Communist Party is working to dampen the enthusiasm. A June article in its main newspaper, the People’s Daily, criticized the “out of control spending” on blind boxes and similar products among minors who are “irrational” in their decisions and called for tighter regulation to prevent such objects from becoming “tools to exploit children’s wallets.” Blind boxes are but one cultural trend to incur the party’s ire. In recent years, Chinese authorities have gone after video games and K-pop, comedy clubs and Halloween parties, gay and lesbian activists and women’s-rights advocates, tech entrepreneurs and financial advisers. The incessant crackdowns, and the campaigns of censorship or censoriousness, suggest that the Chinese regime is intent on not just eliminating opposition, but also micromanaging its people’s lifestyles, consumption, and beliefs.”

Life has Gotten Surreal in China

Michael Schuman | The Atlantic

(Finally) The End of Evergrande

Wed, 13 Aug 2025 00:00:00 GMT

It’s been a long time coming but China Evergrande (which I blogged about nearly 2 years ago!), once China’s largest property developer and who’s debt crisis is widely viewed as the “Lehman Brothers” moment in China’s property crisis, has finally been de-listed, about 18 months after it was ordered to be liquidated.

What remains to be seen is what China’s government will do as China’s property crisis continues. It’s somewhat ironic given that it was the Chinese government which triggered the property sector’s implosion when it tried to crack down on the property bubble. Only time will tell how much pain the central government is willing for the sector to endure — the US was unable to stomach this pain during the Great Financial Crisis and ultimately made “heroic” moves to save the housing sector.

New Age Bioprospecting for Antimicrobials

Mon, 11 Aug 2025 00:00:00 GMT

One of the major contributors to improved healthcare outcomes and reduced mortality in the post-World War II era was the widespread use of antibiotics. Able to treat infection, these miracle drugs literally transformed bacteria from humanity’s primary killer to a manageable problem.

But, in recent decades, the decline in the discovery of novel antibacterials (and other antimicrobials like antifungals) and the rapid rise in antimicrobial resistance has threatened this happy state. This has led many experts to worry about what will happen in a post-antibiotic world. After all, without effective antibiotics, we not only lose the ability to treat life-threatening diseases like tuberculosis (not to mention common ailments which are a nuisance today but may become more serious like strep throat, urinary tract infections, and ear infections), but we will also lose the ability to perform many surgeries safely (as recovery oftentimes counts on antibiotics to prevent and hold at bay infections).

While we need to get smarter about how and where we use antibiotics (especially in agricultural applications) to slow the rise of resistance, the other half of this problem is in discovering and commercializing new antimicrobials. This is something we’ve largely failed to do since the 1960s, as the figure below from the C&EN article derived from data in a 2019 paper shows.

The “golden age” of antimicrobial discovery that ended in the 1960s came largely from researchers searching for these miracle chemicals in soil samples (“bioprospecting”), where bacteria and fungi, in order to compete, release them as “chemical weapons” against others. But, having long ago exhausted the “easy” antimicrobials, we were unable to replicate this success in the decades following the 1960s

But maybe this is about to change. In March, two completely new classes of antimicrobial were published in two weeks (both in Nature) — equaling the total output of the 2010s! Both papers (one by a group from China Pharmaceutical University in Nanjing, China and the other from a collaboration between University of Illinois in Chicago and McMaster University in Canada) were centered around identifying biosynthetic gene clusters (BGCs) responsible, and with the lower cost of metagenomic sequencing and new computational methods, this may point to a new era of discovering novel antimicrobials to help us innovate our way out of our current antimicrobial resistance dilemma by re-inventing bioprospecting.

A good reminder that what we need is the political and scientific willpower to keep funding this type of research and the types of genomic and protein databases that make this type of innovation possible!

Reading Excerpt • C&EN

”

“In March, the journal Nature published the discoveries of two paradigm-breaking antimicrobial compounds in as many weeks: a polyene macrolide antifungal called mandimycin and a lasso peptide antibiotic called lariocidin. Both these compounds, which use never-before-seen antimicrobial mechanisms, were found using techniques that let researchers look deep into the chemical diversity of microbes—much deeper than a typical antibiotic or antifungal screen might go. And it’s not just these two molecules. Scientists are using the new approaches to discover countless other antimicrobial compounds with the potential to become drugs.”

Bioprospectors mine microbial genomes for antibiotic gold

Max Barnhart | C&EN

Google’s New Weapon in AI — Cloudflare

Mon, 04 Aug 2025 00:00:00 GMT

Tech strategy is difficult AND fascinating because it’s unpredictable. In addition to worrying about the actions of direct competitors (i.e. Samsung vs Apple), companies need to also worry about the actions of ecosystem players (i.e. smartphones and AI vendors) who may make moves that were intended for something else but have far-reaching consequences.

In the competition between frontier AI models, it is no surprise that Google, where the Transformer architecture virtually all LLMs are based on was created, was caught off-guard by the rapid rise of OpenAI and AI-powered search vendors like Perplexity and Chinese participants like DeepSeek and Alibaba/Qwen. While Google (and its subsidiary DeepMind) have doubled down on their own impressive AI efforts, the general perception in the tech industry has been that Google is on defense.

But, as I started, tech strategy is not just about your direct competition. It’s also about the ecosystem. Cloudflare, which offers distributed internet security solutions (which protect this blog and let me access my home server remotely) recently announced that it would start blocking the webscrapers that AI companies use due to concerns from publishers of websites that their content is being used without compensation.

However, because search is still a key source of traffic for most websites, this “default block” is almost certainly not turned on (at least by most website owners) for Google’s own scrapers, giving Google’s internal AI efforts a unique data advantage over it’s non-search-engine rivals.

Time will tell how the major AI vendors will adapt to this, but judging by the announcement this morning that Cloudflare is now actively flagging AI-powered search engine Perplexity as a bad agent, Cloudflare may have just given Google a powerful new weapon in it’s AI competition.

Reading Excerpt • Gabriel Corral, Vaibhav Singal, Brian Mitchell, Reid Tatoris

”

“The Internet as we have known it for the past three decades is rapidly changing, but one thing remains constant: it is built on trust. There are clear preferences that crawlers should be transparent, serve a clear purpose, perform a specific activity, and, most importantly, follow website directives and preferences. Based on Perplexity’s observed behavior, which is incompatible with those preferences, we have de-listed them as a verified bot and added heuristics to our managed rules that block this stealth crawling.”

Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives

Gabriel Corral, Vaibhav Singal, Brian Mitchell, Reid Tatoris

Helsinki goes a full year without a traffic death

Wed, 30 Jul 2025 00:00:00 GMT

Helsinki, Finland (population: ~650,000) has not had a single car cash related death in over a year! That is impressive! They believe a combination of lower speed limits, improved pedestrian / cycling infrastructure, public transit improvement, and traffic camera all contributed.

While Americans love cars (this writer included!), the fact that roughly 40,000 Americans die each year from motor vehicle fatalities (not to mention ~2.6 million emergency department visits related to motor vehicle crashes in 2022 and $470 billion in medical costs) should push us to question if we can do better by learning from the experiences of other places.

I don’t relish driving even slower in a city, but it’s hard to deny the alternative is even grimmer.

Reading Excerpt • YLE News

”

“According to Utriainen, more than half of Helsinki’s streets now have a speed limit of 30 km/h. Fifty years ago, that proportion featured 50 km/h limits. Earlier this summer, Helsinki decided to lower speed limits near schools to 30 km/h, a measure that is set to take effect as the academic year begins. Street design has also played a key role. Pedestrian and cycling infrastructure has been significantly upgraded in recent years. In addition, cooperation with traffic police has intensified and more traffic cameras and automated enforcement systems have been introduced. “Public transport in Helsinki is excellent, which reduces car use, and with it, the number of serious accidents,” Utriainen noted. Vehicle technology has also improved, making both cars and other personal transport options safer than ever.”

Helsinki goes a full year without a traffic death

YLE News

Cloudflare Tunnels for Your Home Server

Fri, 25 Jul 2025 00:00:00 GMT

(Note: this is part of my ongoing series on cheaply selfhosting)

If you’ve been following my selfhosting journey, you’ll know that I host some web applications — among them, network storage for my family, tools for working with PDFs, media streamers, a newsreader, etc. — from an inexpensive home server on my home network. It’s mostly a fun hobby but it’s helped teach me a great deal about web applications and Docker containers, and has helped me save some time and money by having applications I can control (and offer).

But one of the big questions facing every self-hoster is how to access these applications securely when I’m not at home? This is a conundrum as the two traditional options available have major tradeoffs

Opening up ports to the internet — One way to do this is to open up ports on your internet router and to forward traffic to those ports to your server. While this is the most direct solution to the problem of granting access to your hosted applications, this has several issues
- First, some internet service providers and routers don’t actually let you do this!
- Second, by opening up a port on your router, you’ll be opening up a door for everyone on the internet to access. This could expose your home network to malicious actors, and requires you to either accept the risk or set up additional security mechanisms to protect yourself.
- Third, unless your internet service provider has granted you a static IP address (which is relatively rare for consumer internet plans), the IP address of your home will randomly change. Therefore in order to access your home server, you’ll need to setup a Dynamic DNS service which adds additional complexity to manage.
VPNs or VPN-like solutions (Twingate, Tailscale, etc) — The alternative to opening up a port is to leverage VPN and VPN-like technologies. This is much more secure (and, in fact, I use Twingate, a great VPN-like service, for this type of secure remote access). But, this has one major downside: it requires each device that wants to access your hosted applications to have a special client installed. This can be a hassle (especially if you want to grant access to someone less tech-savvy), and, in some cases, near impossible (if you’re dealing with devices like a connected TV or eReader or if the device is behind a firewall that doesn’t like VPNs).

I wanted a third option that:

would work nicely and securely with practically any internet-connected device
didn’t require client installation or configuration
didn’t require me to open up any new ports on my home router or expose a public IP address
could integrate authentication (as an extra layer of security)

That’s how I landed on Cloudflare tunnels!

Cloudflare Tunnels

Enter Cloudflare Tunnels, a product in the Cloudflare Zero Trust family of offerings. By running a small piece of software called cloudflared on your home network (i.e. as a Docker container on your home server), you can link:

the services/resources on your home network
domains hosted and secured by Cloudflare
third party authentication services (like Google login)

What that means is my local Stirling PDF tools (which live on my home server at the domain pdf.home) can now be reached by any internet-connected device at https://pdf.[mydomain.com] while locked behind a Google login which only allows users with specific email addresses through (i.e. my wife and myself)! All for free!

How to Setup

Transferring Your Domains

To get started, transfer your domains to Cloudflare. The specific instructions for this will vary by domain registrar (see some guidelines from Cloudflare). While you can technically just change the nameservers, I would highly recommend fully transferring your domains to Cloudflare for three reasons

Cost: Cloudflare (as of this writing) offers at-cost domain registration. This means they don’t add any markup on top of what it costs to actually register the domain and so it’s typically cheaper to buy and renew domains with Cloudflare
Security: Cloudflare offers free and automatic HTTPS protection on all domains and basic DDOS protection as well
Extra Configurable Protection: I am not a cybersecurity expert but Cloudflare, even on their free tier, offer generous protection and domain features that you can further customize: bot protection, analytics, a sophisticated web application firewall, etc.

Creating the Tunnel

Once you have your domains transferred to Cloudflare, go into your Cloudflare dashboard and create your tunnel. Start by clicking on Zero Trust on the sidebar. Then go to Networks > Tunnels and click on Create Tunnel

Select the Cloudflared option

You will be asked to name your connector — pick any name that suits you, I went with OMV (since my homeserver is an OpenMediaVault server).

Then copy the installation command. Paste it somewhere and extract the really long token that starts with “ey…” as you’ll need it for the next step.

Setting up `Cloudflared`

Set up cloudflared. The following are instructions for OpenMediaVault. Depending on your home server setup, you may need to do different things to get a Docker container up and running using Docker compose but the Docker compose file and the general order of operations should match. Assuming you use OpenMediaVault…

If you haven’t already, make sure you have OMV Extras and Docker Compose installed (refer to the section Docker and OMV-Extras in my previous post, you’ll want to follow all 10 steps as I refer to different parts of the process throughout this post) and have a static local IP address assigned to your server.
Login to your OpenMediaVault web admin panel, and then go to [Services > Compose > Files] in the sidebar. Press the button in the main interface to add a new Docker compose file.

Under Name put down cloudflared and under File, adapt the following. Copy the token from the installation command

services: 
  cloudflared: 
    image: cloudflare/cloudflared 
    container_name: cloudflare-tunnel 
    restart: unless-stopped 
    command: tunnel run 
    environment: 
      - TUNNEL_TOKEN={{the long token from before that starts with ey...}}

Once you’re done, hit Save and you should be returned to your list of Docker compose files. Notice that the new Cloudflared entry you created has a Down status, showing the container has yet to be initialized.
To start your Duplicati container, click on the new Cloudflared entry and press the (up) button. This will create the container, download any files needed, and run it.

Go back to your Cloudflare Zero Trust dashboard and click on Networks > Tunnels. If your Docker container worked, you should see a HEALTHY status showing that your Cloudflared container is up and running and connected to Cloudflare

Connecting your Services to the Tunnel

Click on your now active tunnel in the Cloudflare interface and click on Edit (or use the three-dot menu on the right hand side and select Configure) and then click on the Public Hostnames tab at the top. Press the Add a public hostname button.

For each service you want to make available, you will need to enter:

The Domain you wish to use (and have transferred to Cloudflare)
- The Subdomain you want to map that service to — if the domain you wish to use is example.com, an example subdomain would be subdomain.example.com. If you leave this blank, it will map the “naked” domain (in this case example.com)
- The Path you want to map the service to — if the domain and subdomain is subdomain.example.com and you add a path /path, then the service would be mapped to subdomain.example.com/path
The Type of service — Cloudflare will map many different types of resources, but chances are it’ll be HTTP.
The URL of the service relative to your network — this is the IP address (including port) that you use within your network. For example: 192.168.85.22:5678 (assuming your home server’s local IP is 192.168.85.22 and the port the service you want to link is set to 5678)

Press Save once you’re done and go ahead and test the subdomain/domain/path you just added (i.e. go to https://subdomain.example.com/path). It should take you straight to your application, except now it’s through a publicly accessible URL secured behind Cloudflare SSL!

Suggestions on Service Configuration

You need to repeat the above process for every selfthosted application that you want to make publicly available. Some suggestions based on what I did:

I made public every service I host with a few exceptions related to security, such as:
- The OpenMediaVault console & WeTTY — Since this controls my entire home server setup (and grants access to all my network attached storage), it felt a little too important to make it easy to access (at least not without a VPN-like solution like the one I use, Twingate)
- The PiHole administrative console — Similarly, because my PiHole is so vital to how the internet functions on my home network and regulates DNS in my home, it felt like locking this behind Twingate was reasonable
- The NAS — As there are important and sensitive files on the OpenMediaVault file server, this was again one of the things where security trumped expediency.
- Duplicati — I was less concerned about security here, but Duplicati is largely a “set it and forget it” type of backup tool, so it felt like there was little benefit to make this publicly available (and only potential risks)
- The Ubooquity Admin interface — I’m again not super concerned about security here, but I have rarely needed to use it, so it didn’t make sense to add to my “surface area of attack” by exposing this as well
For a media server like Plex (or Jellyfin or Emby), you don’t have to, but I’d encourage you to connect two domains:
- One that is easily memorable by you (i.e. plex.yourdomain.com) for you to access via browser over HTTPS and protected by authentication and access control (see later in the post for how to configure)
- One that has a long, hard-to-guess subdomain (i.e. hippo-oxygen-face.yourdomain.com) that will still be served over HTTPS but will not be protected by authentication. This will allow access to devices like smart TVs and the Plex clients which do not expect the servers to have additional authentication on top of them.If you have Plex and you follow this second suggestion, you can further secure you server by going into your Plex configuration panel from a browser and pressing the wrench icon in the upper right (which takes you to settings)
Under your server settings (not Plex Web or your account settings which are above), go to Settings > Remote Access and press the Disable Remote Access button. This disables Plex’s built-in Relay feature which, while reasonably functional, is not under your control and limited in bandwidth / typically forces your server to transcode more than necessary

To allow Plex apps (such as those on a TV or smartphone) to access your server, you’ll need to let Plex know what the right URL is. To do that go to Settings > Network and scroll down to Customer server access URLs. Here you’ll enter your hard-to-guess subdomain (i.e. https://hippo-oxygen-face.yourdomain.com) and press Save Changes. This informs Plex (and therefore all Plex clients) where to look for your media server

To confirm it all works, login to your Plex account at <https://app.plex.tv/> and confirm that your server shows up (you may have to wait the first time you do this as Plex connects to your server).

Because this approach does NOT have extra access control and authentication, and because there are malicious actors who scan the internet for unguarded media server domains, it’s important that your subdomain here be long and hard-to-guess.

Authentication and Access Control

Because Cloudflare Tunnels are part of Cloudflare’s enterprise offering to help IT organizations make their applications secure & accessible, it comes with authentication support and access controls built-in for any application connected to your Cloudflare tunnel. This means you can easily protect your web applications against unwanted access.

To set this up, log back in to the Cloudflare dashboard, go to Zero Trust, and then go to Access > Policies in the sidebar and press the Add a policy button.

Enter a Policy name (pick something that describes how you’re restricting access, like “Jack and Jill only“).

You can then add the specific rules that govern the policy. Cloudflare supports a wide range of rules (including limiting based on IP address, country, etc), but assuming you just want to restrict access to specific individuals, I’d pick Emails under Selector and add the emails of the individual who are being granted access under Value. Once you’re set, press the Save button at the bottom!

Now you have a policy which can restrict a given application only to users with specific email addresses 🙌🏻.

Now, we just need to set up Cloudflare to apply that policy (and a specific login method) to the services in question. To do that, in the Cloudflare Zero Trust dashboard, go to Access > Applications in the sidebar and press the Add an application button in the screen that comes up.

Select the Self-hosted option. And then enter your Application name. Press the Add public hostname button and enter in the Subdomain, Domain, and Path for your previously-connected subdomain.

Scroll down to Access Policies and press the Select existing policies button and check the policy you just created and then hit the Confirm button. You should see something like the following

Finally you can configure which login methods you want to support. Out of the box, Cloudflare supports one-time PIN as a login method. Any user who lands on the domain in question the first time will be prompted to enter their email and, to verify the user is who they say they are, they’ll be sent a PIN number to that email address which they’ll then need to enter. This is straightforward, and if that’s all you want, accept the current default settings.

However, if, like me, you prefer to have your users login via a 3rd party authentication service (like Google or Facebook), then you have a little bit of extra work to do. Press the Manage login methods link where you’ll be taken to a screen in a new tab to configure your Authentication options. Where it says Login methods, press the Add new button.

You’ll be given the ability to add support for 3rd party logins through a number of identity providers (see below).

You can select any identity provider you wish — I went with Google — but whatever you select, Cloudflare will provide instructions for how to connect that provider to Cloudflare Zero Trust. These instructions can be quite complicated (see the Google instructions below) but if you follow Cloudflare’s instructions, you should be fine.

Once you’re done, press the Save button and return to the tab where you were configuring the application.

Under Login methods you should see that Cloudflare has checked the Accept all available identity providers toggle. You can keep that option, but as I configured only want my users to use Google, I unchecked that toggle and un-selected the One-time PIN option. I also checked the Instant Auth option (only available if there’s only one authentication method selected) which skips the authentication method selection step for your users. Then I pressed Next

The next two screens have additional optional configuration options which you can skip through by pressing Next and Save. Et voila! You have now configured an authentication and access control system on top of your now publicly accessible web service. Repeat this process for every service you want to put authentication & access control on and you’ll be set!

I have a few services I share access to with my wife and a few that are just for myself and so I’ve configured two access policies which I apply to my services differently. For services I intend to let anyone without access control reach (for example my Plex server for Plex apps), I simply don’t add them as an application in Cloudflare for access control (and just host them via subdomain).

I hope this is helpful for anyone who wants to make their selfhosted services accessible securely through the web. If you’re interested in how to setup a home server on OpenMediaVault or how to self-host different services, check out all my posts on the subject!

LLMs Get Trounced at Chess

Thu, 10 Jul 2025 00:00:00 GMT

While Large Language Models (LLMs) have demonstrated they can do many things well enough, it’s important to remember that these are not “thinking machines” so much as impressively competent “writing machines” (able to figure out what words are likely to follow).

Case in point: both OpenAI’s ChatGPT and Microsoft Copilot lost to the chess playing engine of an old Atari game (Video Chess) which takes up a mere 4 KB of memory to work (compared with the billions of parameters and GB’s of specialized accelerator memory needed to make LLMs work).

It’s a small (yet potent) reminder that (1) different kinds of AI are necessary for different tasks (i.e. Google’s revolutionary AlphaZero probably would’ve made short work of the Atari engine) and (2) don’t underestimate how small but highly specialized algorithms can perform.

The War on Harvard’s Private Equity/VC Collateral Damage

Wed, 02 Jul 2025 00:00:00 GMT

Republicans have declared a “war on Harvard” in recent months and one front of that is a request to the SEC to look at how Harvard’s massive endowment values illiquid assets like venture capital and private equity.

What’s fascinating is that in targeting Harvard in this way the Republicans may have declared war on Private Equity and Venture Capital in general. As their holdings (in privately held companies) are highly illiquid, it is considered accounting “standard practice” to simply ask the investment funds to provide “fair market” valuations of those assets.

This is a practical necessity, as it is highly difficult to value these companies (which rarely trade and where even highly paid professionals miss the mark). But, it means that investment firms are allowed to “grade their own homework”, pretending that valuations for some companies are much higher than they actually have a right to be, resulting in quite a bit of “grade inflation” across the entire sector.

If Harvard is forced to re-value these according to a more objective standard — like the valuations of these assets according to a 409a valuation or a secondary transaction (where shares are sold without the company being involved) both of which artificially deflate prices — then it wouldn’t be a surprise to see significant “grade deflation” which could have major consequences for private capital:

Less capital for private equity / venture capital: Many institutional investors (LPs) like private equity / venture capital in part because the “grade inflation” buffers the price turbulence that more liquid assets (like stocks) experience (so long as the long-term returns are good). Those investors will find private equity and venture capital less attractive if the current practices are replaced with something more like “grade deflation”
A shift in investments from higher risk companies to more mature ones: If Private Equity / Venture Capital investments need to be graded on a harsher scale, they will be less likely to invest in higher risk companies (which are more likely to experience valuation changes under stricter methodologies) and more likely to invest in more mature companies with more predictable financials (ones that are closer to acting like publicly traded companies). This would be a blow to smaller and earlier stage companies.

CAR-T Bests Solid Tumors

Tue, 03 Jun 2025 00:00:00 GMT

The promise of immune cell therapies is to direct the incredible sophistication of the immune system to the medical targets we want to hit. In the case of CAR-T cells, we take a patient’s own immune T-cells and “program” them through genetic modification to go after cancer cells. While the process of the genetic modification to create those cells is still incredibly expensive and challenging, we’ve seen amazing progress with CAR-T in liquid tumors (e.g., leukemias, lymphomas, etc).

But, when it comes to solid tumors, it’s been far more challenging. Enter this Phase II clinical trial from China (summarized in Nature News). The researchers performed a random controlled trial on 266 patients with gastric or gastro-esophageal cancer who resisted previous treatment and assigned 2/3 to receive CAR-T or best-medical-care (the control) otherwise. The results (see the survival curve below) are impressive — while the median progression-free survival is only about 1.5 months different, it’s very clear that by month 8 there are no progression-free patients in the control group but something like ~25% of the CAR-T group.

The side effect profile is still challenging (with 99% of patients in CAR-T group experiencing moderately severe side effects) but this is (sadly) to be expected with CAR-T treatments.

While it remains to be seen how this scales up in a Phase III study with a larger population, this is incredibly promising finding — giving clinicians a new tool in their arsenal for dealing with a wider range of cancer targets as well as suggesting that cell therapies still have more tricks up their sleeves

Reading Excerpt • Nature News

”

“The phase II clinical trial in China tested chimeric antigen receptor (CAR) T cells in people with advanced gastric cancer or gastro-oesophageal junction cancer, which are solid tumours. To create CAR-T-cell therapies, T cells are collected from a person with cancer and tweaked to produce proteins that target cancer cells. The T cells are then infused back into the same person. CAR-T-cell therapy has revolutionized cancer treatment but has been most successful against blood cancers. “Solid tumours generally don’t respond well to CAR-T-cell therapy,” says Lisa Mielke, a cancer researcher at the Olivia Newton John Cancer Research Institute in Heidelberg, Australia. The trials are among the first in which CAR-T-cell therapy has had promising results against solid tumours. They provide “evidence that there is potential for CAR T cells to be further optimized for future treatment of patients with solid tumours”, adds Mielke.”

Cancer-fighting CAR T cells show promising results for hard-to-treat tumours

Rachel Fieldhouse | Nature News

Great Expectations

Sun, 01 Jun 2025 00:00:00 GMT

This is an old piece from Morgan Housel from May 2023. It highlights how optimistic expectations can serve as a “debt” that needs to be “paid off”.

To illustrate this, he gives a fascinating example — the Japanese stock market. From 1965 to 2022, both the Japanese stock market and the S&P500 (a basket of mostly American large companies) had similar returns. As most people know, Japan has had a miserable 3 “lost decades” of growth and stock performance. But Housel presents this fact in an interesting light: it wasn’t that Japan did poorly, it just did all of its growth in a 25 year run between 1965-1990 and then spent the following two decades “paying off” that “expectations debt”.

Housel concludes, as he oftentimes does, with wisdom for all of us: “An asset you don’t deserve can quickly become a liability … reality eventually catches up, and demands repayment in equal proportion to your delusions – plus interest”.

Manage your great expectations.

Why Self-Hosting is Having a Moment

Sat, 24 May 2025 00:00:00 GMT

I am a selfhoster (as anyone who’s read many of my more recent blog posts knows). I’m also a fan of the selfh.st site (which documents a lot of news & relevant interviews from the self-host world) so I was delighted to see the owner of selfh.st get interviewed in Ars Technica.

Nothing earth-shattering but I appreciated (and agreed with) his breakdown of why self-hosting is flourishing today (excerpt below). For me, personally, the ease with which Docker makes setting up selfhosted services and the low cost of storage and mini-PCs turned this from an impractical idea into one that I’ve come to rely on for my own “personal tech stack”.

Reading Excerpt • Ars Technica

”

“What gave rise to self-hosting’s relative recent popularity? That led Sholly to a few answers, many of them directly relating to the corporate cloud services people typically use instead of self-hosting: Privacy for photos, files, and other data Cost of cloud hosting and storage Accessibility of services, through GitHub, Reddit, and sites like his Installation with Docker (“a game-changer for lots of people”) and Unraid Single-board computers (SBCs) like the Raspberry Pi NUCS, mini-PCs, workstations, and other pandemic-popular hardware Finally, there’s the elephant in any self-hosting discussion: piracy.”

Self-hosting is having a moment. Ethan Sholly knows why

Kevin Purdy | Ars Technica

I’m just a Medical Guideline

Tue, 21 Jan 2025 00:00:00 GMT

Medical Guidelines are incredibly important — they impact everything from your doctor’s recommendations and insurance coverage to the medications your insurance covers — but are somewhat shrouded in mystery.

This piece from Emily Oster’s ParentData is a good overview of what they are (and aren’t) — and give a pretty good explanation of why a headline from the popular press is probably not capturing the nuance and review of clinical evidence that goes into them.

(and yes, that title is a Schoolhouse Rock reference)

On (Stock Market) Bubbles

Mon, 13 Jan 2025 00:00:00 GMT

I spotted this memo from Oaktree Capital founder Howard Marks and thought it was a sobering and grounded take on what makes a stock market bubble and reasons to be alarmed about the current concentration of market capitalization in the so-called “Magnificent Seven” and how eerily similar this was to the “Nifty Fifty” or the “Dot Com Bubble” eras of irrational exuberance. Whether you agree with him or not, it’s a worthwhile piece of wisdom to remember.

This graph that Marks borrowed from JP Morgan is also quite intriguing (terrifying?)

Reading Excerpt • Howard Marks

”

“There’s usually a grain of truth that underlies every mania and bubble. It just gets taken too far. It’s clear that the internet absolutely did change the world – in fact, we can’t imagine a world without it. But the vast majority of internet and e-commerce companies that soared in the late ’90s bubble ended up worthless. When a bubble burst in my early investing days, The Wall Street Journal would run a box on the front page listing stocks that were down by 90%. In the aftermath of the TMT Bubble, they’d lost 99%. When something is on the pedestal of popularity, the risk of a decline is high. When people assume – and price in – an expectation that things can only get better, the damage done by negative surprises is profound. When something is new, the competitors and disruptive technologies have yet to arrive. The merit may be there, but if it’s overestimated it can be overpriced, only to evaporate when reality sets in. In the real world, trees don’t grow to the sky.”

On Bubble Watch

Howard Marks

Pivoting from Consumer to Utility: Span

Fri, 10 Jan 2025 00:00:00 GMT

As a Span customer, I’ve always appreciated their vision: to make home electrification cleaner, simpler, and more efficient through beautifully designed, tech-enabled electrical panels. But, let’s be honest, selling a product like this directly to consumers is tough. Electrical panels are not top-of-mind for most people until there’s a problem — and explaining the value proposition of “a smarter electrical panel” to justify the high price tag can be a real challenge. That’s why I’m unsurprised by their recent shift in strategy towards utilities.

This pivot to partnering with utility companies makes a lot of sense. Instead of trying to convince individual homeowners to upgrade, Span can now work directly with those who can impact community-scale electrification.

While the value proposition of avoiding costly service upgrades is undeniably beneficial for utilities, understanding precisely how that translates into financial savings for the utilities needs much more nuance. That, along with the fact that rebates & policy will vary wildly by locality, raises many uncertainties about pricing strategy (not to mention that there are other, larger smart electric panel companies like Leviton and Schneider Electric, albeit with less functional and less well-designed offerings).

I wish the company well. We need better electrical infrastructure in the US (and especially California, where I live) and one way to achieve that is for companies like Span to find a successful path to market.

RISC-V in Computers

Thu, 09 Jan 2025 00:00:00 GMT

One of the most exciting technological developments from the semiconductor side of things is the rapid development of the ecosystem around the open-source RISC-V instruction set architecture (ISA). One landmark in its rise is that the architecture appears to be moving beyond just behind-the-scenes projects to challenging Intel/AMD’s x86 architecture and ARM (used by Apple and Qualcomm) in customer-facing applications.

This article highlights this crucial development by reporting on early adopters embracing RISC-V to move into higher-end devices like laptops. Companies like Framework and DeepComputing have just launched or are planning to launch RISC-V laptops. While RISC-V-powered hardware still have a steep mountain to climb of software and performance challenges (as evidenced by the amount of time it’s taken for the ARM ecosystem to be credible in PCs), Intel’s recent setbacks and ARM’s legal battles with Qualcomm over licensing (pretty much guaranteeing every company that uses ARM is now going to work on RISC-V) coupled with the open source nature of RISC-V potentially allowing for a lot more innovation in form factors and functionality may have created an opening here for enterprising companies willing to make the investment.

Revenge of the Plug-In Hybrid

Mon, 06 Jan 2025 00:00:00 GMT

While growing vehicle electrification is inevitable, it always surprised me that US automakers would drop past plug-in hybrid (PHEV) technology to only embrace all-electric. While many have attacked Toyota’s more deliberate “slow-and-steady” approach to vehicle electrification, it always seemed to me that, until we had broadly available, high quality electric vehicle charging infrastructure and until all-electric vehicles were broadly available at the price point of a non-luxury family car (i.e. a Camry or RAV4), that electric vehicles were going to be more of a upper middle class/wealthy phenomena. Considering their success in the Chinese automotive market (and growing faster than all-electric vehicles!), it always felt odd that the category wouldn’t make its way into the US market as the natural next step in vehicle electrification.

It sounds like Dodge Ram (a division of Stellantis) agrees. It intends to delay its all-electric version of its Ram 1500 in favor of starting with its extended range plug-in hybrid version, the Ramcharger. Extended range electric vehicles (EREVs) are plug-in hybrids similar to the Chevy Volt. They employ an electric powertrain and a generator which can run on gasoline to supply additional range when the battery runs low.

Interestingly, Nissan, Hyundai, Mazda, and GM’s Buick have made similar announcements as well.

While it still remains to be seen how well these EREVs/PHEVs are adopted — the price points that are being discussed still feel too high to me — seeing broader adoption of plug-in hybrid technology (supplemented with gas-powered range extension) feels like the natural next step on our path to vehicle electrification.

Helping Multi-Agent AI Experimentation

Sun, 22 Dec 2024 00:00:00 GMT

Inspired by some work from a group at Stanford on building a lab from AI agents, I’ve been experimenting with multi-agent AI conversations and workflows. But, because the space (at least to me) has seemed more focused on building more capable agents rather than coordinating and working with more agents, the existing tools and libraries have made it difficult to carry out fast experiments.

To facilitate some of my own exploration work, I built what I’m calling the Multi-Agent ChatLab — a browser-based, completely portable setup to define multiple AI agents and facilitate conversations between them. This has made my experimentation work vastly simpler and I hope it can help others.

Interactive Web Application

MultiAgent ChatLab (v1.0)

Open Fullscreen ↗

Design & Architecture: Read the full technical breakdown and system instructions in the Multi-Agent ChatLab Portfolio Article.
Open Source Code: Grab the single-file source directly from the GitHub Repository.
Direct Web Access: Launch the standalone applet in a separate window at /multiagent-chatlab.html.

And, to show off the tool, and for your amusement (and given my love of military history), here is a screengrab from the tool where I set up two AI Agents — one believing itself to be Napoleon Bonaparte and one believing itself to be the Duke of Wellington (the British commander who defeated Napoleon at Waterloo) — and had them describe (and compare!) the hallmarks of their military strategy.

Decarbonizing Shipping with Wind

Thu, 19 Dec 2024 00:00:00 GMT

The shipping industry is known for being fairly dirty environmentally due largely to the fact that the most common fuel used in shipping — bunker fuel — contributes both to carbon emissions, significant air pollution, and water pollution (from spills and due to the common practice of dumping the byproduct of sulphur scrubbing to curtail air pollution).

While much of the effort to green shipping has focused on the use of alternative fuels like hydrogen, ammonia and methanol as replacements for bunker fuel, I recently saw an article on the use of automated & highly durable sail technology to le ships leverage wind as a means to reduce fuel consumption.

I don’t have any inside information on what the cost / speed tradeoffs are for the technology, nor whether or not there’s a credible path to scaling to handle the massive container ships that dominate global shipping, but it’s a fascinating technology vector, and a direct result of the growing realization by the shipping industry that it needs to green itself.

Google’s Quantum Error Correction Breakthrough

Wed, 11 Dec 2024 00:00:00 GMT

One of the most exciting areas of technology development, but that doesn’t get a ton of mainstream media coverage, is the race to build a working quantum computer that exhibits “below threshold quantum computing” — the ability to do calculations utilizing quantum mechanics accurately.

One of the key limitations to achieving this has been the sensitivity of quantum computing systems — in particular the qubits that capture the superposition of multiple states that allow quantum computers to exploit quantum mechanics for computation — to the world around them. Imagine if your computer’s accuracy would change every time someone walked in the room — even if it was capable of amazing things, it would not be especially practical. As a result, much research to date has been around novel ways of creating physical systems that can protect these quantum states.

Google has (in a pre-print in Nature) demonstrated their new Willow quantum computing chip which demonstrates a quantum error correction method that spreads the quantum state information of a single “logical” qubit across multiple entangled “physical” qubits to create a more robust system. Beyond proving that their quantum error correction method worked, what is most remarkable to me, is that they’re able to extrapolate a scaling law for their error correction — a way of guessing how much better their system is at avoiding loss of quantum state as they increase the number of physical qubits per logical qubit — which could suggest a “scale up” path towards building functional, practical quantum computers.

I will confess that quantum mechanics was never my strong suit (beyond needing it for a class on statistical mechanics eons ago in college), and my understanding of the core physics underlying what they’ve done in the paper is limited, but this is an incredibly exciting feat on our way towards practical quantum computing systems!

Reading Excerpt • Nature News

”

“The company’s new chip, called Willow, is a larger, improved version of that technology, with 105 physical qubits. It was developed in a fabrication laboratory that Google built at its quantum-computing campus in Santa Barbara, California, in 2021. As a first demonstration of Willow’s power, the researchers showed that it could perform, in roughly 5 minutes, a task that would take the world’s largest supercomputer an estimated 1025 years, says Hartmut Neven, who heads Google’s quantum-computing division. This is the latest salvo in the race to show that quantum computers have an advantage over classical ones. And, by creating logical qubits inside Willow, the Google team has shown that each successive increase in the size of a logical qubit cuts the error rate in half. “This is a very impressive demonstration of solidly being below threshold,” says Barbara Terhal, a specialist in quantum error correction at the Delft University of Technology in the Netherlands. Mikhail Lukin, a physicist at Harvard University in Cambridge, Massachusetts, adds, “It clearly shows that the idea works.””

\u2018A truly remarkable breakthrough\u2019: Google\u2019s new quantum chip achieves accuracy milestone

Davide Castelvecchi | Nature News

Cynefin

Sun, 08 Dec 2024 00:00:00 GMT

I had never heard of this framework for thinking about how to address problems before. Shout-out to my friend Chris Yiu and his new Substack Secret Weapon about improving productivity for teaching me about this. It’s surprisingly insightful about when to think about something as a process problem vs an expertise problem vs experimentation vs direction.

Reading Excerpt • Secret Weapon

”

“[The Cynefin framework] organises problems into four primary domains: Clear. Cause and effect are obvious; categorise the situation and apply best practice. Example: baking a cake. Rewards process. Complicated. Cause and effect are knowable but not immediately apparent; analyse the situation carefully to find a solution. Example: coaching a sports team. Rewards expertise. Complex. Cause and effect are only apparent with hindsight; focus on spotting and exploiting patterns as they emerge. Example: playing poker. Rewards [experimentation — Chris’s original term was entrepreneurship, I think experimentation is more clear & actionable]. Chaotic. Cause and effect are impossible to parse; act on instinct, in the hope of imposing some order on the immediate situation. Example: novel crisis response. Rewards direction.”

Problems come in many forms

Chris Yiu | Secret Weapon

The Hits Business — Games Edition

Fri, 06 Dec 2024 00:00:00 GMT

The best return on investment in terms of hours of deep engagement per dollar in entertainment is with games. When done right, they blend stunning visuals and sounds, earworm-like musical scores, compelling story and acting, and a sense of progression that are second to none.

Case in point: I bought the complete edition of the award-winning The Witcher 3: Wild Hunt for $10 during a Steam sale in 2021. According to Steam, I’ve logged over 200 hours (I had to doublecheck that number!) playing the game, between two playthroughs and the amazing expansions Hearts of Stone and Blood and Wine — an amazing 20 hours/dollar spent. Even paying full freight (as of this writing, the complete edition including both expansions costs $50), that would still be a remarkable 4 hours/dollar. Compare that with the price of admission to a movie or theater or concert.

The Witcher 3 has now surpassed 50 million sales — comfortably earning over $1 billion in revenue which is an amazing feat for any media property.

But as amazing and as lucrative as these games can be, these games cannot escape the cruel hit-driven basis of their industry, where a small number of games generate the majority of financial returns. This has resulted in studios chasing ever more expensive games with familiar intellectual property (i.e. Star Wars) that has, to many game players, cut the soul from the games and has led to financial instability in even popular game studios.

This article from IGN summarizes the state of the industry well — with so-called AAA games now costing $200 million to create, not to mention $100’s of millions to market, more and more studios have to wind down as few games can generate enough revenue to cover the cost of development and marketing.

The article predicts — and I hope it’s right — that the games industry will learn some lessons that many studios in Hollywood/the film industry have been forced to: embrace more small budget games to experiment with new forms and IP. Blockbusters will have their place but going all-in on blockbusters is a recipe for a hollowing out of the industry and a cutting off of the creativity that it needs.

Or, as the author so nicely puts it: “Maybe studios can remember that we used to play video games because they were fun – not because of their bigger-than-last-year maps carpeted by denser, higher-resolution grass that you walk across to finish another piece of side content that pushes you one digit closer to 100% completion.”

Reading Excerpt • IGN

”

“Just five years ago, AAA projects’ average budget ranged $50 – $150 million. Today, the minimum average is $200 million. Call of Duty’s new benchmark is $300 million, with Activision admitting in the Competition & Market Authority’s report on AAA development that it now takes the efforts of one-and-a-half studios just to complete the annual Call of Duty title. It’s far from just Call of Duty facing ballooning costs. In the same CMA report, an anonymous publisher admits that development costs for one of its franchises reached $660 million. With $550 million of marketing costs on top, that is a $1.2 billion game. To put that into perspective, Minecraft – the world’s best-selling video game of all time – has of last year only achieved $3 billion. It took 12 years to reach that figure, having launched in 2011.”

The Blockbuster Games Bubble Has Burst

Sab Astley | IGN

Tabs: 1997-2023 P&L History ↗ Clinical & Utility Tabs ↗ Cost-Effectiveness Model ↗

Why is it so Hard to Build a Diagnostic Business?

Wed, 04 Dec 2024 00:00:00 GMT

Everywhere you look, the message seems clear: early detection (of cancer & disease) saves lives. Yet behind the headlines, companies developing these screening tools face a different reality. Many tests struggle to gain approval, adoption, or even financial viability. The problem isn’t that the science is bad — it’s that the math is brutal.

This piece unpacks the economic and clinical trade-offs at the heart of the early testing / disease screening business. Why do promising technologies struggle to meet cost-effectiveness thresholds, despite clear scientific advances? And what lessons can diagnostic innovators take from these challenges to improve their odds of success? By the end, you’ll have a clearer view of the challenges and opportunities in bringing new diagnostic tools to market—and why focusing on the right metrics can make all the difference.

The brutal math of diagnostics

Image Credit: Wikimedia

Technologists often prioritize metrics like sensitivity (also called recall) — the ability of a diagnostic test to correctly identify individuals with a condition (i.e., if the sensitivity of a test is 90%, then 90% of patients with the disease will register as positives and the remaining 10% will be false negatives) — because it’s often the key scientific challenge and aligns nicely with the idea of getting more patients earlier treatment.

But when it comes to adoption and efficiency, specificity — the ability of a diagnostic test to correctly identify healthy individuals (i.e., if the specificity of a test is 90%, then 90% of healthy patients will register as negatives and the remaining 10% will be false positives) — is usually the more important and overlooked criteria.

The reason specificity is so important is that it can have a profound impact on a test’s Positive Predictive Value (PPV) — whether or not a positive test result means a patient actually has a disease (i.e., if the positive predictive value of a test is 90%, then a patient that registers as positive has a 90% chance of having the disease and 10% chance of actually being healthy — being a false positive).

What is counter-intuitive, even to many medical and scientific experts, is that because (by definition) most patients are healthy, many high accuracy tests have disappointingly low PPV as most positive results are actually false positives.

Let me present an example (see table below for summary of the math) that will hopefully explain:

There are an estimated 1.2 million people in the US with HIV — that is roughly 0.36% (the prevalence) of the US population
Let’s say we have an HIV test with 99% sensitivity and 99% specificity — a 99% (very) accurate test!
If we tested 10,000 Americans at random, you would expect roughly 36 of them (0.36% x 10,000) to be HIV positive. That means, roughly 9,964 are HIV negative
- 99% sensitivity means 99% of the 36 HIV positive patients will test positive (99% x 36 = ~36)
- 99% specificity means 99% of the 9,964 HIV negative patients will test negative (99% x 9,964 = ~9,864) while 1% (1% x 9,964 = ~100) would be false positives
This means that even though the test is 99% accurate, it only has a positive predictive value of ~26% (36 true positives out of 136 total positive results)

Math behind the hypothetical HIV test example (Google Sheet link)

Below (if you’re on a browser) is an embedded calculator which will run this math for any values of disease prevalence and sensitivity / specificity (and here is a link to a Google Sheet that will do the same), but you’ll generally find that low disease rates result in low positive predictive values for even very accurate diagnostics.

Interactive Clinical Model

Diagnostic PPV & NNS Calculator

Operate Mode

Clinical Presets:

Sensitivity (%)

Specificity (%)

Prevalence (%)

Test Accuracy

84.45%

Positive Predictive Value (PPV)

3.73%

Number Needed to Screen (NNS)

166.68

Typically, introducing a new diagnostic means balancing true positives against the burden of false positives. After all, for patients, false positives will result in anxiety, invasive tests, and, sometimes, unnecessary treatments. For healthcare systems, they can be a significant economic burden as the cost of follow-up testing and overtreatment add up, complicating their willingness to embrace new tests.

Below (if you’re on a browser) is an embedded calculator which will run the basic diagnostic economics math for different values of the cost of testing and follow-up testing to calculate the cost of testing and follow-up testing per patient helped (and here is a link to a Google Sheet that will do the same)

Interactive Health-Economic Model

Diagnostic Screening Economics Calculator

Operate Mode

Sensitivity (%)

Specificity (%)

Prevalence (%)

Cost of Test ($)

Cost of Follow-Up Diagnostic ($)

Test Accuracy

84.45%

PPV

3.73%

NNS

166.68

Cost / True Positive

$123,548

Finally, while diagnostics businesses face many of the same development hurdles as drug developers — the need to develop cutting-edge technology, to carry out large clinical studies to prove efficacy, and to manage a complex regulatory and reimbursement landscape — unlike drug developers, diagnostic businesses face significant pricing constraints. Successful treatments can command high prices for treating a disease. But successful diagnostic tests, no matter how sophisticated, cannot, because they ultimately don’t treat diseases, they merely identify them.

Case Study: Exact Sciences and Cologuard

Let’s take Cologuard (from Exact Sciences) as an example. Cologuard is a combination genomic and immunochemistry test for colon cancer carried out on patient stool samples. It’s two primary alternatives are:

a much less sensitive fecal immunochemistry test (FIT) — which uses antibodies to detect blood in the stool as a potential, imprecise sign of colon cancer
colonoscopies — a procedure where a skilled physician uses an endoscope to enter and look for signs of cancer in a patient’s colon. It’s considered the “gold standard” as it functions both as diagnostic and treatment (a physician can remove or biopsy any lesion or polyp they find). But, because it’s invasive and uncomfortable for the patient, this test is typically only done every 4-10 years

Cologuard is (as of this writing) Exact Science’s primary product line, responsible for a large portion of Exact Science’s $2.5 billion in 2023 revenue. It can detect earlier stage colon cancer as well as pre-cancerous growths that could lead to cancer. Impressively, Exact Sciences also commands a gross margin greater than 70%, a high margin achieved mainly by pharmaceutical and software companies that have low per-unit costs of production. This has resulted in Exact Sciences, as of this writing, having a market cap over $11 billion.

Yet for all its success, Exact Sciences is also a cautionary note, illustrating the difficulties of building a diagnostics company.

The company was founded in 1995, yet didn’t see meaningful revenue from selling diagnostics until 2014 (nearly 20 years later, after it received FDA approval for Cologuard)
The company has never had a profitable year (this includes the last 10 years it’s been in-market), losing over $200 million in 2023, and in the first three quarters of 2024, it has continued to be unprofitable.
Between 1997 (the first year we have good data from their SEC filings as summarized in this Google Sheet) and 2014 when it first achieved meaningful diagnostic revenue, Exact Sciences lost a cumulative $420 million, driven by $230 million in R&D spending, $88 million in Sales & Marketing spending, and $33 million in CAPEX. It funded those losses by issuing over $624 million in stock (diluting investors and employees)
From 2015-2023, it has needed to raise an additional $3.5 billion in stock and convertible debt (net of paybacks) to cover its continued losses (over $3 billion from 2015-2023)
Prior to 2014, Exact Sciences attempted to commercialize colon cancer screening technologies through partnerships with LabCorp (ColoSure and PreGenPlus). These were not very successful and led to concerns from the FDA and insurance companies. This forced Exact Sciences to invest heavily in clinical studies to win over the payers and the FDA, including a pivotal ~10,000 patient study to support Cologuard which recruited patients from over 90 sites and took over 1.5 years.
It took Exact Sciences 3 years after FDA approval of Cologuard for its annual diagnostic revenues to exceed what it spends on sales & marketing. It continues to spend aggressively there ($727M in 2023).

Interactive Spreadsheet Model

Exact Sciences 1997–2023 Longitudinal Financial Model

Historical financial analysis tracking 27 years of SEC 10-K filings: cumulative net losses ($4.1B+), R&D vs. Sales & Marketing expenditures, and equity dilution.

Tabs: Clinical Comparison (2014) ↗ Cost per Cancer Found ↗ 2024 Cologuard+ Data ↗

While it’s difficult to know precisely what the company’s management / investors would do differently if they could do it all over again, the brutal math of diagnostics certainly played a key role.

From a clinical perspective, Cologuard faces the same low positive predictive value problem all diagnostic screening tests face. From the data in their study on ~10,000 patients, it’s clear that, despite having a much higher sensitivity for cancer (92.3% vs 73.8%) and higher AUROC (94% vs 89%) than the existing FIT test, **the PPV of Cologuard is only 3.7% (**lower than the FIT test: 6.9%)****.

Even using a broader disease definition that includes the pre-cancerous advanced lesions Exact Sciences touted as a strength, the gap on PPV does not narrow (Cologuard: 23.6% vs FIT: 32.6%)

Interactive Spreadsheet Model

Cologuard vs. FIT Head-to-Head Clinical & Economic Model

Clinical utility and unit economics modeling comparing Cologuard and FIT screening across 10,000+ patient trial data (NEJM 2014 & CMS fee schedule).

From ACS Clinical Congress 2022 Presentation

The economic comparison with a FIT test fares even worse due to the higher cost of Cologuard as well as the higher rate of false positives. Under the Center for Medicare & Medicaid Service’s 2024Q4 laboratory fee schedule, a FIT test costs $16 (CPT code: 82274), but Cologuard costs $509 (CPT code: 81528), over 30x higher! If each positive Cologuard and FIT test results in a follow-up colonoscopy (which has a cost of $800-1000 according to this 2015 analysis), the screening cost per cancer patient is 5.2-7.1x higher for Cologuard than for the FIT test.

This quick math has been confirmed in several studies.

A study by a group at the University Medical Center of Rotterdam concluded that “Compared to nearly all other CRC screening strategies reimbursed by CMS (Medicare), [Cologuard] is less effective and considerably more costly, making it an inefficient screening option” and would only be comparable at a much lower cost (~$6-18!)
Another study presented at the American College of Surgeons Clinical Congress in 2022 (see image below) looked at insurance reimbursement in the Alleghany Health Network and found that the FIT test was 5x cheaper in addressing early stage colon cancer.

While Medicare and the US Preventive Services Task Force concluded that the cost of Cologuard and the increase in false positives / colonoscopy complications was worth the improved early detection of colon cancer, it stayed largely silent on comparing cost-efficacy with the FIT test. It’s this unfavorable comparison that has probably required Exact Sciences to invest so heavily in sales and marketing to drive sales. That Cologuard has been so successful is a testament both to the value of being the only FDA-approved test on the market as well as Exact Science’s efforts in making Cologuard so well-known (how many other diagnostics do you know have an SNL skit dedicated to them?).

Not content to rest on the laurels of Cologuard, Exact Sciences recently published a ~20,000 patient study on their next generation colon cancer screening test: Cologuard Plus. While the study suggests Exact Sciences has improved the test across the board, the company’s marketing around Cologuard Plus having both >90% sensitivity and specificity is misleading, because the figures for sensitivity and specificity are for different conditions: sensitivity for colorectal cancer but specificity for colorectal cancer OR advanced precancerous lesion (see the table below).

Sensitivity and Specificity by Condition for Cologuard Plus Study
(Google Sheet link)

Disentangling these numbers shows that while Cologuard Plus has narrowed its PPV disadvantage (now worse by 1% on colorectal cancer and even on cancer or lesion) and its cost-efficacy disadvantage (now “only” 4.4-5.8x more expensive) vs the FIT test (see tables below), it still hasn’t closed the gap.

Clinical: Cologuard+ vs FIT (Google Sheet link)

Economic: Cologuard+ vs FIT (Google Sheet link)

Time will tell if this improved test performance translates to continued sales performance for Exact Sciences, but it is telling that despite the significant time and resources that went into developing Cologuard Plus, the data suggests it’s still likely more cost effective for health systems to adopt FIT over Cologuard Plus as a means of preventing advanced colon cancer.

Lessons for diagnostics companies

The underlying math of the diagnostics business and the lessons from Exact Sciences’ long path to dramatic sales has several key lessons for diagnostic entrepreneurs:

Focus on specificity — For diagnostic technologists, too little attention is paid to specificity while too much attention is paid on sensitivity. Positive predictive value and the cost-benefit for a health system are largely going to swing on specificity.
Aim for higher value tests — Because the development and required validation for a diagnostic can be as high as that of a drug or medical device, it is important to pursue opportunities where the diagnostic can command a high price. These are usually markets where the alternatives are very expensive because they require new technology (e.g. advanced genetic tests) or a great deal of specialized labor (e.g. colonoscopy) or where the diagnostic directly decides on a costly course of treatment (e.g. a companion diagnostic for an oncology drug).
Go after unmet needs — If a test is able to fill a mostly unmet need — for example, if the alternatives are extremely inaccurate or poorly adopted — then adoption will be determined by awareness (because there aren’t credible alternatives) and pricing will be determined by sensitivity (because this drives the delivery of better care). This also simplifies the sales process.
Win beyond the test — Because performance can only ever get to 100%, each incremental point on sensitivity and specificity is both exponentially harder to achieve but also delivers less medical or financial value. As a result, it can be advantageous to focus on factors beyond the test such as regulatory approval / guidelines adoption, patient convenience, time to result, and impact on follow-up tests and procedures. Cologuard gained a great deal from being “the first FDA-approved colon cancer screening test”. Non-invasive prenatal testing, despite low positive predictive values and limited disease coverage, gained adoption in part by helping to triage follow-up amniocentesis (a procedure which has a low but still frighteningly high rate of miscarriage ~0.5%). Rapid antigen tests for COVID have also similarly been adopted despite their lower sensitivity and specificity than PCR tests due to their speed, low cost, and ability to carry out at home.

Diagnostics developers must carefully navigate the intersection of scientific innovation and financial reality, while grappling with the fact that even the most impressive technology may be insufficient without taking into account clinical and economic factors to achieve market success.

Ultimately, the path forward for diagnostic innovators lies in prioritizing specificity, targeting high-value and unmet needs, and crafting solutions that deliver value beyond the test itself. While Exact Science’s journey underscores the difficulty of these challenges, it also illustrates that with persistence, thoughtful investment, and strategic differentiation, it is possible to carve out a meaningful and impactful space in the market.

The Challenge of Capacity

Mon, 25 Nov 2024 00:00:00 GMT

The rise of Asia as a force to be reckoned with in large scale manufacturing of critical components like batteries, solar panels, pharmaceuticals, chemicals, and semiconductors has left US and European governments seeking to catch up with a bit of a dilemma.

These activities largely moved to Asia because financially-motivated management teams in the West (correctly) recognized that:

they were low return in a conventional financial sense (require tremendous investment and maintenance)
most of these had a heavy labor component (and higher wages in the US/European meant US/European firms were at a cost disadvantage)
these activities tend to benefit from economies of scale and regional industrial ecosystems, so it makes sense for an industry to have fewer and larger suppliers
much of the value was concentrated in design and customer relationship, activities the Western companies would retain

What the companies failed to take into account was the speed at which Asian companies like WuXi, TSMC, Samsung, LG, CATL, Trina, Tongwei, and many others would consolidate (usually with government support), ultimately “graduating” into dominant positions with real market leverage and with the profitability to invest into the higher value activities that were previously the sole domain of Western industry.

Now, scrambling to reposition themselves closer to the forefront in some of these critical industries, these governments have tried to kickstart domestic efforts, only to face the economic realities that led to the outsourcing to begin with.

Northvolt, a major European effort to produce advanced batteries in Europe, is one example of this. Despite raising tremendous private capital and securing European government support, the company filed for bankruptcy a few days ago.

While much hand-wringing is happening in climate-tech circles, I take a different view: this should really not come as a surprise. Battery manufacturing (like semiconductor, solar, pharmaceutical, etc) requires huge amounts of capital and painstaking trial-and-error to perfect operations, just to produce products that are steadily dropping in price over the long-term. It’s fundamentally a difficult and not-very-rewarding endeavor. And it’s for that reason that the West “gave up” on these years ago.

But if US and European industrial policy is to be taken seriously here, the respective governments need to internalize that reality and be committed for the long haul. The idea that what these Asian companies are doing is “easily replicated” is simply not true, and the question is not if but when will the next recipient of government support fall into dire straits.

Reading Excerpt • Latitude Media

”

“From the start, Northvolt set out to build something unprecedented. It didn’t just promise to build batteries in Europe, but to create an entire battery ecosystem, from scratch, in a matter of years. It would build the region’s biggest battery factories, develop and source its own materials, and recycle its own batteries. And, with some help from government subsidies, it would do so while matching prices from Asian manufacturers that had dominated global markets. Northvolt’s ambitious attempt to compress decades of industry development into just eight years culminated last week, with its filing for Chapter 11 bankruptcy protection and the departure of several top executives, including CEO Peter Carlsson. The company’s downfall is a setback for Europe’s battery ambitions — as well as a signal of how challenging it is for the West to challenge Chinese dominance.”

What Northvolt\u2019s bankruptcy means for Europe\u2019s battery ambitions

Maeve Allsup | Latitude Media

A Visual Timeline of Human Migration

Fri, 22 Nov 2024 00:00:00 GMT

Beautiful map laying out when humans settled different parts of the world (from 2013, National Geographic’s Out of Eden project)

Updating my AI News Reader

Thu, 21 Nov 2024 00:00:00 GMT

A few months ago, I shared that I had built an AI-powered personalized news reader which I use (and still do) on a near-daily basis. Since that post, I’ve made a couple of major improvements (which I have just reflected in my public Github).

Switching to JAX

I previously chose Keras 3 for my deep learning algorithm architecture because of its ease of use as well as the advertised ability to shift between AI/ML backends (at least between Tensorflow, JAX, and PyTorch). With Keras creator Francois Chollet noting significant speed-ups just from switching backends to JAX, I decided to give the JAX backend a shot.

Thankfully, Keras 3 lived up to it’s multi-backend promise and made switching to JAX remarkably easy. For my code, I simply had to make three sets of tweaks.

First, I had to change the definition of my container images. Instead of starting from Tensorflow’s official Docker images, I instead installed JAX and Keras on Modal’s default Debian image and set the appropriate environmental variables to configure Keras to use JAX as a backend:

jax_image = (
    modal.Image.debian_slim(python_version='3.11')
    .pip_install('jax[cuda12]==0.4.35', extra_options="-U")
    .pip_install('keras==3.6')
    .pip_install('keras-hub==0.17')
    .env({"KERAS_BACKEND":"jax"}) # sets Keras backend to JAX
    .env({"XLA_PYTHON_CLIENT_MEM_FRACTION":"1.0"})

Second, because tf.data pipelines convert everything to Tensorflow tensors, I had to switch my preprocessing pipelines from using Keras’s ops library (which, because I was using JAX as a backend, expected JAX tensors) to Tensorflow native operations:

ds = ds.map(
    lambda i, j, k, l: 
    (
        preprocessor(i), 
        j, 
        2*k-1, 
    ), 
    num_parallel_calls=tf.data.AUTOTUNE
)

Lastly, I had a few lines of code which assumed Tensorflow tensors (where getting the underlying value required a .numpy() call). As I was now using JAX as a backend, I had to remove the .numpy() calls for the code to work.

Everything else — the rest of the tf.data preprocessing pipeline, the code to train the model, the code to serve it, the previously saved model weights and the code to save & load them — remained the same! Considering that the training time per epoch and the time the model took to evaluate (a measure of inference time) both seemed to improve by 20-40%, this simple switch to JAX seemed well worth it!

Model Architecture Improvements

There were two major improvements I made in the model architecture over the past few months.

First, having run my news reader for the better part of a year now, I now have accumulated enough data where my strategy to simultaneously train on two related tasks (predicting the human rating and predicting the length of an article) no longer required separate inputs. This reduced the memory requirement as well as simplified the data pipeline for training (see architecture diagram below)

Secondly, I was successfully able to train a version of my algorithm which can use dot products natively. This not only allowed me to remove several layers from my previous model architecture (see architecture diagram below), but because the Supabase postgres database I’m using supports pgvector, it means I can even compute ratings for articles through a SQL query:

UPDATE articleuser
SET 
    rating_timestamp = NOW(),
    updated_at = NOW()
FROM 
    articles a, 
    users u
WHERE 
    articleuser.article_id = a.id
    AND articleuser.user_id = u.id
    AND articleuser.ai_rating IS NULL;

The result is much greater simplicity in architecture as well as greater operational flexibility as I can now update ratings from the database directly as well as from serving a deep neural network from my serverless backend.

Model architecture (output from Keras plot_model function)

Making Sources a First-Class Citizen

As I used the news reader, I realized early on that the ability to just have sorted content from one source (i.e. a particular blog or news site) would be valuable to have. To add this, I created and populated a new sources table within the database to track these independently (see database design diagram below) which was linked to the articles table.

Newsreader database design diagram (produced by a Supabase tool)

I then modified my scrapers to insert the identifier for each source alongside each new article, as well as made sure my fetch calls all JOIN‘d and pulled the relevant source information.

With the data infrastructure in place, I added the ability to add a source parameter to the core fetch URLs to enable single (or multiple) source feeds. I then added a quick element at the top of the feed interface (see below) to let a user know when the feed they’re seeing is limited to a given source. I also made all the source links in the feed clickable so that they could take the user to the corresponding single source feed.

<div class="feed-container">
  <div class="controls-container">
    <div class="controls">
      ${source_names && source_names.length > 0 && html`
        <div class="source-info">
          Showing articles from: ${source_names.join(', ')}
        </div>
        &lt;div&gt;
          <a href="/">Return to Main Feed</a>
        </div>
      `}
    </div>
  </div>
</div>

The interface when on a single source feed

Performance Speed-Up

One recurring issue I noticed in my use of the news reader pertained to slow load times. While some of this can be attributed to the “cold start” issue that serverless applications face, much of this was due to how the news reader was fetching pertinent articles from the database. It was deciding at the moment of the fetch request what was most relevant to send over by calculating all the pertinent scores and rank ordering. As the article database got larger, this computation became more complicated.

To address this, I decided to move to a “pre-calculated” ranking system. That way, the system would know what to fetch in advance of a fetch request (and hence return much faster). Couple that with a database index (which effectively “pre-sorts” the results to make retrieval even faster), and I saw visually noticeable improvements in load times.

But with any pre-calculated score scheme, the most important question is how and when re-calculation should happen. Too often and too broadly and you incur unnecessary computing costs. Too infrequently and you risk the scores becoming stale.

The compromise I reached derived itself from the three ways articles are ranked in my system:

The AI’s rating of an article plays the most important role (60%)
How recently the article was published is tied with… (20%)
How similar an article is with the 10 articles a user most recently read (20%

These factors lent themselves to very different natural update cadences:

Newly scraped articles would have their AI ratings and calculated score computed at the time they enter the database
AI ratings for the most recent and the previously highest scoring articles would be re-computed after model training updates
On a daily basis, each article’s score was recomputed (focusing on the change in article recency)
The article similarity for unread articles is re-evaluated after a user reads 10 articles

This required modifying the reader’s existing scraper and post-training processes to update the appropriate scores after scraping runs and model updates. It also meant tracking article reads on the users table (and modifying the /read endpoint to update these scores at the right intervals). Finally, it also meant adding a recurring cleanUp function set to run every 24 hours to perform this update as well as others.

Next Steps

With some of these performance and architecture improvements in place, my priorities are now focused on finding ways to systematically improve the underlying algorithms as well as increase the platform’s usability as a true news tool. To that end some of the top priorities for next steps in my mind include:

Testing new backbone models — The core ranking algorithm relies on Roberta, a model released 5 years ago before large language models were common parlance. Keras Hub makes it incredibly easy to incorporate newer models like Meta’s Llama 2 & 3, OpenAI’s GPT2, Microsoft’s Phi-3, and Google’s Gemma and fine-tune them.
Solving the “all good articles” problem — Because the point of the news reader is to surface content it considers good, users will not readily see lower quality content, nor will they see content the algorithm struggles to rank (i.e. new content very different from what the user has seen before). This makes it difficult to get the full range of data needed to help preserve the algorithm’s usefulness.
Creating topic and author feeds — Given that many people think in terms of topics and authors of interest, expanding what I’ve already done with Sources but with topics and author feeds sounds like a high-value next step

I also endeavor to make more regular updates to the public Github repository (instead of aggregate many updates I had already made into two large ones). This will make the updates more manageable and hopefully help anyone out there who’s interested in building a similar product.

Not your grandma’s geothermal energy

Fri, 15 Nov 2024 00:00:00 GMT

The pursuit of carbon-free energy has largely leaned on intermittent sources of energy — like wind and solar; and sources that require a great deal of initial investment — like hydroelectric (which requires elevated bodies of water and dams) and nuclear (which require you to set up a reactor).

The theoretical beauty of geothermal power is that, if you dig deep enough, virtually everywhere on planet earth is hot enough to melt rock (thanks to the nuclear reactions that heat up the inside of the earth). But, until recently, geothermal has been limited to regions of Earth where well-formed geologic formations can deliver predictable steam without excessive engineering.

But, ironically, it is the fracking boom, which has helped the oil & gas industries get access to new sources of carbon-producing energy, which may help us tap geothermal power in more places. As fracking and oil & gas exploration has led to a revolution in our ability to precisely drill deep underground and push & pull fluids, it also presents the ability for us to tap more geothermal power than ever before. This has led to the rise of enhanced geothermal, the process by which we inject water deep underground to heat, and leverage the steam produced to generate electricity. Studies suggest the resource is particularly rich and accessible in the Southwest of the United States (see map below) and could be an extra tool in our portfolio to green energy consumption.

(Source: Figure 5 from NREL study on enhanced geothermal from Jan 2023)

While there is a great deal of uncertainty around how much this will cost and just what it will take (not to mention the seismic risks that have plagued some fracking efforts), the hunger for more data center capacity and the desire to power this with clean electricity has helped startups like Fervo Energy and Sage Geosystems fund projects to explore.

Reading Excerpt • Nature News

”

“On 17 October, Fervo Energy, a start-up based in Houston, Texas, got a major boost as the US government gave the green light to the expansion of a geothermal plant Fervo is building in Beaver County, Utah. The project could eventually generate as much as 2,000 megawatts — a capacity comparable with that of two large nuclear reactors. Although getting to that point could take a while, the plant already has 400 MW of capacity in the pipeline, and will be ready to provide around-the-clock power to Google’s energy-hungry data centres, and other customers, by 2028. In August, another start-up, Sage Geosystems, announced a partnership with Facebook’s parent company Meta to deliver up to 150 MW of geothermal power to Meta’s data centres by 2027.”

Geothermal power is vying to be a major player in the world\u2019s clean-energy future

David Castelvecchi | Nature News

Who needs humans? Lab of AIs designs valid COVID-binding proteins

Wed, 13 Nov 2024 00:00:00 GMT

A recent preprint from Stanford has demonstrated something remarkable: AI agents working together as a team solving a complex scientific challenge.

While much of the AI discourse focuses on how individual large language models (LLMs) compare to humans, much of human work today is a team effort, and the right question is less “can this LLM do better than a single human on a task” and more “what is the best team-up of AI and human to achieve a goal?” What is fascinating about this paper is that it looks at it from the perspective of “what can a team of AI agents achieve?”

The researchers tackled an ambitious goal: designing improved COVID-binding proteins for potential diagnostic or therapeutic use. Rather than relying on a single AI model to handle everything, the researchers tasked an AI “Principal Investigator” with assembling a virtual research team of AI agents! After some internal deliberation, the AI Principal Investigator selected an AI immunologist, an AI machine learning specialist, and an AI computational biologist. The researchers made sure to add an additional role, one of a “scientific critic” to help ground and challenge the virtual lab team’s thinking.

The team composition and phases of work planned and carried out by the AI principal investigator
(Source: Figure 2 from Swanson et al.)

What makes this approach fascinating is how it mirrors high functioning human organizational structures. The AI team conducted meetings with defined agendas and speaking orders, with a “devil’s advocate” to ensure the ideas were grounded and rigorous.

Example of a virtual lab meeting between the AI agents; note the roles of the Principal Investigator (to set agenda) and Scientific Critic (to challenge the team to ground their work)
(Source: Figure 6 from Swanson et al.)

One tactic that the researchers said helped with boosting creativity that is harder to replicate with humans is running parallel discussions, whereby the AI agents had the same conversation over and over again. In these discussions, the human researchers set the “temperature” of the LLM higher (inviting more variation in output). The AI principal investigator then took the output of all of these conversations and synthesized them into a final answer (this time with the LLM temperature set lower, to reduce the variability and “imaginativeness” of the answer).

The use of parallel meetings to get “creativity” and a diverse set of options
(Source: Supplemental Figure 1 from Swanson et al.)

The results? The AI team successfully designed nanobodies (small antibody-like proteins — this was a choice the team made to pursue nanobodies over more traditional antibodies) that showed improved binding to recent SARS-CoV-2 variants compared to existing versions. While humans provided some guidance, particularly around defining coding tasks, the AI agents handled the bulk of the scientific discussion and iteration.

Experimental validation of some of the designed nanobodies; the relevant comparison is the filled in circles vs the open circles. The higher ELISA assay intensity for the filled in circles shows that the designed nanbodies bind better than their un-mutated original counterparts
(Source: Figure 5C from Swanson et al.)

This work hints at a future where AI teams become powerful tools for human researchers and organizations. Instead of asking “Will AI replace humans?”, we should be asking “How can humans best orchestrate teams of specialized AI agents to solve complex problems?”

The implications extend far beyond scientific research. As businesses grapple with implementing AI, this study suggests that success might lie not in deploying a single, all-powerful AI system, but in thoughtfully combining specialized AI agents with human oversight. It’s a reminder that in both human and artificial intelligence, teamwork often trumps individual brilliance.

I personally am also interested in how different team compositions and working practices might lead to better or worse outcomes — for both AI teams and human teams. Should we have one scientific critic, or should their be specialist critics for each task? How important was the speaking order? What if the group came up with their own agendas? What if there were two principal investigators with different strengths?

The next frontier in AI might not be building bigger models, but building better teams.

Updating OpenMediaVault

Sat, 05 Oct 2024 00:00:00 GMT

(Note: this is part of my ongoing series on cheaply selfhosting)

I’ve been using OpenMediaVault 6 on a cheap mini-PC as a home server for over a year. Earlier this year, OpenMediaVault 7 was announced which upgrades the underlying Linux to Debian 12 Bookworm and made a number of other security, compatibility, and user interface improvements.

Not wanting to start over fresh, I decided to take advantage of OpenMediaVault’s built-in command line tool to handle upgrades and, if like me, you are looking for a quick and clean way of upgrading from OpenMediaVault 6 to OpenMediaVault 7, look no further:

SSH into your system / connect directly with a keyboard and monitor. While normally I would recommend WeTTY (accessible via [Services > WeTTY] from the web console interface) to handle any command line activity on your server, because WeTTY relies on the server to be running and updating the operating system necessitates shutting the server down, you’ll need to either plug in a keyboard and monitor or use SSH.
Run sudo omv-upgrade in the command-line. This will start a long process of downloading and installing necessary files to complete the operating system update. From time to time you’ll be asked to accept / approve a set of changes via keyboard. If you’re on the online administrative panel, you’ll be booted off as the server shuts down.
Restart the server. Once everything is complete, you’ll need to restart the server to make sure everything “takes”. This can be done by running reboot in the command line or by manually turning off and on the server.

Assuming everything went smoothly, after the server completes its reboot (which will take a little bit of extra time after an operating system upgrade), upon logging into the administrative console as you had done before, you’ll be greeted by the new OMV 7 login screen. Congratulations!

A Digital Twin of the Whole World in the Cloud

Sun, 22 Sep 2024 00:00:00 GMT

As a kid, I remember playing Microsoft Flight Simulator 5.0 — while I can’t say I really understood all the nuances of the several hundred page manual (which explained how ailerons and rudders and elevators worked), I remember being blown away with the idea that I could fly anywhere on the planet and see something reasonably representative there.

Flash forward a few decades and Microsoft Flight Simulator 2024 can safely be said to be one of the most detailed “digital twins” of the whole planet ever built. In addition to detailed photographic mapping of many locations (I would imagine a combination of aerial surveillance and satellite imagery) and an accurate real world inventory of every helipad (including offshore oil rigs!) and glider airport, they also simulate flocks of animals, plane wear and tear, how snow vs mud vs grass behave when you land on it, wake turbulence, and more! And, just as impressive, it’s being streamed from the cloud to your PC/console when you play!

Who said the metaverse is dead?

Reading Excerpt • IGN

”

“People are dressed in clothes and styles matching their countries of origin. They speak in the language of their home countries. Flying from the US to Finland on a commercial plane? Walk through the cabin: you’ll hear both English and Finnish being spoken by the passengers. Neumann, who has a supervising producer credit on 2013’s Zoo Tycoon and a degree in biology, has a soft-spot for animals and wants to make sure they’re also being more realistically simulated in MSFS 2024. “I really didn’t like the implementation of the animal flights in 2020,” he admitted. “It really bothered me, it was like, ‘Hey, find the elephants!’ and there’s a stick in the UI and there’s three sad-looking elephants. “There’s an open source database that has all wild species, extinct and living, and it has distribution maps with density over time,” Neumann continued. Asobo is drawing from that database to make sure animals are exactly where they’re supposed to be, and that they have the correct population densities. In different locations throughout the year, “you will find different stuff, but also they’re migrating,” so where you spot a herd of wildebeests or caribou one day might not be the same place you find them the next.”

Microsoft Flight Simulator 2024: The First Preview

Seth G. Macy | IGN

The Startup Battlefield: Lessons from History’s Greatest Military Leaders

Wed, 18 Sep 2024 00:00:00 GMT

It is hard to find good analogies for running a startup that founders can learn from. Some of the typical comparisons — playing competitive sports & games, working on large projects, running large organizations — all fall short of capturing the feeling that the odds are stacked against you that founders have to grapple with.

But the annals of military history offer a surprisingly good analogy to the startup grind. Consider the campaigns of some of history’s greatest military leaders — like Alexander the Great and Julius Caesar — who successfully waged offensive campaigns against numerically superior opponents in hostile territory. These campaigns have many of the same hallmarks as startups:

Bad odds: Just as these commanders faced superior enemy forces in hostile territory, startups compete against incumbents with vastly more resources in markets that favor them.
Undefined rules: Unlike games with clear rules and a limited set of moves, military commanders and startup operators have broad flexibility of action and must be prepared for all types of competitive responses.
Great uncertainty: Not knowing how the enemy will act is very similar to not knowing how a market will respond to a new offering.

As a casual military history enthusiast and a startup operator & investor, I’ve found striking parallels in how history’s most successful commanders overcame seemingly insurmountable odds with how the best startup founders operate, and think that’s more than a simple coincidence.

In this post, I’ll explore the strategies and campaigns of 9 military commanders (see below) who won battle after battle against numerically superior opponents across a wide range of battlefields. By examining their approach to leadership and strategy, I found 5 valuable lessons that startup founders can hopefully apply to their own ventures.

Leader	Represented	Notable Victories	Legacy
Alexander the Great	Macedon (336-323 BCE)	Tyre, Issus, Gaugamela, Persian Gate, Hydapses	Conquered the Persian Empire before the age of 32; spread Hellenistic culture across Eurasia and widely viewed in the West as antiquity’s greatest conqueror
Hannibal Barca	Carthage (221-202 BCE)	Ticinus, Trebia, Trasimene, Cannae	Brought Rome the closest to its defeat until its fall in 5th century CE; he operated freely within Italy for over a decade
Han Xin (韓信)	Han Dynasty (漢朝) (206-202 BCE)	Jingxing (井陘), Wei River (濰水), Anyi (安邑)	Despite being a commoner, his victories led to the creation of the Han Dynasty (漢朝) and his being remembered as one of “the Three Heroes of the Han Dynasty” (漢初三傑)
Gaius Julius Caesar	Rome (59-45 BCE)	Alesia, Pharsalus	Established Rome’s dominance in Gaul (France); became undisputed leader of Rome, effectively ending the Roman Republic, and his name has since become synonymous with “emperor” in the West
Subutai	Mongol Empire (1211-1248)	Khunan, Kalka River, Sanfengshan ( 三峰山), Mohi	Despite being a commoner, became one of the most successful military commanders in the Mongol Empire. Successfully won battles in more theaters than any other commander (China, Central Asia, and Eastern Europe)
Timur	Timurid Empire (1370-1405)	Kondurcha River, Terek River, Dehli, Ankara	Created Central Asian empire with dominion over Turkey, Persia, Northern India, Eastern Europe, and Central Asia. His successors would eventually create the Mughal Empire in India which continued until the 1850s
John Churchill, Duke of Marlborough	Britain (1670-1712)	Blenheim, Ramillies	Considered one of the greatest British commanders in history; Paved the way for Britain to overtake France as the pre-eminent military and economic power in Europe
Frederick the Great	Prussia (1740-1779)	Hohenfriedberg, Rossbach, Leuthen	Established Prussia as the pre-eminent Central European power after defeating nearly every major European power in battle; A cultural icon for the creation of Germany
Napoleon Bonaparte	France (1785-1815)	Rivoli, Tarvis, Ulm, Austerlitz, Jena-Auerstedt, Friedland, Dresden	Established a French empire with dominion over most of continental Europe; the Napoleonic code now serves as basis for legal systems around the world and the word Napoleon synonymous with military genius and ambition

Before I dive in, three important call-outs to remember:

Running a startup is not actually warfare — there are limitations to this analogy. Startups are not (and should not be) life-or-death. Startup employees are not bound by military discipline (or the threat of imprisonment if they are derelict). The concept of battlefield deception, which is at the heart of many of the tactics of the greatest commanders, also doesn’t translate well. Treating your employees / co-founders as one would a soldier or condoning violent and overly aggressive tactics would be both an ethical failure and a misread of this analogy.
Drawing lessons from these historical campaigns does not mean condoning the underlying sociopolitical causes of these conflicts, nor the terrible human and economic toll these battles led to. Frankly, many of these commanders were absolutist dictators with questionable motivations and sadistic streaks. This post’s focus is purely on getting applicable insights on strategy and leadership from leaders who were able to win despite difficult odds.
This is not intended to be an exhaustive list of every great military commander in history. Rather, it represents the intersection of offensive military prowess and my familiarity with the historical context. Just because I did not mention a particular commander has no bearing on their actual greatness.

With those in mind, let’s explore how the wisdom of historical military leaders can inform the modern startup journey. In the post, I’ll unpack five key principles (see below) drawn from the campaigns of history’s most successful military commanders, and show how they apply to the challenges ambitious founders face today.


1. Get in the trenches with your team
2. Achieve and maintain tactical superiority
3. Move fast and stay on offense
4. Unconventional teams win
5. Pick bold, decisive battles

Principle 1: Get in the trenches with your team

One common thread unites the greatest military commanders: their willingness to share in the hardships of their soldiers. This exercise of leadership by example, of getting “in the trenches” with one’s team, is as crucial in the startup world as it was on historical battlefields.

Every commander on our list was renowned for marching and fighting alongside their troops. This wasn’t mere pageantry; it was a fundamental aspect of their leadership style that yielded tangible benefits:

Inspiration: Seeing their leader work shoulder-to-shoulder with them motivated soldiers to push beyond their regular limits.
Trust: By sharing in their soldiers’ hardships, commanders demonstrated that they valued their troops and understood their needs.
Insight: Direct involvement gave leaders firsthand knowledge of conditions on the ground, informing better strategic decisions.

Perhaps no figure exemplified this better than Alexander the Great. Famous for being one of the first soldiers to jump into battle, Alexander was wounded seriously multiple times. This shared experience created a deep bond with his soldiers, culminating in his legendary speech at Opis where he was able to quell a mutiny of his soldiers, tired after years of campaigns, with a speech reminding them of their shared experiences:

Alexander the Great from Alexandria, Egypt (3rd Century BCE); Image Credit: Wikimedia

The wealth of the Lydians, the treasures of the Persians, and the riches of the Indians are yours; and so is the External Sea. You are viceroys, you are generals, you are captains. What then have I reserved to myself after all these labors, except this purple robe and this diadem? I have appropriated nothing myself, nor can any one point out my treasures, except these possessions of yours or the things which I am guarding on your behalf. Individually, however, I have no motive to guard them, since I feed on the same fare as you do, and I take only the same amount of sleep.

Nay, I do not think that my fare is as good as that of those among you who live luxuriously; and I know that I often sit up at night to watch for you, that you may be able to sleep.

But some one may say, that while you endured toil and fatigue, I have acquired these things as your leader without myself sharing the toil and fatigue. But who is there of you who knows that he has endured greater toil for me than I have for him? Come now, whoever of you has wounds, let him strip and show them, and I will show mine in turn; for there is no part of my body, in front at any rate, remaining free from wounds; nor is there any kind of weapon used either for close combat or for hurling at the enemy, the traces of which I do not bear on my person.

For I have been wounded with the sword in close fight, I have been shot with arrows, and I have been struck with missiles projected from engines of war; and though oftentimes I have been hit with stones and bolts of wood for the sake of your lives, your glory, and your wealth, I am still leading you as conquerors over all the land and sea, all rivers, mountains, and plains. I have celebrated your weddings with my own, and the children of many of you will be akin to my children.

Alexander the Great (as told by Arrian)

This was not unique to Alexander. Julius Caesar famously slept in chariots and marched alongside his soldiers. Napoleon was called “le petit caporal” by his troops after he was found sighting the artillery himself, a task that put him within range of enemy fire and was usually delegated to junior officers.

Frederick the Great also famously mingled with his soldiers while on tour, taking kindly to the nickname from his men, “Old Fritz”. Frederick understood the importance of this as he once wrote to his nephew:

“You cannot, under any pretext whatever, dispense with your presence at the head of your troops, because two thirds of your soldiers could not be inspired by any other influence except your presence.”

Frederick the Great

“Old Fritz” after the Battle of Hochkirch
Image credit: WikiMedia Commons

For Startups

For founders, the lesson is clear: show up when & where your team is and roll up your sleeves so they can see you work beside them. It’s not just that startups tend to need “all hands on deck”, but being in the trenches also provides “on the ground” context that is valuable and help create the morale needed to succeed.

Elon Musk, for example, famously spent time on the Tesla factory floor — even sleeping on it — while the company worked through issues with its Model 3 production, noting in an interview:

“I am personally on that line, in that machine, trying to solve problems personally where I can,” Musk said at the time. “We are working seven days a week to do it. And I have personally been here on zone 2 module line at 2:00 a.m. on a Sunday morning, helping diagnose robot calibration issues. So I’m doing everything I can.”

Principle 2: Achieve and maintain tactical superiority

To win battles against superior numbers requires a commander to have a strong tactical edge over their opponents. This can be in the form of a technological advantage (i.e. a weapons technology) or an organizational one (i.e. superior training or formations), but these successful commanders always made sure their soldiers could “punch above their weight”.

Alexander the Great, for example, leveraged the Macedonian Phalanx, a modification of the “classical Greek phalanx” used by the Greek city states of the era, that his father Philip II helped create.

Image Credit: RedTony via WikiMedia Commons

The formation relied on “blocks” of heavy infantry equipped with six-meter (!!) long spears called sarissa which could rearrange themselves (to accommodate different formation widths and depths) and “pin” enemy formations down while the heavy cavalry would flank or exploit gaps in the enemy lines. This formation made Alexander’s army highly effective against every military force — Greeks, Persians, and Indians — it encountered.

Macedonian Phalanx with sarissa; Image Credit: Wikimedia Commons

A few centuries later, the brilliant Chinese commander Han Xin (韓信) leaned heavily on the value of military engineering. Han Xin (韓信)’s soldiers would rapidly repair & construct roads to facilitate his army’s movement or, at times, to deceive his enemies about which path he planned to take. His greatest military engineering accomplishment was at the Battle of Wei River (濰水) in 204 BCE. Han Xin (韓信) attacked the larger forces of the State of Qi (齊) and State of Chu (楚) and immediately retreated across the river, luring them to cross. What his rivals had not realized in their pursuit was that the water level of the Wei River was oddly low. Han Xin (韓信) had, prior to the attack, instructed his soldiers to construct a dam upstream to lower the water level. Once a sizable fraction of the enemy’s forces were mid-stream, Han Xin (韓信) ordered the dam released. The rush of water drowned a sizable portion of the enemy’s forces and divided the Chu (楚) / Qi (齊) forces letting Han Xin (韓信)’s smaller army defeat and scatter them.

A century and a half later, Roman statesman and military commander Gaius Julius Caesar also famously advocated military engineering capability in his wars with the Germanic tribes in Gaul. He became the first Roman commander to cross the Rhine (twice!) by building bridges to make the point to the Germanic tribes that he could invade them whenever he wanted. At the Battle of Alesia in 52 BCE, after trading battles with the skilled Gallic commander Vercingetorix who had united the tribes in opposition to Rome, Caesar besieged Vercingetorix’s fortified settlement of Alesia while simultaneously holding off Gallic reinforcements. Caesar did this by building 25 miles of fortifications surrounding Alesia in a month, all while outnumbered and under constant harassment from both sides by the Gallic forces! Caesar’s success forced Vercingetorix to surrender, bringing an end to organized resistance to Roman rule in Gaul for centuries.

Vercingetorix Throws Down his Arms at the Feet of Julius Caesar by Lionel Royer; Image Credit: Wikimedia

The Mongol commander Subutai similarly made great use of Mongol innovations to overcome defenders from across Eurasia. The lightweight Mongol composite bow gave Mongol horse archers a devastating combination of long range (supposedly 150-200 meters!) and speed (because they were light enough to be fired while on horseback). The Mongol horses themselves were another “biotechnological” advantage in that they required less water and food which let the Mongols wage longer campaigns without worrying about logistics.

Mongol horse archers, Image credit: Wikimedia Commons

In the 18th century, Frederick the Great transformed warfare on the European continent with a series of innovations. First, he drilled his soldiers stressing things like firing speed. It is said that lines of Prussian riflemen could fire over twice as fast as other European armies they faced, making them exceedingly lethal in combat.

Frederick’s Leibgarde Batallion in action; Image credit: Military Heritage

Frederick was also famous for a battle formation: the oblique order. Instead of attacking an opponent head on, the oblique order involves confronting the enemy line at an angle with soldiers massed towards one end of the formation. If one’s soldiers are well-trained and disciplined, then even with a smaller force in aggregate, the massed wing can overwhelm the opponent in one area and then flank or surround the rest. Frederick famously boasted that the oblique order could allow a skilled force to win over an opposing one three times its size.

Finally, Frederick is credited with popularizing horse artillery, the use of horse-drawn light artillery guns, in European warfare. With horse artillery units, Frederick was able to increase the adaptability of his forces and their ability to break through even numerically superior massed infantry by concentrating artillery fire where it was needed.

Horse-drawn artillery unit; Image credit: Wikimedia Commons

A few decades later, Napoleon Bonaparte became the undisputed master of much of continental Europe by mastering army-level logistics and organization. While a brilliant tactician and artillery commander, what set Napoleon’s military apart was its embrace of the “corps system”, which subdivided his forces into smaller, self-contained corps that were capable of independent operations. This allowed Napoleon the ability to pursue grander goals, knowing that he could focus his attention on the most important fronts of battle, while the other corps could independently pin an enemy down or pursue a different objective in parallel.

Napoleon triumphantly entering Berlin by Charles Meynier; Image Credit: Wikimedia Commons

Additionally, Napoleon invested heavily in overhauling military logistics, using a combination of forward supply depots and teaching his forces to forage for food and supplies in enemy territory (and, just as importantly, how to estimate what foraging can do to help determine the necessary supplies to take). This investment led to the invention of modern canning technology, first used to support the marches of the French Grande Armée. The result was Napoleon could field larger armies over longer campaigns all while keeping his soldiers relatively well-fed.

For Startups

Founders need to make sure they have a strong tactical advantage that fits their market(s). As evidenced above, it does not need to be something as grand as an unassailable advantage, but it needs to be a reliable winner and something you continuously invest in if you plan on competing with well-resourced incumbents in challenging markets.

The successful payments company Stripe started out by making sure they would always win on developer ease of use, even going so far as to charge more than their competition during their Beta to make sure that their developer customers were valuing them for their ease of use. Stripe’s advantage here, and continuous investment in maintaining that advantage, ultimately let it win any customer that needed a developer payment integration, even against massive financial institutions. This advantage laid the groundwork for Stripe’s meteoric growth and expansion into adjacent categories from its humble beginnings.

Principle 3: Move fast and stay on offense

In both military campaigns and startups, speed and a focus on offense plays an outsized role in victory, because the ability to move quickly creates opportunities and increases resiliency to mistakes.

Few understood this principle as well as the Mongol commander Subutai who frequently took advantage of the greater speed and discipline of the Mongol cavalry to create opportunities to win.

In the Battle of the Kalka River (1223), Subutai took what initially appeared to be a Mongol defeat — when the Kievan Rus and their Cuman allies successfully entrapped the Mongol forces in the area — and turned it into a victory. The Mongols began a 9 day feigned retreat (many historians believe this was a real retreat that Subutai turned into a feigned one once he realized the situation), constantly tempting the enemy by staying just out of reach into overextending themselves in pursuit.

After 9 days, Subutai’s forces took advantage of their greater speed to lay a trap. Once the Mongols crossed the river they reformed their lines to lie in ambush. As soon as the Rus forces crossed the Kalka River, they found themselves surrounded and confronted with a cavalry charge they were completely unprepared for. After all, they had been pursuing what they thought was a fleeing enemy! Their backs against the river, the Rus forces (including several major princes) were annihilated.

Battle of Kalka River; Image Credit: Wikimedia Commons

Subutai took advantage of the Mongol speed advantage in a number of his campaigns, coordinating fast-moving Mongol divisions across multiple objectives. In its destruction of the Central Asian Khwarazmian empire, the Mongols, under the command of Subutai and Mongol ruler Genghis Khan, overwhelmed the defenders with coordinated maneuvers. While much of the Mongol forces attacked from the East, where the Khwarazmian forces massed, Subutai used the legendary Mongol speed to go around the Khwarazmian lines altogether, ending up at Bukhara, 100 miles to the West of the Khwarazmian defensive position! In a matter of months, the empire was destroyed and its rulers chased out, never to return.

Map of the Mongol force movements in the Mongol invasion of Khwarazmian Empire; Image Credit: Paul K. Davis, Masters of the Battlefield

A few hundred years later, the Englishman John Churchill, the Duke of Marlborough also proved the value of speed in 1704 when he boldly marched an army of 21,000 Dutch and English troops on a 250-mile march across Europe in just five weeks to place themselves between French and Bavarian forces and their target of Vienna. Had Vienna been attacked, it would have forced England’s ally the Holy Roman Empire out of the conflict, giving France the victory in the War of the Spanish Succession. This march was made all the more challenging as Marlborough had to find a way to feed and equip his army along this march without unnecessarily burdening the neutral and friendly territories they were marching through.

Marlborough’s “march to the Danube”; Image Credit: Rebel Redcoat

Marlborough’s maneuver threw the Bavarian and French forces off-balance. What originally was supposed to be an “easy” French victory culminated in a crushing defeat for the French at Blenheim which turned the momentum of the war. This victory solidified Marlborough’s reputation and even resulted in the British government agreeing to build a lavish palace (called Blenheim Palace in honor of the battle) as a reward to Marlborough.

Marlborough proved the importance of speed again at the Battle of Oudenarde. In 1708, French forces captured Ghent and Bruges (in modern day Belgium), threatening the alliance’s ability to maintain contact with Britain. Recognizing this, Marlborough force-marched his army to the city of Oudenarde, marching 30 miles in about as many hours. The French, confident from their recent victories and suffering from an internal leadership squabble, misjudged the situation, allowing Marlborough’s forces to build five pontoon bridges to move his 80,000 soldiers across the nearby river.

When the French commander received news that the allies were already at Oudenarde building bridges, he said, “If they are there, then the devil must have carried them. Such marching is impossible!“

Marlborough’s forces, not yet at full strength, engaged the French, buying sufficient time for his forces to cross and form up. Once in formation, they counterattacked and collapsed one wing of the French line, saving the Allied position in the Netherlands, and resulting in a bad defeat for French forces.

The Battle of Oudenarde, showing the position of the bridges the Allied forces needed to cross to get into position; Image Credit: WikiMedia Commons

For Startups

The pivotal role speed played in achieving victory for Subutai and the Duke of Marlborough apply in the startup domain as well. The ability to make fast decisions, to quickly shift focus to rapidly adapt to a new market context creates opportunities that slower moving incumbents (and military commanders!) cannot seize. Speed also gifts resiliency against mistakes and weak positions, in much the same way that speed let the Mongols and the Anglo-Prussian-Dutch alliance overcome their initial missteps at Kalka River and Oudenarde. Founders would be wise to remember to embrace speed of action in all they do.

Facebook and it’s (now in)famous “move fast, break things” motto is one classic example of how a company can internalize speed as a culture. It leveraged that to ship products and features which has kept it a leader in social and AI even in the face of constant competition and threats from well-funded companies like Google, Snapchat, and Bytedance.

Principle 4: Unconventional teams win

Another unifying hallmark of the great commanders is that they made unconventional choices with regards to their army composition. Relative to their peers, these commanders tended to build armies that were more diverse in class and nationality. While this required exceptional communication and inspiration skills, it gave the commanders significant advantages:

Ability to recruit in challenging conditions: For many of the commanders, the unconventional team structure was a necessity to build up the forces they needed given logistical / resource constraints while operating in enemy territory.
Operational flexibility from new tactics: Bringing on personnel from different backgrounds let commanders incorporate additional tactics and strategies, creating a more effective and flexible fighting force.

The Carthaginian general Hannibal Barca for example famously fielded a multi-nationality army consisting of Carthaginians, Libyans, Iberians, Numidians, Balearic soldiers, Gauls, and Italians. This allowed Hannibal to raise an army in hostile territory — after all, waging war in the heart of Italy against Rome made it difficult to get reinforcements from Carthage.

Illustration of troop types employed in the Second Punic War by Carthage/Hannibal Barca; Image Credit: Travis’s Ancient History

But, it also gave Hannibal’s army flexibility in tactics. Balearic slingers provided superior long range attack to the best Roman-used bows of the time. Numidian light cavalry provided Hannibal with fast reconnaissance and a quick way to flank and outmaneuver Roman forces. Gallic and Iberian soldiers provided shock infantry and cavalry. Each of these groups of soldiers added their own distinctive capabilities to Hannibal’s armies and great victories over Rome.

The Central Asian conqueror Timur similarly fielded a diverse army which included Mongols, Turks, Persians, Indians, Arabs, and others. This allowed Timur to field larger armies for his campaigns by recruiting from the countries he forced into submission. Like with Hannibal, it also gave Timur’s army access to a diverse set of tactics: war elephants (from India), infantry and siege technology from the Persians, gunpowder from the Ottomans, and more. This combination of operational flexibility and ability to field large armies let Timur build an empire which defeated every major power in Central Asia and the Middle East.

The Defeat by Timur of the Sultan of Dehli (from the Imperial Library of Emperor Akbar);
Image credit: Wikimedia

It should not be a surprise that some of the great commanders were drawn towards assembling unconventional teams as several of them were ultimately “commoners”. Subutai (a son of a blacksmith who Genghis Khan took interest in), Timur (a common thief), and Han Xin (韓信, who famously had to beg for food in his childhood) all came from relatively humble origins. Napoleon, famous for declaring the military “la carrier est ouvérte aux talents” (“the career open to the talents”) and creating the first modern order of merit Légion d’honneur (open to all, regardless of social class), was similarly motivated by the difficulties he faced in securing promotion early in his career due to his not being from the French nobility.

But, by embracing more of a meritocracy, Napoleon was ultimately able to field some of the largest European armies in existence as he waged war successfully against every other major European power (at once).

First Légion d’Honneur Investiture by Jean-Baptiste Debret;
Image Credit: Wikimedia

For Startups

Hiring is one of the key tasks for startup founders. While hiring the people that larger, better-resourced companies want to can be helpful for a startup, it’s important to always remember that transformative victories require unconventional approaches. Leaning on unconventional hires may help you get out of a salary bidding war with those deeper-pocketed competitors. Choosing unconventional hires may also add different skills and perspectives to the team.

In pursuing this strategy, it’s also vital to excel at communication & organization as well as fostering a shared sense of purpose. All teams require strong leadership to be effective but this is especially true with an unconventional team composition facing uphill odds.

The enterprise API company Zapier is one example of taking an unconventional approach to team construction by having been 100% remote from inception (pre-COVID even). This let the company assemble a team without being confined by location and eliminate the need to spend on unnecessary facilities. They’ve had to invest in norms around documentation and communication to make this work, and, while it’d be too far of a leap to argue all startups should go 100% remote, for Zapier’s market and team culture, it’s worked.

Principle 5: Pick bold, decisive battles

When in a challenging environment with limited resources, it’s important to prioritize decisive moves — actions which can result in a huge payoff — even if risky over safer, less impactful ones. This is as true for startups, which have limited runway and need to make a big splash in order to fundraise, as for military commanders who need more than just battlefield wins but strategic victories.

Few understood this as well as the Carthaginian general Hannibal Barca who, in waging the Second Punic War against Rome, crossed the Alps from Spain with his army in 218 BCE (at the age of 29!). Memorialized in many works of art (see below for one by Francisco Goya), this was a dangerous move (one that resulted in the loss of many men and almost his entire troop of war elephants) and was widely considered to be impossible.

The Victorious Hannibal Seeing Italy from the Alps for the First Time by Francisco Goya in Museo del Prado; Image Credit: Wikimedia

While history (rightly) remembers Hannibal’s boldness, it’s important to remember that Hannibal’s move was highly calculated. He realized that the Gauls in Northern Italy, who had recently been subjugated by the Romans, were likely to welcome a Roman rival. Through his spies, he also knew that Rome was planning an invasion of Carthage in North Africa. He knew he had little chance to bypass the Roman navy or Roman defensive placements if he invaded in another way.

And Hannibal’s bet paid off! Having caught the Romans entirely by surprise, they cancelled their planned invasion of Africa, and Hannibal lined up many Gallic allies to his cause. Within two years of his entry into Italy, Hannibal trounced the Roman armies sent to battle him at the River Ticinus, at the River Trebia, and at Lake Trasimene. Shocked by their losses, the Romans elected two consuls with the mandate to battle Hannibal and stop him once and for all.

Knowing this, Hannibal seized a supply depot at the town of Cannae, presenting a tempting target to the Roman consuls to prove themselves. They (foolishly) took the bait. Despite fielding over 80,000 soldiers against Hannibal’s 50,000, Hannibal successfully executed a legendary double-envelopment maneuver (see below) and slaughtered almost the entire Roman force that met him in battle.

Hannibal’s double envelopment of Roman forces at Cannae;
Image Credit: Wikimedia

To put this into perspective, in the 2 years after Hannibal crossed the Alps, Hannibal’s army killed 20% of all male Romans over the age of 17 (including at least 80 Roman Senators and one previous consul). Cannae is today considered one of the greatest examples of military tactical brilliance, and, as historian Will Durant wrote, “a supreme example of generalship, never bettered in history”.

Cannae was a great example of Hannibal’s ability to pick a decisive battle with favorable odds. Hannibal knew that his only chance was to encourage the city-states of Italy to side with him. He knew the Romans had just elected consuls itching for a fight. He chose the field of battle by seizing a vital supply depot at Cannae. Considering the Carthaginians had started and pulled back from several skirmishes with the Romans in the days leading up to the battle, it’s clear Hannibal also chose when to fight, knowing full well the Romans outnumbered him. After Cannae, many Italian city-states and the kingdom of Macedon sided with Carthage. That Carthage ultimately lost the Second Punic War is a testament more to Rome’s indomitable spirit and the sheer odds Hannibal faced than any indication of Hannibal’s skills.

In the Far East, about a decade later, the brilliant Chinese military commander Han Xin (韓信) was laying the groundwork for the creation of the Han Dynasty (漢朝) in a China-wide civil war known as the the Chu-Han contention between the State of Chu (楚) and the State of Han (漢) led by Liu Bang (劉邦, who would become the founding emperor Gaozu 高祖 of the Han Dynasty 漢朝).

Under the leadership of Han Xin (韓信), the State of Han (漢) won many victories over their neighbors. Overconfident from those victories, his king Liu Bang (劉邦) led a Han (漢) coalition to a catastrophic defeat when he briefly captured but then lost the Chu (楚) capital of Pengcheng (彭城) in 205 BCE. Chu forces (楚) were even able to capture the king’s father and wife as hostages, and several Han (漢) coalition states switched their loyalty to the Chu (楚).

Map of the 18 states that existed at the start of the Chu-Han Contention, the two sides being the Han (in light purple on the Southwest) and the Chu (in green on the East); Image Credit: Wikimedia

To fix his king’s blunder, Han Xin (韓信) tasked the main Han (漢) army with setting up fortified positions in the Central Plain, drawing Chu (楚) forces there. Han Xin (韓信) would himself take a smaller force of less experienced soldiers to attack rival states in the North to rebuild the Han (漢) military position.

After successfully subjugating the State of Wei (魏), Han Xin (韓信)’s forces moved to attack the State of Zhao (趙, also called Dai 代) through the Jingxing Pass (井陘關) in late 205 BCE. The Zhao (趙) forces, which outnumbered Han Xin (韓信)’s, encamped on the plain just outside the pass to meet them.

Sensing an opportunity to deal a decisive blow to the overconfident Zhao (趙), Han Xin (韓信) ordered a cavalry unit to sneak into the mountains behind the Zhao (趙) camp and to remain hidden until battle started. He then ordered half of his remaining army to position themselves in full view of the Zhao (趙) forces with their backs to the Tao River (洮水), something Sun Tzu’s Art of War (孫子兵法) explicitly advises against (due to the inability to retreat). This “error” likely reinforced the Zhao (趙) commander’s overconfidence, as he made no move to pre-emptively flank or deny the Han (漢) forces their encampment.

Han Xin (韓信) then deployed his full army which lured the Zhao (趙) forces out of their camp to counterattack. Because the Tao River (洮水) cut off all avenues of escape, the outnumbered Han (漢) forces had no choice but to dig in and fight for their lives, just barely holding the Zhao (趙) forces at bay. By luring the enemy out for what appeared to be “an easy victory”, Han Xin (韓信) created an opportunity for his hidden cavalry unit to capture the enemy Zhao (趙) camp, replacing their banners with those of the Han (漢). The Zhao (趙) army saw this when they regrouped, which resulted in widespread panic as the Zhao (趙) army concluded they must be surrounded by a superior force. The opposition’s morale in shambles, Han Xin (韓信) ordered a counter-attack and the Zhao (趙) army crumbled, resulting in the deaths of the Zhao (趙) commander and king!

Han Xin (韓信) bet his entire outnumbered command on a deception tactic based on little more than an understanding of his army’s and the enemy’s psychology. He won a decisive victory which helped reverse the tide of the war. The State of Zhao (趙) fell, and the State of Jiujiang (九江) and the State of Yan (燕) switched allegiances to the Han (漢). This battle even inspired a Chinese expression “fighting a battle with one’s back facing a river” (背水一戰) to describe fighting for survival in a “last stand”.

Caesar crosses the Rubicon by Bartolomeo Pinelli; Image Credit: Wikimedia

Roughly a century later, on the other side of the world, the Roman statesman and military commander Julius Caesar made a career of turning bold, decisive bets into personal glory. After Caesar conquered Gaul, Caesar’s political rivals led by Gnaeus Pompeius Magnus (Pompey the Great), a famed military commander, demanded Caesar return to Rome and give up his command. Caesar refused and crossed the Rubicon, a river marking the boundary of Italy, in January 49 BCE starting a Roman Civil War and coining at least two famous expressions (including alea iacta est – “the die is cast”) for “the point of no return”.

This bold move came as a complete shock to the Roman elite. Pompey and his supporters fled Rome. Taking advantage of this, Caesar captured Italy without much bloodshed. Caesar then pursued Pompey to Macedon, seeking a decisive land battle which Pompey, wisely, given his broad network of allies and command of the Roman navy, refused to give him. Instead, Caesar tried and failed to besiege Pompey at Dyrrhachium which forced him into retreat in Greece.

Pompey’s supporters, however, lacked Pompey’s patience (and judgement). Overconfident from their naval strength, numerical advantage, and Caesar’s failure at Dyrrhachium, they pressured Pompey into a battle with Caesar who was elated at the opportunity. In the summer of 48 BCE, the two sides met at the Battle of Pharsalus.

The initial battle formations at the Battle of Pharsalus; Image Credit: Wikimedia

Always cautious, Pompey took up a position on a mountain and oriented his forces such that his larger cavalry wing would have ability to overpower Caesar’s cavalry and then flank Caesar’s forces while his numerically superior infantry would be arranged deeper to smash through or at least hold back Caesar’s lines.

Caesar made a bold tactical choice when he saw Pompey’s formation. He thinned his (already outnumbered) lines to create a 4th reserve line of veterans which he positioned behind his cavalry at an angle (see battle formation above).

Caesar initiated the battle and attacked with two of his infantry lines. As Caesar expected, Pompey ordered a cavalry charge which soon forced back Caesar’s outnumbered cavalry. But Pompey’s cavalry then encountered Caesar’s 4th reserve line which had been instructed to use their javelins to stab at the faces of Pompey’s cavalry like bayonets. Pompey’s cavalry, while larger in size, was made up of relatively inexperienced soldiers and the shock of the attack caused them to panic. This let Caesar’s cavalry regroup and, with the 4th reserve line, swung around Pompey’s army completing an expert flanking maneuver. Pompey’s army, now surrounded, collapsed once Caesar sent his final reserve line into battle.

Caesar’s boldness and speed of action let him take advantage of a lapse in Pompey’s judgement. Seeing a rare opportunity to win a decisive battle, Caesar was even willing to risk a disadvantage in infantry, cavalry, and position (Pompey’s army had the high ground and had forced Caesar to march to him). But this strategic and tactical gamble (thinning his lines to counter Pompey’s cavalry charge) paid off as Pharsalus shattered the myth of Pompey’s inevitability. Afterwards, Pompey’s remaining allies fled or defected to Caesar, and Pompey himself fled to Egypt where he was assassinated (by a government wishing to win favor with Caesar). And, all of this — from Gaul to crossing the Rubicon to the Civil War — paved the way for Caesar to become the undisputed master of Rome.

For Startups

Founders need to take bold, oftentimes uncomfortable bets that have large payoffs. While a large company can take its time winning a war of attrition, startups need to score decisive wins quickly in order to attract talent, win deals, and shift markets towards them. Only taking the “safe and rational” path is a failure to recognize the opportunity cost when operating with limited resources.

In other words, founders need to find their own Alps / Rubicons to cross.

In the startup world, few moves are as bold (while also uncomfortable and risky) as big pivots. But, there are examples of incredible successes like Slack that were able to make this work. In Slack’s case, the game they originally developed ended up a flop, but CEO & founder Stewart Butterfield felt the messaging product they had built to support the game development had potential. Leaning on that insight, over the skepticism of much of his team and some high profile investors, Butterfield made a bet-the-company move similar to Han Xin (韓信) digging in with no retreat which created a seminal product in the enterprise software space.

Summary

I hope I’ve been able to show that history’s greatest military commanders can offer valuable lessons on leadership and strategy for startup founders.

The five principles derived from studying some of the commanders’ campaigns – the importance of getting in the trenches, achieving tactical superiority, moving fast, building unconventional teams, and picking bold, decisive battles – played a key role in the commanders’ success and generalize well to startup execution.

After all, what is a more successful founder than one who can recruit teams despite resource constraints (unconventional teams), inspire them (by getting in the trenches alongside them), and move with speed & urgency (move fast) to take a competitive edge (achieve tactical superiority) and apply it where there is the greatest chance of a huge impact on the market (pick bold, decisive battles).

Making a Movie to Make Better Video Encoding

Fri, 12 Jul 2024 00:00:00 GMT

Until I read this Verge article, I had assumed that video codecs were a boring affair. In my mind, every few years, the industry would get together and come up with a new standard that promised better compression and better quality for the prevailing formats and screen types and, after some patent licensing back and forth, the industry would standardize around yet another MPEG standard that everyone uses. Rinse and repeat.

The article was an eye-opening look at how video streamers like Netflix are pushing the envelope on using video codecs. Since one of a video streamer’s core costs is the cost of video bandwidth, it would make sense that they would embrace new compression approaches (like different kinds of compression for different content, etc.) to reduce those costs. As Netflix embraces more live streaming content, it seems they’ll need to create new methods to accommodate.

But what jumped out to me the most was that, in order to better test and develop the next generation of codec, they produced a real 12 minute noir film called Meridian (you can access it on Netflix, below is someone who uploaded it to YouTube) which presents scenes that have historically been more difficult to encode with conventional video codecs (extreme lights and shadows, cigar smoke and water, rapidly changing light balance, etc).

Absolutely wild.

Reading Excerpt • The Verge

”

“While contributing to the development of new video codecs, Aaron and her team stumbled across another pitfall: video engineers across the industry have been relying on a relatively small corpus of freely available video clips to train and test their codecs and algorithms, and most of those clips didn’t look at all like your typical Netflix show. “The content that they were using that was open was not really tailored to the type of content we were streaming,” recalled Aaron. “So, we created content specifically for testing in the industry.” In 2016, Netflix released a 12-minute 4K HDR short film called Meridian that was supposed to remedy this. Meridian looks like a film noir crime story, complete with shots in a dusty office with a fan in the background, a cloudy beach scene with glistening water, and a dark dream sequence that’s full of contrasts. Each of these shots has been crafted for video encoding challenges, and the entire film has been released under a Creative Commons license. The film has since been used by the Fraunhofer Institute and others to evaluate codecs, and its release has been hailed by the Creative Commons foundation as a prime example of “a spirit of cooperation that creates better technical standards.””

Inside Netflix\u2019s Bet on Advanced Video Encoding

Janko Roettgers | The Verge

Games versus Points

Sun, 07 Jul 2024 00:00:00 GMT

The Dartmouth College Class of 2024, for their graduation, got a very special commencement address from tennis legend Roger Federer.

There is a wealth of good advice in it, but the most interesting point that jumped out to me is that while Federer won a whopping 80% of the matches he played in his career, he only won 54% of the points. It underscores the importance of letting go of small failures (“When you lose every second point, on average, you learn not to dwell on every shot”) but also of keeping your eye on the right metric (games, not points).

Biopharma scrambling to handle Biosecure Act

Thu, 27 Jun 2024 00:00:00 GMT

Strong regional industrial ecosystems like Silicon Valley (tech), Boston (life science), and Taiwan (semiconductors) are fascinating. Their creation is rare and requires local talent, easy access to supply chains and distribution, academic & government support, business success, and a good amount of luck.

But, once set in place, they can be remarkably difficult to unseat. Take the semiconductor industry as an example. It’s geopolitical importance has directed billions of dollars towards re-creating a domestic US industry. But, it faces an uphill climb. After all, it’s not only a question of recreating the semiconductor manufacturing factories that have gone overseas, but also:

the advanced and low-cost packaging technologies and vendors that are largely based in Asia
the engineering and technician talent that is no longer really in the US
the ecosystem of contractors and service firms that know exactly how to maintain the facilities and equipment
the supply chain for advanced chemicals and specialized parts that make the process technology work
the board manufacturers and ODMs/EMSs who do much of the actual work post-chip production that are also concentrated in Asia

A similar thing has happened in the life sciences CDMO (contract development and manufacturing organization) space. In much the same way that Western companies largely outsourced semiconductor manufacturing to Asia, Western biopharma companies outsourced much of their core drug R&D and manufacturing to Chinese companies like WuXi AppTec and WuXi Biologics. This has resulted in a concentration of talent and an ecosystem of talent and suppliers there that would be difficult to supplant.

Enter the BIOSECURE Act, a bill being discussed in the House with a strong possibility of becoming a law. It prohibits the US government from working with companies that obtain technology from Chinese biotechnology companies of concern (including WuXi AppTec and WuXi Biologics, among others). This is causing the biopharma industry significant anxiety as they are forced to find (and potentially fund) an alternative CDMO ecosystem that currently does not exist at the level of scale and quality as it does with WuXi.

Reading Excerpt • C&EN

”

“According to [Harvey Berger, CEO of Kojin Therapeutics], China’s CDMO industry has evolved to a point that no other country comes close to. “Tens of thousands of people work in the CDMO industry in China, which is more than the rest of the world combined,” he says. Meanwhile, Sound’s Kil says he has worked with five CDMOs over the past 15 years and is sure he wouldn’t return to three of them. The two that he finds acceptable are WuXi and a European firm. “When we asked the European CDMO about their capacity to make commercial-stage quantities, they told us they would have to outsource it to India,” Kil says. WuXi, on the other hand, is able to deliver large quantities very quickly. “It would be terrible for anyone to restrict our ability to work with WuXi AppTec.””

Industry braces for Biosecure Act impact

Aayushi Pratap | C&EN

Freedom and Prosperity Under Xi Jinping

Wed, 22 May 2024 00:00:00 GMT

Fascinating chart from Bloomberg showing level of economic freedom and prosperity under different Chinese rulers and how Xi Jinping is the first Chinese Communist Party ruler in history to have presided over sharp declines in both freedom and prosperity.

Given China’s rising influence in economic and geopolitical affairs, how it’s leaders (and in particular, Xi) and it’s people react to this will have significant impacts on the world

My Two-Year Journey to Home Electrification

Mon, 20 May 2024 00:00:00 GMT

Summary

Electrifying our (Bay Area) home was a complex and drawn-out process, taking almost two years.
Installing solar panels and storage was particularly challenging, involving numerous hurdles and unexpected setbacks.
We worked with a large solar installer (Sunrun) and, while the individuals we worked with were highly competent, handoffs within Sunrun and with other entities (like local utility PG&E and the local municipality) caused significant delays.
While installing the heat pumps, smart electric panel, and EV charger was more straightforward, these projects also featured greater complexity than we expected.
The project resulted in significant quality of improvements around home automation and comfort. However, bad pricing dynamics between electricity and natural gas meant direct cost savings from electrifying gas loads are, at best, small. While solar is an economic slam-dunk (especially given the rising PG&E rates our home sees), the batteries, in the absence of having backup, have less obvious economic value.
Our experience underscored the need for the industry to adopt a more holistic approach to electrification and for policymakers to make the process more accessible for all homeowners to achieve the state’s ambitious goals.

Why

The decision to electrify our home was an easy one. From my years of investing in & following climate technologies, I knew that the core technologies were reliable and relatively inexpensive. As parents of young children, my wife and I were also determined to contribute positively to the environment. We also knew there were abundant financial supports from local governments and utilities to help make this all work.

Yet, as we soon discovered, what we expected to be a straightforward path turned into a nearly two-year process!

Even for a highly motivated household which had budgeted significant sums for it all, it was still shocking how long (and much money) it took. It made me skeptical that households across California would be able to do the same to meet California’s climate goals without additional policy changes and financial support.

The Plan

Two years ago, we set out a plan:

Smart electrical panel — From my prior experience, I knew that many home electrification projects required a main electrical panel upgrade. These were typically costly and left you at the mercy of the utility to actually carry them out (I would find out how true this was later!). Our home had an older main panel rated for 125 A and we suspected we would normally need a main panel upgrade to add on all the electrical loads we were considering.

To try to get around this, we decided to get a smart electrical panel which could:
- use software smarts to deal with the times where peak electrical load got high enough to need the entire capacity of the electrical line
- give us the ability to intelligently manage backups and track solar production
  In doing our research, Span seemed like the clear winner. They were the most prominent company in the space and had the slickest looking device and app (many of their team had come from Tesla). They also had an EV charger product we were interested in, the Span Drive.
Heat pumps — To electrify is to ditch natural gas. As the bulk of our gas consumption was heating air and water, this involved replacing our gas furnace and gas water heater with heat pumps. In addition to significant energy savings — heat pumps are famous for their >200% efficiency (as they move heat rather than “create” it like gas furnaces do) — heat pumps would also let us add air conditioning (just run the heat pump in reverse!) and improve our air quality (from not combusting natural gas indoors). We found a highly rated Bay Area HVAC installer who specializes in these types of energy efficiency projects (called Building Efficiency) and trusted that they would pick the right heat pumps for us.
Solar and Batteries — No electrification plan is complete without solar. Our goal was to generate as much clean electricity as possible to power our new electric loads. We also wanted energy storage for backup power during outages (something that, while rare, we seemed to run into every year) and to take advantage of time-of-use rates (by storing solar energy when the price of electricity is low and then using it when the price is high).

We looked at a number of solar installers and ultimately chose Sunrun. A friend of ours worked there at the time and spoke highly of a prepaid lease they offered that was vastly cheaper all-in than every alternative. It offered minimum energy production guarantees, came with a solid warranty, and the “peace of mind” that the installation would be done with one of the largest and most reputable companies in the solar industry.
EV Charger — Finally, with our plan to buy an electric vehicle, installing a home charger at the end of the electrification project was a simple decision. This would allow us to conveniently charge the car at home, and, with solar & storage, hopefully let us “fuel up” more cost effectively. Here, we decided to go with the Span Drive. It’s winning feature was the ability to provide Level 2 charging speeds without a panel upgrade (it does this by ramping up or down charging speeds depending on how much electricity the rest of the house needed). While pricey, the direct integration into our Span smart panel (and its app) and the ability to hit high charging rates without a panel upgrade felt like the smart path forward.
What We Left Out — There were two appliances we decided to defer “fully going green” on.

The first was our gas stove (with electric oven). While induction stoves have significant advantages, because our current stove is still relatively new, works well, uses relatively little gas, and an upgrade would have required additional electrical work (installing a 240 V outlet), we decided to keep our current stove and consider a replacement at it’s end of life.

The second was our electric resistive dryer. While heat pump based dryers would certainly save us a great deal of electricity, the existing heat pump dryers on the market have much smaller capacities than traditional resistive dryers, which may have necessitated our family of four doing additional loads of drying. As our current dryer was also only a few years old, and already running on electricity, we decided we would also wait to consider heat pump dryer only after it’s end of life.

With what we thought was a well-considered plan, we set out and lined up contractors.

But as Mike Tyson put it, “Everyone has a plan ’till they get punched in the face.”

The Actual Timeline

Smart Panel

The smart panel installation was one of the more straightforward parts of our electrification journey. Span connected us with a local electrician who quickly assessed our site, provided an estimate, and completed the installation in a single day. However, getting the permits to pass inspection was a different story.

We failed the first inspection due to a disagreement over the code between the electrician and the city inspector. This issue nearly turned into a billing dispute with the electrician, who wanted us to cover the extra work needed to meet the code (an unexpected cost). Fortunately, after a few adjustments and a second inspection, we passed.

The ability to control and monitor electric flows with the smart panel is incredibly cool. For the first few days, I checked the charts in the apps every few minutes tracking our energy use while running different appliances. It was eye-opening to see just how much power small, common household items like a microwave or an electric kettle could draw!

However, the true value of a smart panel is only achieved when it’s integrated with batteries or significant electric loads that necessitate managing peak demand. Without these, the monitoring and control benefits are more novelties and might not justify the cost.

Note: if you, like us, use Pihole to block tracking ads, you’ll need to disable it for the Span app. The app uses some sort of tracker that Pihole flags by default. It’s an inconvenience, but worth mentioning for anyone considering this path.

Heating

Building Efficiency performed an initial assessment of our heating and cooling needs. We had naively assumed they’d be able to do a simple drop-in replacement for our aging gas furnace and water heater. While the water heater was a straightforward replacement (with a larger tank), the furnace posed more challenges.

Initially, they proposed multiple mini-splits to provide zoned control, as they felt the crawlspace area where the gas furnace resided was too small for a properly sized heat pump. Not liking the aesthetics of mini-splits, we requested a proposal involving two central heat pump systems instead.

Additionally, during the assessment, they found some of our old vents, in particular the ones sending air to our kids’ rooms, were poorly insulated and too small (which explains why their rooms always seemed under-heated in the winter). To fix this, they had to cut a new hole through our garage concrete floor (!!) to run a larger, better-insulated vent from our crawlspace. They also added insulation to the walls of our kids’ rooms to improve our home’s ability to maintain a comfortable temperature (but which required additional furniture movement, drywall work, and a re-paint).

Building Efficiency spec’d an Ecobee thermostat to control the two central heat pumps. As we already had a Nest Learning Thermostat (with Nest temperature sensors covering rooms far from the thermostat), we wanted to keep our temperature control in the Nest app. At the time, we had gotten a free thermostat from Nest after signing with Sunrun. We realized later, what Sunrun gifted us was the cheaper (and, less attractive) Nest Thermostat which doesn’t support Nest temperature sensors (why?), so we had to buy our own Nest Learning Thermostat to complete the setup.

Despite some of these unforeseen complexities, the whole process went relatively smoothly. There were a few months of planning and scheduling, but the actual installation was completed in about a week. It was a very noisy (cutting a hole through concrete is not quiet!) and chaotic week, but, the process was quick, and the city inspection was painless.

Solar & Storage

The installation of solar panels and battery storage was a lengthy ordeal. Sunrun proposed a system with LONGI solar panels, two Tesla Powerwalls, a SolarEdge inverter, and a Tesla gateway. Despite the simplicity of the plan, we encountered several complications right away.

First, a main panel upgrade was required. Although we had installed the Span smart panel to avoid this, Sunrun insisted on the upgrade and offered to cover the cost. Our utility PG&E took over a year (!!) to approve our request, which started a domino of delays.

After PG&E’s approval, Sunrun discovered that local ordinances needed a concrete pad to be poured and safety fence erected around the panel, requiring a subcontractor and yet more coordination.

After the concrete pad was in place and the panel installed, we faced another wait for PG&E to connect the new setup. Ironically, during this wait, I received a request from Sunrun to pour another concrete pad. This was, thankfully, a false alarm and occurred because the concrete pad / safety fence work had not been logged in Sunrun’s tracking system!

The solar and storage installation itself took only a few days, but during commissioning, a technician found that half the panels weren’t connected properly, necessitating yet another visit before Sunrun could request an inspection from the city.

Sadly, we failed our first city inspection. Sunrun’s team had missed a local ordinance that required the Powerwalls to have a minimum distance between them and the sealing off of vents within a certain distance from each Powerwall. This necessitated yet another visit from Sunrun’s crew, and another city inspection (which we thankfully passed).

The final step was obtaining Permission to Operate (PTO) from PG&E. The application for this was delayed due to a clerical error. About four weeks after submission, we finally received approval.

Seeing the flow of solar electricity in my Span app (below) almost brought a tear to my eye. Finally!

EV Charger

When my wife bought a Nissan Ariya in early 2023, it came with a year of free charging with EVgo. We hoped this would allow us enough time to install solar before needing our own EV charger. However, the solar installation took longer than expected (by over a year!), so we had to expedite the installation of a home charger.

Span connected us with the same electrician who installed our smart panel. Within two weeks of our free charging plan expiring, the Span Drive was installed. The process was straightforward, with only two notable complications we had to deal with:

The 20 ft cable on the Span Drive sounds longer than it is in practice. We adjusted our preferred installation location to ensure it comfortably reached the Ariya’s charging port.
The Span software initially didn’t recognize the Span Drive after installation. This required escalated support from Span to reset the software, costing the poor electrician who had expected the commissioning step to be a few minute affair to stick around my home for several hours.

Result

So, “was it worth it?” Yes! There are significant environmental (our carbon footprint is meaningfully lower) benefits. But there were also quality of life improvements and financial gains from these investments in what are just fundamentally better appliances.

Quality of Life

Our programmable, internet-connected water heater allows us to adjust settings for vacations, saving energy and money effortlessly. It also lets us program temperature cycles to avoid peak energy pricing, heating water before peak rates hit.

With the new heat pumps, our home now has air conditioning, which is becoming increasingly necessary in the Bay Area’s warmer summers. Improved vents and insulation have also made our home (and, in particular, our kids’ rooms) more comfortable. We’ve also found that the heat from the heat pumps is more even and less drying compared to the old gas furnace, which created noticeable hot spots.

Backup power during outages is another significant benefit. Though we haven’t had to use it since we received permission to operate, we had an accidental trial run early on when a Sunrun technician let our batteries be charged for a few days in the winter. During two subsequent outages in the ensuing months, our system maintained power to our essential appliances, ensuring our kids didn’t even notice the disruptions!

The EV charger has also been a welcome change. While free public charging was initially helpful, reliably finding working and available fast chargers could be time-consuming and stressful. Now, charging at home is convenient and cost-effective, reducing stress and uncertainty.

Financial

There are two financial aspects to consider: the cost savings from replacing gas-powered appliances with electric ones and the savings from solar and storage.

On the first, the answer is not promising.

The chart below comes from our PG&E bill for Jan 2023. It shows our energy usage year-over-year. After installing the heat pumps in late October 2022, our natural gas consumption dropped by over 98% (from 5.86 therms/day to 0.10), while our electricity usage more than tripled (from 15.90 kWh/day to 50.20 kWh/day). Applying the conversion of 1 natural gas therm = ~29 kWh of energy shows that our total energy consumption decreased by over 70%, a testament to the much higher efficiency of heat pumps.

Our PG&E bill from Feb 2023 (for Jan 2023)

Surprisingly, however, our energy bills remained almost unchanged despite this! The graph below shows our PG&E bills over the 12 months ending in Jan 2023. Despite a 70% reduction in energy consumption, the bill stayed roughly the same. This is due to the significantly lower cost of gas in California compared to the equivalent amount of energy from electricity. It highlights a major policy failing in California: high electricity costs (relative to gas) will deter households from switching to greener options.

Our PG&E bill from Feb 2023 (for Jan 2023)

Solar, however, is a clear financial winner. With our prepaid lease, we’d locked in savings compared to 2022 rates (just by dividing the total prepaid lease amount by the expected energy production over the lifetime of the lease), and these savings have only increased as PG&E’s rates have risen:

Interactive Spreadsheet Model

PG&E Electricity Rate Escalation & Solar Lease Payback Model

Historical tracking of PG&E tier and Time-of-Use rate inflation (2022 vs. 2024) benchmarking prepaid solar lease generation costs ($0.11/kWh) against grid escalation.

Tabs: Rate Comparison (2022 vs 2024) Solar Lease ROI TOU Schedule

Batteries, on the other hand, are much less clear-cut financially due to their high initial cost and only modest savings from time-shifting electricity use. However, the peace of mind from having backup power during outages is valuable (not to mention the fact that, without a battery, solar panels can’t be used to power your home during an outage), and, with climate change likely to increase both peak/off-peak rate disparities and the frequency of outages, we believe this investment will pay off in the long run.

Taking Advantage of Time of Use Rates

Time of Use (TOU) rates, like PG&E’s electric vehicle time of use rates, offer a smart way to reduce electricity costs for homes with solar panels, energy storage, and smart automation. This approach has fundamentally changed how we manage home energy use. Instead of merely conserving energy by using efficient appliances or turning off devices when not needed, we now view our home as a giant configurable battery. We “save” energy when it’s cheap and use it when it’s expensive.

Backup Reserve: We’ve set our Tesla Powerwall to maintain a 25% reserve. This ensures we always have a good supply of backup power for essential appliances (roughly 20 hours for our highest priority circuits by the Span app’s latest estimates) during outages
Summer Strategy: During summer, our Powerwall operates in “Self Power” mode, meaning solar energy powers our home first, then charges the battery, and lastly any excess goes to the grid. This maximizes the use of our “free” solar energy. We also schedule our heat pumps to run during midday when solar production peaks and TOU rates are lower. This way, we “store” cheaper energy in the form of pre-chilled or pre-heated air and water which helps maintain the right temperatures for us later (when the energy is more expensive).
Winter Strategy: In winter, we will switch the Powerwall to “Time-Based Control.” This setting preferentially charges the battery when electricity is cheap and discharges it when prices are high, maximizing the financial value of our solar energy during the months where solar production is likely to be limited.

This year will be our first full cycle with all systems in place, and we expect to make adjustments as rates and energy usage evolve. For those considering home electrification, hopefully these strategies give hints to what is possible to improve economic value of your setup.

Takeaways

Two years is too long: The average household might not have started this journey if they knew the extent of time and effort involved. This doesn’t even consider the amount of carbon emissions from running appliances off grid energy due to the delays. Streamlining the process is essential to make electrification more accessible and appealing.
Align gas and electricity prices with climate goals: The current pricing dynamics make it financially challenging for households to switch from gas appliances to greener options like heat pumps. To achieve California’s ambitious climate goals, it’s crucial to align the cost of electricity more closely with electrification.
Streamline permitting: Electrification projects are slowed by complex, inconsistent permitting requirements across different jurisdictions. Simplifying and unifying these processes will reduce time and costs for homeowners and their contractors.
Accelerate utility approvals: The two-year timeframe was largely due to delays from our local utility, PG&E. As utilities lack incentives to expedite these processes, regulators should build in ways to encourage utilities to move faster on home electrification-related approvals and activities, especially as many homes will likely need main panel upgrades to properly electrify.
Improve financing accessibility: High upfront costs make it difficult for households to adopt electrification, even when there are significant long-term savings. Expanding financing options (like Sunrun’s leases) can encourage more households to invest in these technologies. Policy changes should be implemented so that even smaller installers have the ability to offer attractive financing options to their clients.
Break down electrification silos: Coordination between HVAC specialists, solar installers, electricians, and smart home companies is sorely missing today. As a knowledgeable early adopter, I managed to integrate these systems on my own, but this shouldn’t be the expectation if we want broad adoption of electrification. The industry (in concert with policymakers) should make it easier for different vendors to coordinate and for the systems to interoperate more easily in order to help homeowners take full advantage of the technology.

This long journey highlighted to me, in a very visceral way, both the rewards and practical challenges of home electrification. While the environmental, financial, and quality-of-life benefits are clear, it’s also clear that we have a ways to go on the policy and practical hurdles before electrification becomes an easy choice for many more households. I only hope policymakers and technologists are paying attention. Our world can’t wait much longer.

How the Jones Act makes energy more expensive and less green

Mon, 06 May 2024 00:00:00 GMT

The Merchant Marine Act of 1920 (aka “The Jones Act”) is a law which requires ships operating between US ports to be owned by, made in, and crewed by US citizens.

While many “Made in the USA” laws are on the books and attract the anger of economists and policy wonks, the Jones Act is particularly egregious as the costs and effects are so large. The Jones Act costs states like Hawaii and Alaska and territories like Puerto Rico dramatically as they rely so much on ships for basic commerce that it was actually cheaper for Hawaii and New England to import oil from other countries (like Hawaii did from Russia until the Ukraine war) than it was to have oil shipped from the Gulf of Mexico (where American oil is abundant).

In the case of offshore wind, the Jones Act has pushed those companies willing to experiment with the promising technology, to ship the required parts and equipment from overseasbecause there are no Jones Act-compliant ships capable of moving the massive equipment that is involved.

This piece from Canary Media captures some of the dynamics and the “launch” of the still-in-construction $625 million Jones Act-compliant ship the Charybdis Dominion Energy will use to support its offshore wind facility.

Reading Excerpt • Canary Media

”

“To satisfy that mandate, Dominion commissioned the first-ever Jones Act–compliant vessel for offshore wind installation, which hit the water in Brownsville, Texas, last week. The hull welding on the 472-foot vessel is complete, as are its four enormous legs, which will hoist it out of the water during turbine installation. This $625 million leviathan, named Charybdis after the fearsome sea-monster foe of Odysseus, still needs some finishing touches before it sets sail to Virginia, which is expected to happen later this year. Charybdis’ completion will be a win for what’s left of the American shipbuilding industry. The Jones Act, after all, was intended to bolster American shipbuilders and merchant seamen in the isolationist spell following World War I. But a century later, it creates a series of confounding and counterintuitive challenges for America’s energy industry, which frequently redound poorly for most Americans. … Elsewhere in the energy industry, the expense and difficulty associated with finding scarce Jones Act–compliant ships push certain American communities to rely more on foreign energy suppliers. Up until 2022, Hawaii turned to Russia for one-third of the oil that powered its cars and power plants. The Jones Act made it too hard or costly to import abundant American oil to the U.S. state, leaving Hawaii scrambling for other sources when Russia invaded Ukraine. Over in New England, constraints on fossil-gas pipelines sometimes force the region to import gas via LNG terminals. The U.S. has plenty of fossil gas to tap in the Gulf of Mexico, but a lack of U.S. ships pushes Massachusetts and its neighbors to buy gas from other countries instead.”

US\u00a0offshore wind needs American-made ships. The first is nearly ready

Julian Spector | Canary Media

Backup Your Home Server with Duplicati

Wed, 24 Apr 2024 00:00:00 GMT

(Note: this is part of my ongoing series on cheaply selfhosting)

Through some readily available Docker containers and OpenMediaVault, I have a cheap mini-PC which serves as:

ad blocker for all the devices in my household
media streamer (so I can play movies and read downloaded ebooks/documents anywhere that has internet access)
acts as my personal RSS/newsreader
handles every PDF related operation you can think of
functions as network storage for my family

But, over time, as the server has picked up more uses, it’s also become a vulnerability. If any of the drives on my machine ever fail, I’ll lose data that is personally (and sometimes economically) significant.

I needed a home server backup plan.

Duplicati

Duplicati is open source software that helps you efficiently and securely backup specific partitions and folders to any destination. This could be another home server or it can be a cloud service provider (like Amazon S3 or Backblaze B2 or even a consumer service like Dropbox, Google Drive, and OneDrive). While there are many other tools that can support backup, I went with Duplicati because I wanted:

Support for consumer storage services as a target: I am a customer of Google Drive (through Google One) and Microsoft 365 (which comes with generous OneDrive subscription) and only intend to backup some of the files I’m currently storing (mainly some of the network storage I’m using to hold important files)
A web-based control interface so I could access this from any computer (and not just whichever machine had the software I wanted)
An active user forum so I could find how-to guides and potentially get help
Available as a Docker container on linuxserver.io: linuxserver.io is well-known for hosting and maintaining high quality and up-to-date Docker container images

Installation

Update 2024 Dec 18: One reason Duplicati is a great solution is that it is actively being developed. However, occasionally this can introduce breaking changes. Since version 2.0.9.105, Duplicati now requires a password. This has required an update to the Docker compose setup to include an Encryption Key, a Password, and an earlier update required Nginx proxy to pass additional headers to handle the Websocket-based interface the web interface now uses to keep its interface dynamic. I’ve changed the text below to reflect these

To install Duplicati on OpenMediaVault:

If you haven’t already, make sure you have OMV Extras and Docker Compose installed (refer to the section Docker and OMV-Extras in my previous post, you’ll want to follow all 10 steps as I refer to different parts of the process throughout this post) and have a static local IP address assigned to your server.
Login to your OpenMediaVault web admin panel, and then go to [Services > Compose > Files] in the sidebar. Press the button in the main interface to add a new Docker compose file.

Under Name put down Duplicati and under File, adapt the following (making sure the number of spaces are consistent)

---
services:
   duplicati:
     image: lscr.io/linuxserver/duplicati:latest
     container_name: duplicati
     ports:
       - &lt;unused port number&gt;:8200
     environment:
       - TZ: 'America/Los_Angeles'
       - PUID=&lt;UID of Docker User&gt;
       - PGID=&lt;GID of Docker User&gt;
       - DUPLICATI__WEBSERVICE_PASSWORD=&lt;Password to access interface&gt;
       - SETTINGS_ENCRYPTION_KEY=&lt;random set of at least 8 characters/numbers&gt;
     volumes:
       - &lt;absolute paths to folders to backup&gt;:&lt;names to use in Duplicati interface&gt;
       - &lt;absolute path to shared config folder&gt;/Duplicati:/config
     restart: unless-stopped

Under ports:, make sure to add an unused port number (I went with 8200).

Replace <absolute path to shared config folder> with the absolute path to the config folder where you want Docker-installed applications to store their configuration information (accessible by going to [Storage > Shared Folders] in the administrative panel).

You’ll notice there’s extra lines under volumes: for <absolute paths to folders to backup>. This should correspond with the folders you are interested in backing up. You should map them to names that will show up in the Duplicati interface that you recognize. For example, I directed my <absolute path to shared config folder> to /containerconfigs as one of the things I want to make sure I backup are my containers.

Once you’re done, hit Save and you should be returned to your list of Docker compose files for the next step. Notice that the new Duplicati entry you created has a Down status, showing the container has yet to be initialized.
To start your Duplicati container, click on the new Duplicati entry and press the (up) button. This will create the container, download any files needed, and run it.

To show it worked, go to your-servers-static-ip-address:8200 from a browser that’s on the same network as your server (replacing 8200 if you picked a different port in the configuration file above) and you should see the Duplicati web interface which should look something like below
You can skip this step if you didn’t set up Pihole and local DNS / Nginx proxy or if you don’t care about having a user-readable domain name for Duplicati. But, assuming you do and you followed my instructions, open up WeTTy (which you can do by going to wetty.home in your browser if you followed my instructions or by going to [Services > WeTTY] from OpenMediaVault administrative panel and pressing Open UI button in the main panel) and login as the root user. Run:

cd /etc/nginx/conf.d
ls
nano &lt;your file name&gt;.confCode language: Shell Session (shell)

This opens up the text editor nano with the file you just listed. Use your cursor to go to the very bottom of the file and add the following lines (making sure to use tabs and end each line with a semicolon)

server {
    listen             80;
    server_name        &lt;duplicati.home or the domain you'd like to use&gt;;
    location / {
        proxy_pass             http://&lt;your-server-static-ip&gt;:&lt;duplicati port no.&gt;;
        proxy_http_version     1.1;
        proxy_set_header       Upgrade $http_upgrade;
        proxy_set_header       Connection "upgrade";
        proxy_set_header       Host $host;
        proxy_set_header       X-Real-IP $remote_addr;
        proxy_set_header       X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header       X-Forwarded-Proto $scheme;
    }
}Code language: HTML, XML (xml)

And then hit Ctrl+X to exit, Y to save, and Enter to overwrite the existing file. Then in the command line run the following to restart Nginx with your new configuration loaded.

systemctl restart nginx

Now, if your server sees a request for duplicati.home (or whichever domain you picked), it will direct them to Duplicati. With the additional proxy_http_version and proxy_set_header‘s, it will also properly forward the Websocket requests the web interface uses.
Login to your Pihole administrative console (you can just go to pi.hole in a browser) and click on [Local DNS > DNS Records] from the sidebar. Under the section called Add a new domain/IP combination, fill out under Domain: the domain you just added above (i.e. duplicati.home) and next to IP Address: you should add your server’s static IP address. Press the Add button and it will show up below.
To make sure it all works, enter the domain you just added (duplicati.home if you went with my default) in a browser and you should see the Duplicati interface!

Configuring your Backups

Duplicati conceives of each “backup” as a “source” (folder of files to backup), a “destination” (the place the files should be backed up to), a schedule (how often does the backup run), and some options to configure how the backup works.

After logging in (with the password you specified in the Docker compose file), to configure a “backup”, click on +Add Backup button on the menu on the lefthand side. I’ll show you the screens I went through to backup my Docker container configurations:

Add a name (I called it DockerConfigs) and enter a Passphrase (you can use the Generate link to create a strong password) which you’d use to restore from backup. Then hit Next
Enter a destination. Here, you can select another computer or folder connected to your network. You can also select an online storage service.

I’m using Microsoft OneDrive — for a different service, a quick Google search or a search of the Duplicati how-to forum can give you more specific instructions, but the basic steps of generating an AuthID link appear to be similar across many services.

I selected Microsoft OneDrive v2 and picked a path in my OneDrive for the backup to go to (Backup/dockerconfigs). I then clicked on the AuthID link and went through an authentication process to formally grant Duplicati access to OneDrive. Depending on the service, you may need to manually copy a long string of letters and numbers and colons into the text field. After all of that, to prove it all worked, press Test connection!

Then hit Next
Select the source. Use the folder browsing widget on the interface to select the folder you wish to backup.

If you recall in my configuration step, I mapped the <absolute path to shared config folder> to /containerconfigs which is why I selected this as a one-click way to backup all my Docker container configurations. If necessary, feel free to shut down and delete your current container and start over with a configuration where you point and map the folders in a better way.

Then hit Next
Pick a schedule. Do you want to backup every day? Once a week? Twice a week? Since my docker container configurations don’t change that frequently, I decided to schedule weekly backups on Saturday early morning (so it wouldn’t interfere with something else I might be doing).

Pick your option and then hit Next
Select your backup options. Unless you have a strong reason to, I would not change the remote volume size from the default (50 MB). The backup retention, however, is something you may want to think about. Duplicati gives you the option to hold on to every backup (something I would not do unless you have a massive amount of storage relative to the amount of data you want to backup), to hold on to backups younger than a certain age, to hold on to a specific number of backups, or customized permutations of the above.

The option you should choose depends on your circumstances, but to share what I did. For some of my most important files, I’m using Duplicati’s smart backup retention option (which gives me one backup from the last week, one for each of the last 4 weeks, and one for each of the last 12 months). For some of my less important files (for example, my docker container configurations), I’m holding on to just the last 2 weeks worth of backups.

Then hit Save and you’re set!

I hope this helps you on your self-hosted backup journey.

If you’re interested in how to setup a home server on OpenMediaVault or how to self-host different services, check out all my posts on the subject!

The California home insurance conundrum

Mon, 15 Apr 2024 00:00:00 GMT

As a California homeowner, I’ve watched with dismay as homeowner insurance provider after homeowner insurance provider have fled the state in the face of wildfire risk.

It was quite the shock when I discovered recently (HT: Axios Markets newsletter) that, according to NerdWallet, California actually has some of the cheapest homeowners insurance rates in the country!

Data Analysis

Homeowners Insurance Average Cost by State (2024)

Source: Bankrate / Infogram

Rank / State	Avg Annual Premium ($300k Dwelling)	vs National Average ($1,759)
1. Florida	$5,538 / yr	+215%
2. Louisiana	$4,863 / yr	+176%
3. Oklahoma	$4,570 / yr	+160%
21. California	$1,452 / yr	-17% (Artificially Suppressed)
50. Hawaii	$582 / yr	-67%

*Note: California's Proposition 103 regulatory regime kept headline premium rates below the national average despite soaring wildfire loss costs, contributing to insurer non-renewals.

It begs the Econ 101 question — is it really that the cost of wildfires are too high? Or that the price insurance companies can charge (something heavily regulated by state insurance commissions) is kept too low / not allowed to vary enough based on actual fire risk?

Why Intel has to make its foundry business work

Wed, 10 Apr 2024 00:00:00 GMT

Historically, Intel has (1) designed and (2) manufactured its chips that it sells (primarily into computer and server systems). It prided itself on having the most advanced (1) designs and (2) manufacturing technology, keeping both close to its chest.

In the late 90s/00s, semiconductor companies increasingly embraced the “fabless model”, whereby they would only do the (1) design while outsourcing the manufacturing to foundries like TSMC. This made it much easier and less expensive to build up a burgeoning chip business and is the secret to the success of semiconductor giants like NVIDIA and Qualcomm.

Companies like Intel scoffed at this, arguing that the combination of (1) design and (2) manufacturing gave their products an advantage, one that they used to achieve a dominant position in the computing chip segment. And, it’s an argument which underpins why they have never made a significant effort in becoming a contract manufacturer — after all, if part of your technological magic is the (2) manufacturing, why give it to anyone else?

The success of TSMC has brought a lot of questions about Intel’s advantage in manufacturing and, given recent announcements by Intel and the US’s CHIPS Act, a renewed focus on actually becoming a contract manufacturer to the world’s leading chip designers.

While much of the attention has been paid to the manufacturing prowess rivalry and the geopolitical reasons behind this, I think the real reason Intel has to make the foundry business work is simple: their biggest customers are all becoming chip designers.

While a lot of laptops and desktops and servers are still sold in the traditional fashion, the reality is more and more of the server market is being dominated by a handful of hyperscale data center operators like Amazon, Google, Meta/Facebook, and Microsoft, companies that have historically been able to obtain the best prices from Intel because of their volume. But, in recent years, in the chase for better and better performance and cost and power consumption, they have begun designing their own chips adapted to their own systems (as this latest Google announcement for Google’s own ARM-based server chips shows).

Are these chips as good as Intel’s across every dimension? Almost certainly not. It’s hard to overtake a company like Intel’s decades of design prowess and market insight. But, they don’t have to be. They only have to be better at the specific use case Google / Microsoft / Amazon / etc need it to be for.

And, in that regard, that leaves Intel with really only one option: it has to make the foundry business work, or it risks losing not just the revenue from (1) designing a data center chip, but from the (2) manufacturing as well.

Reading Excerpt • Google Blog

”

“Axion processors combine Google’s silicon expertise with Arm’s highest performing CPU cores to deliver instances with up to 30% better performance than the fastest general-purpose Arm-based instances available in the cloud today, up to 50% better performance and up to 60% better energy-efficiency than comparable current-generation x86-based instances1. That’s why we’ve already started deploying Google services like BigTable, Spanner, BigQuery, Blobstore, Pub/Sub, Google Earth Engine, and the YouTube Ads platform on current generation Arm-based servers and plan to deploy and scale these services and more on Axion soon.”

Introducing Google Axion Processors, our new Arm-based CPUs

Amin Vahdat | Google Blog

Starlink in the wrong hands

Tue, 26 Mar 2024 00:00:00 GMT

On one level, this shouldn’t be a surprise. Globally always available satellite constellation = everyone and anyone will try to access this. This was, like many technologies, always going to have positive impacts — i.e. people accessing the internet where they otherwise couldn’t due to lack of telecommunications infrastructure or repression — and negative — i.e. terrorists and criminal groups evading communications blackouts.

The question is whether or not SpaceX had the foresight to realize this was a likely outcome and to institute security processes and checks to reduce the likelihood of the negative.

That remains to be seen…

Reading Excerpt • Bloomberg

”

“In Yemen, which is in the throes of a decade-long civil war, a government official conceded that Starlink is in widespread use. Many people are prepared to defy competing warring factions, including Houthi rebels, to secure terminals for business and personal communications, and evade the slow, often censored internet service that’s currently available. Or take Sudan, where a year-long civil war has led to accusations of genocide, crimes against humanity and millions of people fleeing their homes. With the regular internet down for months, soldiers of the paramilitary Rapid Support Forces are among those using the system for their logistics, according to Western diplomats.”

Elon Musk\u2019s Starlink Terminals Are Falling Into the Wrong Hands

Bruce Einhorn, Loni Prinsloo, Marissa Newman, Simon Marks | Bloomberg

Why don’t we (still) have rapid viral diagnostics?

Mon, 25 Mar 2024 00:00:00 GMT

One of the most disappointing outcomes in the US from the COVID pandemic was the rise of the antivaxxer / public health skeptic and the dramatic politicization of public health measures.

But, not everything disappointing has stemmed from that. Our lack of cheap rapid tests for diseases like Flu and RSV is a sad reminder of our regulatory system failing to learn from the COVID crisis of the value of cheap, rapid in-home testing or adopting to the new reality that many Americans now know how to do such testing.

Reading Excerpt • NBC News

”

“Dr. Michael Mina, the chief science officer for the at-home testing company eMed, said the FDA tends to have strict requirements for over-the-counter tests. The agency often asks manufacturers to conduct studies that demonstrate that people can administer at-home tests properly — a process that may cost millions of dollars and delay the test’s authorization by months or years, Mina said. “It’s taken a very long time in the past to get new self-tests authorized, like HIV tests or even pregnancy tests,” he said. “They’ve taken years and years and years and years. We have a pretty conservative regulatory approach.””

Rapid tests for Covid, RSV and the flu are available in Europe. Why not in the U.S.?

Aria Bendix | NBC News

Huggingface: security vulnerability?

Wed, 20 Mar 2024 00:00:00 GMT

Anyone who’s done any AI work is familiar with Huggingface. They are a repository of trained AI models and maintainer of AI libraries and services that have helped push forward AI research. It is now considered standard practice for research teams with something to boast to publish their models to Huggingface for all to embrace. This culture of open sharing has helped the field make its impressive strides in recent years and helped make Huggingface a “center” in that community.

However, this ease of use and availability of almost every publicly accessible model under the sun comes with a price. Because many AI models require additional assets as well as the execution of code to properly initialize, Huggingface’s own tooling could become a vulnerability. Aware of this, Huggingface has instituted their own security scanning procedures on models they host.

But security researchers at JFrog have found that even with such measures, have identified a number of models that exploit gaps in Huggingface’s scanning which allow for remote code execution. One example model they identified baked into a Pytorch model a “phone home” functionality which would initiate a secure connection between the server running the AI model and another (potentially malicious) computer (seemingly based in Korea).

The JFrog researchers were also able to demonstrate that they could upload models which would allow them to execute other arbitrary Python code which would not be flagged by Huggingface’s security scans.

While I think it’s a long way from suggesting that Huggingface is some kind of security cesspool, the research reminds us that so long as a connected system is both popular and versatile, there will always be the chance for security risk, and it’s important to keep that in mind.

Reading Excerpt • JFrog blog

”

“As with other open-source repositories, we’ve been regularly monitoring and scanning AI models uploaded by users, and have discovered a model whose loading leads to code execution, after loading a pickle file. The model’s payload grants the attacker a shell on the compromised machine, enabling them to gain full control over victims’ machines through what is commonly referred to as a “backdoor”. This silent infiltration could potentially grant access to critical internal systems and pave the way for large-scale data breaches or even corporate espionage, impacting not just individual users but potentially entire organizations across the globe, all while leaving victims utterly unaware of their compromised state.”

Data Scientists Targeted by Malicious Hugging Face ML Models with Silent Backdoor

David Cohen | JFrog blog

Nope, the Dunning-Kruger Effect is just bad statistics

Mon, 18 Mar 2024 00:00:00 GMT

The Dunning-Kruger effect encapsulates something many of us feel familiar with: that the least intelligent oftentimes assume they know more than they actually do. Wrap that sentiment in an academic paper written by two professors at an Ivy League institution and throw in some charts and statistics and you’ve got a easily citable piece of trivia to make yourself feel smarter than the person who you just caught commenting on something they know nothing about.

Well, according to this fascinating blog post (HT: Eric), we have it all wrong. The way that Dunning-Kruger constructed their statistical test was designed to always construct a positive relationship between skill and perceived ability.

The whole thing is worth a read, but they showed that using completely randomly generated numbers (where there is no relationship between perceived ability and skill), you will always find a relationship between the “skill gap” (perceived ability – skill) and skill, or to put it more plainly,

With y being perceived ability and x being actual measured ability.

What you should be looking for is a relationship between perceived ability and measured ability (or directly between y and x) and when you do this with data, you find that the evidence for such a claim generally isn’t there!

In other words:

A Heart Atlas

Fri, 15 Mar 2024 00:00:00 GMT

The human heart is an incredibly sophisticated organ that, in addition to being one of the first organs developed while embryos develop, is quite difficult to understand at a cellular level (where are the cells, how do they first develop, etc.).

Neil Chi’s group at UCSD (link to Nature paper) were able to use multiplex imaging of fluorescent-tagged RNA molecules to profile the gene expression profiles of different types of heart cells and see where they are located and how they develop!

The result is an amazing visualization, check it out at the video:

Cat Bond Fortunes

Thu, 14 Mar 2024 00:00:00 GMT

Until recently, I only knew of the existence of cat(astrophe) bonds — financial instruments used to raise money for insurance against catastrophic events where investors profit when no disaster happens.

I had no idea, until reading this Bloomberg article about the success of Fermat Capital Management, how large the space had gotten ($45 billion!!) or how it was one of the most profitable hedge fund strategies of 2023!

This is becoming an increasingly important intersection between climate change and finance as insurance companies and property owners struggle with the rising risk of rising damage from extreme climate events. Given how young much of the science of evaluating these types of risks is, it’s no surprise that quantitative minds and modelers are able to profit here.

The entire piece reminded me of Richard Zeckhauser’s famous 2006 article Investing in the Unknown and Unknowable which covers how massive investment returns can be realized by tackling problems that seem too difficult for other investors to understand.

Reading Excerpt • Bloomberg

”

“Investing in cat bonds was the most profitable hedge fund strategy of 2023. Fermat delivered a 20% return, beating the average 8% achieved by hedge funds as a whole. While other cat bond funds did well too, Fermat’s $10 billion portfolio — capturing a quarter of the market — made it by far the most prolific investor to take advantage of a bumper year. Cat bonds investors are gambling on nature. If a disaster they’ve bet on occurs, their money is used to settle insurance claims. If it doesn’t, they get handsome returns. For decades, the instruments were a last resort reserved for super-rare events, such as a cataclysmic storm on the scale of Hurricane Katrina. But multibillion-dollar calamities have become alarmingly frequent on a warmer planet. “The insurance market is on edge,” says Seo. “It’s freaked out about risk and wants as little as possible.””

How a Physics Whiz Made a Fortune Betting on Nature\u2019s Catastrophes

Gautam Naik | Bloomberg

Shein and Temu now drive global cargo

Tue, 12 Mar 2024 00:00:00 GMT

Maybe you have shopped on Shein or Temu. Maybe you only know someone (younger?) who has. Maybe you only know Temu because of their repeat Superbowl ads.

But these Chinese eCommerce companies are now the main driver behind air and ship cargo rates with Temu and Shein combined accounting for 9,000 tons per day of shipments!

This is scale.

Geothermal data centers

Fri, 08 Mar 2024 00:00:00 GMT

The data centers that power AI and cloud services are limited by 3 things:

the server hardware (oftentimes limited by access to advanced semiconductors)
available space (their footprint is massive which makes it hard to put them close to where people live)
availability of cheap & reliable (and, generally, clean) power

If you, as a data center operator, can tap a new source of cheap & reliable power, you will go very far as you alleviate one of the main constraints on the ability to add to your footprint.

It’s no small wonder, then, that Google is willing to explore partnerships with next-gen geothermal startups like Fervo in a meaningful long-term fashion.

Dexcom non-prescription glucose monitor approved

Wed, 06 Mar 2024 00:00:00 GMT

Cheap and accurate continuous glucose monitoring is a bit of a holy grail for consumer metabolic health as it allows people to understand how their diet and exercise impact their blood sugar levels, which can vary from person to person.

It’s also a holy grail for diabetes care as making sure blood sugar levels are neither too high nor too low is critical for health (too low and you can pass out or risk seizure or coma; too high and you risk diabetic neuropathy, kidney disease, and cardiovascular problems). For Type I diabetics and severe Type II diabetics, it’s also vital for dosing insulin.

Because insulin dosing needs to be done just right, I was always under the impression that one of two things would happen along the way to producing a cheap continuous glucose monitor, either:

The FDA would be hesitant to approve a device that wasn’t highly accurate to avoid the risk of a consumer using the reading to mis-dose insulin OR
The device makers (like Dexcom) would be hesitant to create an accurate enough glucose monitor that it might cannibalize their highly profitable prescription glucose monitoring business

As a result, I was pleasantly surprised that Dexcom’s over-the-counter Stelo continuous glucose monitor was approved by the FDA. It remains to be seen what the price will be and what level of information the Stelo will share with the customer, but I view this as a positive development and (at least for now) tip my hat to both the FDA and Dexcom here.

(Thanks to Erin Brodwin from Axios for sharing the news on X)

Reading Excerpt • CNBC

”

“Dexcom shared Stelo’s name, and that the device had been submitted to the FDA for review in February. The sensor will be worn on the upper arm and lasts for up to 15 days before it needs to be replaced, according Dexcom. Jake Leach, chief operating officer at Dexcom, told CNBC in February that Stelo will have a unique platform and branding. The platform will be tailored to the needs of these Type 2 patients, he said, which means it will not include many of the alerts and notifications meant for diabetes patients at risk of experiencing more serious emergencies. “It’s designed to be a simpler experience,” Leach said in an interview. “There’s a lot of people who could benefit.””

Dexcom\u2019s first-ever over-the-counter glucose monitor patch gets FDA clearance

Ashley Capoot | CNBC

“Corporate” Design

Tue, 05 Mar 2024 00:00:00 GMT

Read an introspective piece by famed ex-Frog Design leader Robert Fabricant about the state of the design industry and the unease that he says many of his peers are feeling. While I disagree with some of the concerns he lays out around AI / diversity being the drivers of this unease, he makes a strong case for how this is a natural pendulum swing after years of seeing “Chief Design Officers” and design innovation groups added to many corporate giants.

I’ve had the privilege of working with very strong designers. This has helped me appreciate the value of design thinking as something that goes far beyond “making things pretty” and believe, wholeheartedly, that it’s something that should be more broadly adopted.

At the same time, it’s also not a surprise to me that during a time of layoffs and cost cutting, a design function which has become a little “spoiled” in the past years and of which calculating financial returns is experiencing some painful transition especially for creative-minded designers who struggle with that ROI evolution.

If Phase 1 was getting companies to recognize that design thinking is needed, Phase 2 will be the space learning how to measure, communicate, and optimize what the value of a team of seasoned designers brings to the bottom line.

Reading Excerpt • Fast Company

”

“In recent years, there has also been a shift in the very nature of design leadership inside companies. Where the early corporate pioneers were in “building mode” to establish and integrate design capabilities, these functions are now focused on optimization. That is where the professionals at IBM and Accenture/Song come in, with creativity giving way to utilization as the primary metric of success, particularly in an economic downturn. Magowan of Design Leaders is hearing this sentiment from his clients on the hiring side. “Companies are still adjusting,” he says. “Trying to understand how to embed design properly and not make the mistakes they made by hiring folks into design leadership roles who were not equipped.” This transition is not well-suited to most creative leaders, even proven ones like Powell. “Scaling takes attention,” he says. “But once you get there, then you need to keep turning the dial so that this [design] machine runs more and more efficiently. For me that is where it got to be not as fun.” Perhaps 2024, then, is the right moment for the big handoff to the folks at McKinsey Design to lead the charge armed with their corporate reports and CDO roundtables on business value and design effectiveness?”

The big design freak-out: A generation of design leaders grapple with their future

Robert Fabricant | Fast Company

Costco Love

Mon, 04 Mar 2024 00:00:00 GMT

Nice piece in the Economist about how Costco’s model of operational simplicity leads to a unique position in modern retail: beloved by customers, investors, AND workers:

sell fewer things ➡️
get better prices from suppliers & less inventory needed ➡️
lower costs for customers ➡️
more customers & more willing to pay recurring membership fee ➡️
strong, recurring profits ➡️
ability to pay well and promote from within 📈💪🏻

How packaging tech is changing how we build & design chips

Sat, 02 Mar 2024 00:00:00 GMT

Once upon a time, the hottest thing in chip design was “system-on-a-chip” (SOC). The idea is that you’d get the best cost and performance out of a chip by combining more parts into one piece of silicon. This would result in smaller area (less silicon = less cost) and faster performance (closer parts = faster communication) and resulted in more and more chips integrating more and more things.

While the laws of physics haven’t reversed any of the above, the cost of designing chips that integrate more and more components has gone up sharply. Worse, different types of parts (like on-chip memory and physical/analog componentry) don’t scale down as well as pure logic transistors, making it very difficult to design chips that combine all these pieces.

The rise of new types of packaging technologies, like Intel’s Foveros, Intel’s EMIB, TSMC’s InFO, new ways of separating power delivery from data delivery (backside power delivery), and more, has also made it so that you can more tightly integrate different pieces of silicon and improve their performance and size/cost.

The result is now that many of the most advanced silicon today is built as packages of chiplets rather than as massive SOC projects: a change that has happened over a fairly short period of time.

This interview with IMEC (a semiconductor industry research center)’s head of logic technologies breaks this out…

Reading Excerpt • IEEE Spectrum

”

“What we’re doing in CMOS 2.0 is pushing that idea further, with much finer-grained disintegration of functions and stacking of many more dies. A first sign of CMOS 2.0 is the imminent arrival of backside-power-delivery networks. On chips today, all interconnects—both those carrying data and those delivering power—are on the front side of the silicon [above the transistors]. Those two types of interconnect have different functions and different requirements, but they have had to exist in a compromise until now. Backside power moves the power-delivery interconnects to beneath the silicon, essentially turning the die into an active transistor layer which is sandwiched between two interconnect stacks, each stack having a different functionality.”

What is CMOS 2.0?

Samuel K. Moore | IEEE Spectrum

Store all the things: clean electricity means thermal energy storage boom

Fri, 01 Mar 2024 00:00:00 GMT

Thermal energy storage has been a difficult place for climatetech in years past. The low cost of fossil fuels (the source for vast majority of high temperature industrial heat to date) and the failure of large scale solar thermal power plants to compete with the rapidly scaling solar photovoltaic industry made thermal storage feel like, at best, a market reserved for niche applications with unique fossil fuel price dynamics. This is despite some incredibly cool (dad-joke intended 🔥🥵🤓) technological ingenuity in the space.

But, in a classic case of how cheap universal inputs change market dynamics, the plummeting cost and soaring availability of renewable electricity and the growing desire for industrial companies to get “clean” sources of industrial heat has resulted in almost a renaissance for the space as this Canary Media article (with a very nice table of thermal energy startups) points out.

With cheap renewables (especially if the price varies), companies can buy electricity at low (sometimes near-zero if in the middle of a sunny and windy day) prices, convert that to high-temperature heat with an electric furnace, and store it for use later.

While the devil’s in the details, in particular the round trip energy efficiency (how much energy you can get out versus what you put in), the delivered heat temperature range and rate (how hot and how much power), and, of course, the cost of the system, technologies like this could represent a key technology to green sectors of the economy that would otherwise be extremely difficult to lower carbon output for.

The IE6 YouTube conspiracy

Wed, 28 Feb 2024 00:00:00 GMT

An oldie but a goodie — the story of how the YouTube team, post-Google acquisition, put up a “we won’t support Internet Explorer 6 in the future” message without any permission from anyone. (HT: Eric S)

Reading Excerpt • Chris Zacharias

”

“IE6 had been the bane of our web development team’s existence. At least one to two weeks every major sprint cycle had to be dedicated to fixing new UI that was breaking in IE6. Despite this pain, we were told we had to continue supporting IE6 because our users might be unable to upgrade or might be working at companies that were locked in. IE6 users represented around 18% of our user base at that point. We understood that we could not just drop support for it. However, sitting in that cafeteria, having only slept about a few hours each in the previous days, our compassion for these users had completely eroded away. We began collectively fantasizing about how we could exact our revenge on IE6. One idea rose to the surface that quickly captured everyone’s attention. Instead of outright dropping IE6 support, what if we just threatened to? How would users react? Would they revolt against YouTube? Would they mail death threats to our team like had happened in the past? Or would they suddenly become loud advocates of modern browsers?”

A Conspiracy to Kill IE6

Chris Zacharias

Using your ear to control devices

Tue, 27 Feb 2024 00:00:00 GMT

Very cool that we’re still finding new things we can control that can be applied to making the lives of people better.

Reading Excerpt • EETimes

”

“I found that a small, middle-ear muscle called the tensor tympani can be tensed voluntarily, otherwise known as “ear rumbling.” In studies funded by the National Institute for Health and Care Research (NIHR), we found that about 55% of people could tense the muscle in isolation, and about 80% could do so when yawning. We believe that some people with neurological disabilities retain control over the tensor tympani, even when they’ve lost almost all other functionality. The muscle also moves physiologically as the eyes move, a fact that presents a further opportunity to track people’s intent or augment how we communicate non-verbally. The ear isn’t just an auditory input device; it also has the potential to be a complex input and output tool.”

Meet the Ear Muscle That Could Hold a Key to Assistive Tech

Nick Gompertz | EETimes

Intel’s focus on chip packaging technology

Mon, 26 Feb 2024 00:00:00 GMT

Intel has been interested in entering the foundry (semiconductor contract manufacturing) space for a long time. For years, Intel proudly boasted of being at the forefront of semiconductor technology — being first to market with the FinFET and smaller and smaller process geometries.

So it’s interesting how, with the exception of the RibbonFET (the successor to the FinFET), almost all of Intel’s manufacturing technology announcements (see whitepaper) in it’s whitepaper to appeal to prospective foundry customers, all of it’s announcements pertain to packaging / “back end” technologies.

I think it’s both a recognition that they are no longer the furthest ahead in that race, as well as recognition that Moore’s Law scaling has diminishing returns for many applications. Now, a major cost and performance driver is technology that was once considered easily outsourced to low cost assemblers in Asia is now front and center.

Iovance brings cell therapy to solid tumors

Sat, 24 Feb 2024 00:00:00 GMT

Immune cell therapy — the use of modified immune cells directly to control cancer and autoimmune disease — has shown incredible results in liquid tumors (cancers of the blood and bone marrow like lymphoma, leukemia, etc), but has stumbled in addressing solid tumors.

Iovance, which recently had its drug lifileucel approved by the FDA to treat advanced melanoma, has demonstrated an interesting spin on the cellular path which may prove to be effective in solid tumors. They extract Tumor-Infiltrating Lymphocytes (TILs), immune cells that are already “trying” to attack a solid tumor directly. Iovance then treats those TILs with their own proprietary process to expand the number of those cells and “further activate” them (to resist a tumor’s efforts to inactivate immune cells that may come after them) before reintroducing them to the patient.

This is logistically very challenging (not dissimilar to what patients awaiting other cell therapies or Vertex’s new sickle cell treatment need to go through) as it also requires chemotherapy for lymphocyte depletion in the patient prior to reintroduction of the activated TILs. But, the upshot is that you now have an expanded population of cells known to be predisposed to attacking a solid tumor that can now resist the tumor’s immune suppression efforts.

And, they’ve presented some impressive 4-year followup data on a study of advanced melanoma in patients who have already failed immune checkpoint inhibitor therapy, enough to convince the FDA of their effectiveness!

To me, the beauty of this method is that it can work across tumor types. Iovance’s process (from what I’ve gleamed from their posters & presentations) works by getting more and more activated immune cells. Because they’re derived from the patient, these cells are already predisposed to attack the particular molecular targets of their tumor.

This is contrast to most other immune cell therapy approaches (like CAR-T) where the process is inherently target-specific (i.e. get cells that go after this particular marker on this particular tumor) and each new target / tumor requires R&D work to validate. Couple this with the fact that TILs are already the body’s first line of defense against solid tumors and you may have an interesting platform for immune cell therapy in solid tumors.

The devil’s in the details and requires more clinical study on more cancer types, but suffice to say, I think this is incredibly exciting!

Another Italian merchant invention: the decimal point!

Fri, 23 Feb 2024 00:00:00 GMT

I’ve always been astonished by how many things we use now came from Renaissance Italian merchants: the @ sign, double-entry bookkeeping, banking, maritime insurance, and now the decimal point

Reading Excerpt • Nature

”

“Van Brummelen suggests that Bianchini’s schooling in economics might have been key to his invention, because he wasn’t embedded in sexagesimal numbers from early in his career, as other astronomers were. But his approach was perhaps too revolutionary to catch on at first. “In order to understand what Bianchini was doing, you had to learn a completely new system of arithmetic,” he says. A century and a half later, however, “decimal notation was in the air”. Astronomers working with smaller and smaller subdivisions were inventing different systems, desperate for ways to simplify complex calculations. Clavius’s work influenced later popularizers of decimal fractions, such as Flemish mathematician Simon Stevin, as well as Scottish astronomer and inventor of logarithms John Napier, who adopted the decimal point. Chabás argues that historians should reassess Bianchini’s importance. Although he has been “eclipsed” by other figures, there’s clearly “a path of ideas”, he says, leading back to Bianchini.”

The decimal point is 150 years older than historians thought

Jo Marchant | Nature

Wind and solar closing on fossil fuels in EU power generation

Thu, 22 Feb 2024 00:00:00 GMT

This one chart (published in Canary Media) illustrates both the case for optimism for our ability to deal with climate change as well as a clear case of how geopolitical pressures can dramatically impact energy choices: the rapid increase in use of renewable energy (mainly at the expense of fossil fuels) as source of electricity in the EU.

Don’t Pay for Adobe Acrobat to do Basic PDF Things

Wed, 21 Feb 2024 00:00:00 GMT

(Note: this is part of my ongoing series on cheaply selfhosting)

If you’re like me, every few months you have to do something with PDFs:

Merge them
Rotate them
Crop them
Add / remove a password
Move pages around / remove pages
Sign them
Add text / annotations to them

This ends up either being a pain to do (via some combination of screen shots, printing, scanning, and exporting) or oddly expensive (buying a license to Adobe Acrobat or another pay-PDF manipulation tool).

Enter Stirling PDF tools, a set of free web-based PDF manipulation tools which can also be selfhosted on any server supporting Docker. Given my selfhosting journey these past couple of months, this seemed like a perfect project to take on.

In the hopes that this helps anyone who has ever had to do some PDF manipulation work done, I will share how I set up Stirling PDF tools (on my OpenMediaVault v6 home server)

Stirling PDF

Stirling tools started as a ChatGPT project which has since turned into an open source project with millions of Docker pulls. It handles everything through a simple web interface and on the server (no calls to any remote service). Depending on the version you install, you can also get access to tools converting common Office files to PDF and OCR (optical character recognition, where software can recognize text — even handwriting — in images).

And, best of all, it’s free! (As in beer and as in freedom!)

Installation

To install the Stirling Tools on OpenMediaVault:

If you haven’t already, make sure you have OMV Extras and Docker Compose installed (refer to the section Docker and OMV-Extras in my previous post, you’ll want to follow all 10 steps as I refer to different parts of the process throughout this post) and have a static local IP address assigned to your server.
Login to your OpenMediaVault web admin panel, and then go to [Services > Compose > Files] in the sidebar. Press the button in the main interface to add a new Docker compose file.

Under Name put down Stirling and under File, adapt the following (making sure the number of spaces are consistent)
```
version: "3.3"  
services:  
   stirling-pdf:  
     image: frooodle/s-pdf:latest  
     ports:  
       - &lt;unused port number like 7331&gt;:8080  
     environment:  
       - DOCKER_ENABLE_SECURITY=false  
     volumes:  
       - '&lt;absolute path to shared config folder&gt;/tesseract:/usr/share/tessdata'  
       - '&lt;absolute path to shared config folder&gt;/Stirling/configs:/config'  
       - '&lt;absolute path to shared config folder&gt;/Stirling/customFiles:/customFiles'  
       - '&lt;absolute path to shared config folder&gt;/Stirling/logs:/logs'  
     restart: unless-stopped
```
Under ports:, make sure to add an unused port number (I went with 7331).

Replace <absolute path to shared config folder> with the absolute path to the config folder where you want Docker-installed applications to store their configuration information (accessible by going to [Storage > Shared Folders] in the administrative panel). You’ll notice there’s an extra line in there for tessdata — this corresponds to the stored files for the Tesseract tool that Stirling uses for OCR

Once you’re done, hit Save and you should be returned to your list of Docker compose files for the next step. Notice that the new FreshRSS entry you created has a Down status, showing the container has yet to be initialized.
To start your Stirling container, click on the new Stirling entry and press the (up) button. This will create the container, download any files needed, and run it.

And that’s it! To prove it worked, go to your-servers-static-ip-address:7331 from a browser that’s on the same network as your server (replacing 7331 if you picked a different port in the configuration above) and you should see the Stirling tools page (see below)
You can skip this step if you didn’t (as I laid out in my last post) set up Pihole and local DNS / Nginx proxy or if you don’t care about having a user-readable domain name for these PDF tools. But, assuming you do and you followed my instructions, open up WeTTy (which you can do by going to wetty.home in your browser if you followed my instructions or by going to [Services > WeTTY] from OpenMediaVault administrative panel and pressing Open UI button in the main panel) and login as the root user. Run:
```
cd /etc/nginx/conf.d  
ls
```
Pick out the file you created before for your domains and run
```
nano &lt;your file name&gt;.conf
```
This opens up the text editor nano with the file you just listed. Use your cursor to go to the very bottom of the file and add the following lines (making sure to use tabs and end each line with a semicolon)
```
server {  
    listen             80;  
    server_name        &lt;pdf.home or the domain you'd like to use&gt;;  
    location / {  
        proxy_pass     http://&lt;your-server-static-ip&gt;:&lt;PDF port number&gt;;  
    }  
}
```
And then hit Ctrl+X to exit, Y to save, and Enter to overwrite the existing file. Then in the command line run the following to restart Nginx with your new configuration loaded.
```
systemctl restart nginx
```
Now, if your server sees a request for pdf.home (or whichever domain you picked), it will direct them to the PDF tools.

Login to your Pihole administrative console (you can just go to pi.hole in a browser) and click on [Local DNS > DNS Records] from the sidebar. Under the section called Add a new domain/IP combination, fill out under Domain: the domain you just added above (i.e. pdf.home) and next to IP Address: you should add your server’s static IP address. Press the Add button and it will show up below.

To make sure it all works, enter the domain you just added (pdf.home if you went with my default) in a browser and you should see the Stirling PDF tools page.
Lastly, to make the PDF tools actually useable, you’ll want to increase the maximum allowable file upload size in OpenMediaVault’s default webserver Nginx (so that you can use the tools with PDFs larger than the incredibly tiny default minimum size of 1 MB). To do this, log back into your server using WeTTy (follow the instructions above) and run:
```
cd /etc/nginx/  
nano nginx.conf
```
This opens up the text editor nano with the master configuration file for Nginx. Use your cursor to go to some spot after http { but before the closing }. This configures how Nginx will process HTTP requests (basically anything coming from a website). Enter the two lines below (making sure to use tabs and end the second line with a semicolon; to be clear "... stuff that comes by default..." is just placeholder text that you don’t need to write or add, it’s just to show that the two lines you enter need to be inside the {})
```
http {  
    ... stuff that comes by default ...  
    ## adding larger file upload limit  
    client_max_body_size 100M;  
    ... more stuff that comes by default ...  
}
```
And then hit Ctrl+X to exit, Y to save, and Enter to overwrite the existing file. Then in the command line run the following to restart Nginx with your new configuration loaded.
```
systemctl restart nginx
```
Now, the PDF tools can handle file uploads up to 100 MB in size!
Lastly, to make full use of OCR, you’ll want to download the language files you’re most interested in from Tesseract repository (the slower but more accurate files are here and the faster but less accurate files are here; simple click on the file you’re interested in from the list and then select Download from the “three dot” menu or by hitting Ctrl+Shift+s) and place them in the /tesseract folder you mapped in the Docker compose file. To verify that those files are properly loaded, simply go to the PDF tools, select the one called OCR / Cleanup scans (or visit <URL to PDF tools>/ocr-pdf) and the language files that you’ve downloaded should show up as a checkbox.

And now, you have a handy set of PDF tools in your (home server) back pocket!

(If you’re interested in how to setup a home server on OpenMediaVault or how to self-host different services, check out all my posts on the subject)

The Opportunity in Lagging Edge Semiconductors

Tue, 20 Feb 2024 00:00:00 GMT

While much attention is (rightly) focused on the role of TSMC (and its rivals Samsung and Intel) in “leading edge” semiconductor technology, the opportunity at the so-called “lagging edge” — older semiconductor process technologies which continue to be used — is oftentimes completely ignored.

The reality of the foundry model is that fab capacity is expensive to build and so the bulk of the profit made on a given process technology investment is when it’s years old. This is a natural consequence of three things:

Very few semiconductor designers have the R&D budget or the need to be early adopters of the most advanced technologies. (That is primarily relegated to the sexiest advanced CPUs, FPGAs, and GPUs, but ignores the huge bulk of the rest of the semiconductor market)
Because only a small handful of foundries can supply “leading edge” technologies and because new technologies have a “yield ramp” (where the technology goes from low yield to higher as the foundry gets more experience), new process technologies are meaningfully more expensive.
Some products have extremely long lives and need to be supported for decade-plus (i.e. automotive, industrial, and military immediately come to mind)

As a result, it was very rational for GlobalFoundries (formerly AMD’s in-house fab) to abandon producing advanced semiconductor technologies in 2018 to focus on building a profitable business at the lagging edge. Foundries like UMC and SMIC have largely made the same choice.

This means giving up on some opportunities (those that require newer technologies) — as GlobalFoundries is finding recently in areas like communications and data center — but provided you have the service capability and capacity, can still lead to not only a profitable outcome, but one which is still incredibly important to the increasingly strategic semiconductor space.

Reading Excerpt • Anandtech

”

“When GlobalFoundries abandoned development of its 7 nm-class process technology in 2018 and refocused on specialty process technologies, it ceased pathfinding, research, and development of all technologies related to bleeding-edge sub-10nm nodes. At the time, this was the correct (and arguably only) move for the company, which was bleeding money and trailing behind both TSMC and Samsung in the bleeding-edge node race. But in the competitive fab market, that trade-off for reduced investment was going to eventually have consequences further down the road, and it looks like those consequences are finally starting to impact the company. In a recent earnings call, GlobalFoundries disclosed that some of the company’s clients are leaving for other foundries, as they adopt sub-10nm technologies faster than GlobalFoundries expected.”

GlobalFoundries: Clients Are Migrating to Sub-10nm Faster Than Expected

Anton Shilov | Anandtech

NVIDIA to make custom AI chips? Tale as old as time

Sun, 18 Feb 2024 00:00:00 GMT

Every standard products company (like NVIDIA) eventually gets lured by the prospect of gaining large volumes and high margins of a custom products business.

And every custom products business wishes they could get into standard products to cut their dependency on a small handful of customers and pursue larger volumes.

Given the above and the fact that NVIDIA did used to effectively build custom products (i.e. for game consoles and for some of its dedicated autonomous vehicle and media streamer projects) and the efforts by cloud vendors like Amazon and Microsoft to build their own Artificial Intelligence silicon it shouldn’t be a surprise to anyone that they’re pursuing this.

Or that they may eventually leave this market behind as well.

Reading Excerpt • Anandtech

”

“While using NVIDIA’s A100 and H100 processors for AI and high-performance computing (HPC) instances, major cloud service providers (CSPs) like Amazon Web Services, Google, and Microsoft are also advancing their custom processors to meet specific AI and general computing needs. This strategy enables them to cut costs as well as tailor capabilities and power consumption of their hardware to their particular needs. As a result, while NVIDIA’s AI and HPC GPUs remain indispensable for many applications, an increasing portion of workloads now run on custom-designed silicon, which means lost business opportunities for NVIDIA. This shift towards bespoke silicon solutions is widespread and the market is expanding quickly. Essentially, instead of fighting custom silicon trend, NVIDIA wants to join it.”

Report: NVIDIA Forms Custom Chip Unit for Cloud Computing and More

Anton Shilov | Anandtech

Which jobs are the most [insert gender or race]?

Fri, 16 Feb 2024 00:00:00 GMT

Fascinating data from the BLS on which jobs have the greatest share of a particular gender or race. The following two charts are from the WSJ article I linked. I never would have guessed that speech-language pathologists (women), property appraisers (white), postal service workers (black), or medical scientists (Asian) would have such a preponderance of a particular group.

Reading Excerpt • WSJ

”

“The Bureau of Labor Statistics each year publishes data looking at the gender and racial composition of hundreds of occupations, offering a snapshot of how workers sort themselves into many of the most important jobs in the country. There are sociology textbooks’ worth of explanations for these numbers. One clear conclusion: Many occupations skew heavily toward one gender or race, leading to a workforce where 96.7% of preschool and kindergarten teachers are women, two-thirds of manicurists and pedicurists are Asian, and 92.4% of pilots and flight engineers are white.”

40% of Lawyers Are Women. 7% Are Black. America\u2019s Workforce in Charts

Lauren Weber & Stephanie Stamm | WSJ

Trouble in commercial real estate

Thu, 15 Feb 2024 00:00:00 GMT

Commercial real estate (and, by extension, community banks) are in a world of hurt as hybrid/remote work, higher interest rates, and property bubbles deflating/popping collide…

Reading Excerpt • Bloomberg

”

“Many banks still prefer to work out deals with existing landlords, such as offering loan extensions in return for capital reinvestments toward building upgrades. Still, that approach may not be viable in many cases; big companies from Blackstone to a unit of Pacific Investment Management Co. have walked away from or defaulted on properties they don’t want to pour more money into. In some cases, buildings may be worth even less today than the land they sit on. “When people hand back keys, that’s not the end of it — the equity is wiped but the debt is also massively impaired,” said Dan Zwirn, CEO of asset manager Arena Investors, which invests in real estate debt. “You’re talking about getting close to land value. In certain cases people are going to start demolishing things.””

The Brutal Reality of Plunging Office Values Is Here

Natalie Wong & Patrick Clark | Bloomberg

Selfhosting FreshRSS

Mon, 15 Jan 2024 00:00:00 GMT

(Note: this is part of my ongoing series on cheaply selfhosting)

It’s been a few months since I started down the selfhosting/home server journey. Thanks to Docker, it has been relatively smooth sailing. Today, I have a cheap mini-PC based server that:

blocks ads / online trackers on all devices
stores and streams media (even for when I’m out of the house)
acts as network storage (for our devices to store and share files)
serves as a personal RSS/newsreader

The last one is new since my last post and, in the hopes that this helps others exploring what they can selfhost or who maybe have a home server and want to start deploying services, I wanted to share how I set up FreshRSS, a self-hosted RSS reader (on an OpenMediaVault v6 server)

Why a RSS Reader?

Like many who used it, I was a massive Google Reader fan. Until 2013 when it was unceremoniously shut down, it was probably the most important website I used after Gmail.

I experimented with other RSS clients over the years, but found that I did not like most commercial web-based clients (which were focused on serving ads or promoting feeds I was uninterested in) or desktop clients (which were difficult to sync between devices). So, I switched to other alternatives (i.e. Twitter) for a number of years.

FreshRSS

Wanting to return to the simpler days where I could simply follow the content I was interested in, I stumbled on the idea of self-hosting an RSS reader. Looking at the awesome-selfhosted feed reader category, I looked at the different options and chose to go with FreshRSS for a few reasons:

It had the most Github stars of any feed reader — an imperfect but reasonable sign of a well-liked project.
It has been around for over 10 years! and as a result has attracted a community of extension developers and even cloud RSS service providers — suggests the product has real stability and traction which will be helpful if things go wrong
It’s available as a Docker container on linuxserver.io — linuxserver.io is well-known for hosting and maintaining high quality and up-to-date Docker container images

Installation

To install FreshRSS on OpenMediaVault:

If you haven’t already, make sure you have OMV Extras and Docker Compose installed (refer to the section Docker and OMV-Extras in my previous post, you’ll want to follow all 10 steps as I refer to different parts of the process throughout this post) and have a static local IP address assigned to your server.
Login to your OpenMediaVault web admin panel, and then go to [Services > Compose > Files] in the sidebar. Press the button in the main interface to add a new Docker compose file.

Under Name put down FreshRSS and under File, adapt the following (making sure the number of spaces are consistent)
```
version: "2.1"  
services:  
   freshrss:  
     container_name: freshrss  
     image: lscr.io/linuxserver/freshrss:latest  
     ports:  
       - &lt;unused port number like 3777&gt;:80  
     environment:  
       - TZ: 'America/Los_Angeles'  
       - PUID=&lt;UID of Docker User&gt;  
       - PGID=&lt;GID of Docker User&gt;  
     volumes:  
       - '&lt;absolute path to shared config folder&gt;/FreshRSS:/config'  
     restart: unless-stopped
```
You’ll need to replace <UID of Docker User> and <GID of Docker User> with the UID and GID of the Docker user you created (which will be 1000 and 100 if you followed the steps I laid out, see Step 10 in the section “Docker and OMV-Extras” in my initial post)

I live in the Bay Area so I set the timezone TZ to America/Los_Angeles. You can find yours here.

Under ports:, make sure to add an unused port number (I went with 3777).

Replace <absolute path to shared config folder> with the absolute path to the config folder where you want Docker-installed applications to store their configuration information (accessible by going to [Storage > Shared Folders] in the administrative panel).

Once you’re done, hit Save and you should be returned to your list of Docker compose files for the next step. Notice that the new FreshRSS entry you created has a Down status, showing the container has yet to be initialized.
To start your FreshRSS container, click on the new FreshRSS entry and press the (up) button. This will create the container, download any files needed, and run it.

And that’s it! To prove it worked, go to your-servers-static-ip-address:3777 from a browser that’s on the same network as your server (replacing 3777 if you picked a different port in the configuration above) and you should see the FreshRSS installation page (see below)
You can skip this step if you didn’t (as I laid out in my last post) set up Pihole and local DNS / Nginx proxy or if you don’t care about having a user-readable domain name for FreshRSS. But, assuming you do and you followed my instructions, open up WeTTy (which you can do by going to wetty.home in your browser if you followed my instructions or by going to [Services > WeTTY] from OpenMediaVault administrative panel and pressing Open UI button in the main panel) and login as the root user. Run:
```
cd /etc/nginx/conf.d  
ls
```
Pick out the file you created before for your domains and run
```
nano &lt;your file name&gt;.conf
```
This opens up the text editor nano with the file you just listed. Use your cursor to go to the very bottom of the file and add the following lines (making sure to use tabs and end each line with a semicolon)
```
server {  
    listen             80;  
    server_name        &lt;rss.home or the domain you'd like to use&gt;;  
    location / {  
        proxy_pass     http://&lt;your-server-static-ip&gt;:&lt;FreshRSS port number&gt;;  
    }  
}
```
And then hit Ctrl+X to exit, Y to save, and Enter to overwrite the existing file. Then in the command line run the following to restart Nginx with your new configuration loaded.
```
systemctl restart nginx
```
Now, if your server sees a request for rss.home (or whichever domain you picked), it will direct them to FreshRSS.

Login to your Pihole administrative console (you can just go to pi.hole in a browser) and click on [Local DNS > DNS Records] from the sidebar. Under the section called Add a new domain/IP combination, fill out under Domain: the domain you just added above (i.e. rss.home) and next to IP Address: you should add your server’s static IP address. Press the Add button and it will show up below.

To make sure it all works, enter the domain you just added (rss.home if you went with my default) in a browser and you should see the FreshRSS installation page.
Completing installation is easy. Thanks to the use of Docker, all of your PHP and files will be configured accurately so you should be able to proceed with the default options. Unless you’re planning to store millions of articles served to dozens of people, the default option of SQLite as database type should be sufficient in Step 3 (see below)

This leaves the final task of configuring a username and password (and, again, unless you’re serving this to many users whom you’re worried will hack you, the default authentication method of Web form will work)

Finally, press Complete installation and you will be taken to the login page:

Advice

Once you’ve logged in with the username and password you just set, the world is your oyster. If you’ve ever used an RSS reader, the interface is pretty straightforward, but the key is to use the Subscription management button in the interface to add RSS feeds and categories as you see fit. FreshRSS will, on a regular basis, look for new content from those feeds and put it in the main interface. You can then step through and stay up to date on the sites that matter to you. There are a lot more features you can learn about from the FreshRSS documentation.

On my end, I’d recommend a few things:

How to find the RSS feed for a page — Many (but not all) blog/news pages have RSS feeds. The most reliable way to find it is to right click on the page you’re interested in from your browser and select View source (on Chrome you’d hit Ctrl+U). Hit Ctrl+F to trigger a search and look for rss. If there is an RSS feed, you’ll see something that says "application/rss+xml" and near it will usually be a URL that ends in /rss or /feed or something like that (my blog, for instance, hosted on benjamintseng.com has a feed at benjamintseng.com/rss).
- Once you open up the feed,
Learn the keyboard shortcuts — they’re largely the same as found on Gmail (and the old Google Reader) but they make using this much faster:
- j to go to the next article
- k to go to the previous article
- r to toggle if something is read or not
- v to open up the original page in a new tab
Use the normal view, sorted oldest first — (you do this by tapping the Settings gear in the upper-right of the interface and then selecting Reading under Configuration in the menu). Even though I’ve aggressively curated the feeds I subscribe to, there is a lot of material and the “normal view” allows me to quickly browse headlines to see which ones are more worth my time at a glance. I can also use my mouse to selectively mark somethings as read so I can take a quick Inbox Zero style approach to my feeds. This allows me to think of the j shortcut as “move forward in time” and the k shortcut as “move backwards” and I can use the pulldown menu next to Mark as read button to mark content older than one day / one week as read if I get overwhelmed.
Subscribe to good feeds — probably a given, but here are a few I follow to get you started:
- xkcd (feed URL): classic webcomic
- Saturday Morning Breakfast Cereal (feed URL): another great webcomic
- Paul Graham’s Essays (feed URL): love or hate him, Paul Graham does some great work writing very simple but thoughtful pieces
- Collaborative Fund blog (feed URL): Morgan Housel is one of my favorite writers and he is responsible for most of the posts here
- Benjamin Tseng’s blog (feed URL): you’re already here, aren’t you? 😇 You can also subscribe via email. I write about interesting things I’m reading, how-to guides like this, and my thoughts on tech / science / finance

I hope this helps you get started!

(If you’re interested in how to setup a home server on OpenMediaVault or how to self-host different services, check out all my posts on the subject)

Stocks for the Long Run? Maybe not all the Time

Sun, 14 Jan 2024 00:00:00 GMT

One of the core assumptions of modern financial planning and finance is that stocks have better returns over the long-run than bonds.

The reason “seems” obvious: stocks are riskier. There is, after all, a greater chance of going to zero since bond investors come before stock investors in a legal line to get paid out after a company fails. Furthermore, stocks let an investor participate in the upside (if a company grows rapidly) whereas bonds limits your upside to the interest payments.

A fascinating article by Santa Clara University Professor Edward McQuarrie published in late 2023 in Financial Analysts Journal puts that entire foundation into doubt. McQuarrie collects a tremendous amount of data to compute total US stock and bond returns going back to 1792 using newly available historical records and data from periodicals from that timeframe. The result is a lot more data including:

coverage of bonds and stocks traded outside of New York
coverage of companies which failed (such as The Second Bank of the United States which, at one point, was ~30% of total US market capitalization and unceremoniously failed after its charter was not renewed)
includes data on dividends (which were omitted in many prior studies)
calculates results on a capitalization-weighted basis (as opposed to price-weighted / equal-weighted which is easier to do but less accurately conveys returns investors actually see)

The data is fascinating, as it shows that, contrary to the opinion of most “financial experts” today, it is not true that stocks always beat bonds in the long-run. In fact, much better performance for stocks in the US seems to be mainly a 1940s-1980s phenomena (see Figure 1 from the paper below)

Stock and bond performance (normalized to $1 in 1792, and renormalized in 1982) on a logarithmic scale
Source: Figure 1, McQuarrie et al

Put another way, if you had looked at stocks vs bonds in 1862, the sensible thing to tell someone was “well, some years stocks do better, some years bonds do better, but over the long haul, it seems bonds do better (see Table 1 from the paper below).

The exact opposite of what you would tell them today / having only looked at the post-War world.

Source: Table 1, McQuarrie et al

This problem is compounded if you look at non-US stock returns where, even after excluding select stock market performance periods due to war (i.e. Germany and Japan following World War II), focusing even on the last 5 decades shows comparable performance for non-US stocks as non-US government bonds.

Even assumptions viewed as sacred, like how stocks and bonds can balance each other out because their returns are poorly correlated, shows huge variation over history — with the two assets being highly correlated pre-Great Depression, but much less so (and swinging wildly) afterwards (see Figure 6 below)

Stock and Bond Correlation over Time
Source: Figure 6, McQuarrie et al

Now neither I nor the paper’s author are suggesting you change your fundamental investment strategy as you plan for the long-term (I, for one, intend to continue allocating a significant fraction of my family’s assets to stocks for now).

But, beyond some wild theorizing on why these changes have occurred throughout history, what this has reminded me is that the future can be wildly unknowable. Things can work one way and then suddenly stop. As McQuarrie pointed out recently in a response to a Morningstar commenter, “The rate of death from disease and epidemics stayed at a relatively high and constant level from 1793 to 1920. Then advances in modern medicine fundamentally and permanently altered the trajectory … or so it seemed until COVID-19 hit in February 2020.”

Reading Excerpt • *Financial Analysts Journal*

”

“If stocks are risky, investors will demand a premium to invest. But if stocks cease to be risky once held for a long enough period—if stocks are certain to have strong returns after 20 years and certain to outperform bonds—then investors have no reason to expect a premium over these longer periods, given that no shortfall risk had to be assumed. The expanded historical record shows that stocks can perform poorly in absolute terms and underperform bonds, whether the holding period is 20, 30, 50, or 100 years. That documentation of risk resolves the conundrum.”

Stocks for the Long Run? Sometimes Yes, Sometimes No

Edward F. McQuarrie | *Financial Analysts Journal*

Source: ASH 2023 abstract, Visram et al.

InVision founder retro

Tue, 09 Jan 2024 00:00:00 GMT

As reported in The Information a few days ago, former design tool giant InVision, once valued at $2 billion, is shutting down at the end of this year.

While much of the commentary has been about Figma’s rapid rise and InVision’s inability to respond, I saw this post on Twitter/X from one of InVision’s founders Clark Valberg about what happened. The screenshotted message he left is well-worth a read. It is a great (if slightly self-serving / biased) retrospective.

As someone who was a mere bystander during the events (as a newly minted Product Manager working with designers), it felt very true to the moment.

I remember being blown away by how the entire product design community moved to Sketch (from largely Adobe-based solutions) and then, seemingly overnight, from Sketch to Figma.

While it’s fair to criticize the leadership for not seeing web-based design as a place to invest, I think the piece just highlights how because it wasn’t a direct competitor to InDesign (but to Sketch & Adobe XD) and because the idea of web-based wasn’t on anyone’s radar at the time, it became a lethal blind spot for the company. It’s Tech Strategy 101 and perfectly highlights Andy Grove’s old saying: “(in technology,) only the paranoid survive”.

The only 3 things a startup CEO needs to master

Thu, 28 Dec 2023 00:00:00 GMT

So, you watched Silicon Valley and read some articles on Techcrunch and you envision yourself as a startup CEO 🤑. What does it take to succeed? Great engineering skills? Salesmanship? Financial acumen?

As someone who has been on both sides of the table (as a venture investor and on multiple startup executive leadership teams), there are three — and only three — things a startup CEO needs to master. In order of importance:

Raise Money from Investors (now and in the future): The single most important job of a startup CEO is to secure funding from investors. Funding is the lifeblood of a company, and raising it is a job that only the CEO can drive. Not being great at it means slower growth / fewer resources, regardless of how brilliant you are, or how great your vision. Being good at raising money buys you a lot of buffer in every other area.
Hire Amazing People into the Right Roles (and retain them!): No startup, no matter how brilliant the CEO, succeeds without a team. Thus, recruiting the right people into the right positions is the second most vital job of a CEO. Without the right people in place, your plans are not worth the paper on which they are written. Even if you have the right people, if they are not entrusted with the right responsibilities or they are unhappy, the wrong outcomes will occur. There is a reason that when CEOs meet to trade notes, they oftentimes trade recruiting tips.
Inspire the Team During Tough Times: Every startup inevitably encounters stormy seas. It could be a recession causing a slowdown, a botched product launch, a failed partnership, or the departure of key employees. During these challenging times, the CEO’s job is to serve as chief motivator. Teams that can resiliently bounce back after crises can stand a better chance of surviving until things turn a corner.

It’s a short list. And it doesn’t include:

deep technical expertise
an encyclopedic knowledge of your industry
financial / accounting skills
marketing wizardry
design talent
intellectual property / legal acumen

It’s not that those skills aren’t important for building a successful company — they are. It’s not even that these skills aren’t helpful for a would-be startup CEO — these skills would be valuable for anyone working at a startup to have. For startup CEOs in particular, these skills can help sell investors as to why the CEO is the right one to invest in or convince talent to join or inspire the team that the strategy a CEO has chosen is the right one.

But, the reality is that these skills can be hired into the company. They are not what separates great startup CEOs from the rest of the pack.

What makes a startup CEO great is their ability to nail the jobs that cannot be delegated. And that boils down to fundraising, hiring and retaining the best, and lifting spirits when things are tough. And that is the job.

After all, startup investors write checks because they believe in the vision and leadership of a CEO, not a lackey. And startup employees expect to work for a CEO with a vision, not just a mouthpiece.

So, want to become a startup CEO? Work on:

Storytelling — Learn how to tell stories that compel listeners. This is vital for fundraising (convincing investors to take a chance on you because of your vision), but also for recruiting & retaining people as well as inspiring a team during difficult times.
Reading People — Learn how to accurately read people. You can’t hire a superstar employee with other options, retain an unhappy worker through tough times, or overcome an investor’s concerns unless you understand their position. This means being attentive to what they tell you directly (i.e., over email, text, phone / video call, or in person, etc.) as well as paying attention to what they don’t (i.e., body language, how they act, what topics they discussed vs. didn’t, etc.).
Prioritization — Many startup CEOs got to where they are because they were superstars at one or more of the “unnecessary to be a great startup CEO” skills. But, continuing to focus on that skill and ignoring the skills that a startup CEO needs to be stellar at confuses the path to the starting point with the path to the finish line. It is the CEO’s job to prioritize those tasks that they cannot delegate and to ruthlessly delegate everything else.

Real World Effectiveness vs. Clinical Trial Efficacy

Tue, 19 Dec 2023 00:00:00 GMT

Randomized controlled trials (RCTs) are the “gold standard” in healthcare for proving a treatment works. And for good reason. A well-designed and well-powered (i.e., large enough) clinical trial establishes what is really due to a treatment as opposed to another factor (e.g., luck, reversion to the mean, patient selection, etc.), and it’s a good thing that drug regulation is tied to successful trial results.

But, there’s one wrinkle. Randomized controlled trials are not reality.

RCTs are tightly controlled, where only specific patients (those fulfilling specific “inclusion criteria”) are allowed to participate. Follow-up is organized and adherence to protocol is tightly tracked. Typically, related medical care is also provided free of cost.

This is exactly what you want from a scientific and patient volunteer safety perspective, but, as we all know, the real world is messier. In the real world:

Physicians prescribe treatments to patients that don’t have to fit the exact inclusion criteria of the clinical trial. After all, many clinical trials exclude people who are extremely sick or who are children or pregnant.
Patients may not take their designated treatment on time or in the right dose … and nobody finds out.
Follow-up on side effects and progress is oftentimes random
Cost and free time considerations may change how and when a patient comes in
Physicians also have greater choice in the real world. They only prescribe treatments they think will work, whereas in a RCT, you get the treatment you’ve been randomly assigned to.

These differences beg the question: just how different is the real world from an randomized controlled trial?

A group in Canada studied this question and presented their findings at the recent ASH (American Society of Hematology) meeting. The researchers looked at ~4,000 patients in Canada with multiple myeloma, a cancer with multiple treatment regimens that have been developed and approved, and used Canada’s national administrative database to track how they did after 7 different treatment regimes and compared it to published RCT results for each treatment.

The findings are eye-opening. While there is big variation from treatment to treatment, in general, real world effectiveness was significantly worse, by a wide margin, than efficacy published in randomized controlled trial (see table below).

While the safety profiles (as measured by the rate of “adverse events”) seemed similar between real world and RCT, real world patients did, in aggregate, 44% worse on progression free survival and 75% worse on overall survival when compared with their RCT counterparts!

The only treatment where the real world did better than the RCT was in a study where it’s likely the trial volunteers were much sicker than on average. (Note: that one of seven treatment regimes went the other way but the aggregate still is 40%+ worse shows you that some of the comparisons were vastly worse)

The lesson here is not that we should stop doing or listening to randomized controlled trials. After all, this study shows that they were reasonably good at predicting safety, not to mention that they continue to be our only real tool for establishing whether a treatment has real clinical value prior to giving it to the general public.

But this study imparts two key lessons for healthcare:

Do not assume that the results you see in a clinical trial are what you will see in the real world. Different patient populations, resources, treatment adherence, and many other factors will impact what you see.
Especially for treatments we expect to use with many people, real world monitoring studies are valuable in helping to calibrate expectations and, potentially, identify patient populations where a treatment is better or worse suited.

Public EV chargers suck

Sun, 17 Dec 2023 00:00:00 GMT

We have a Nissan Ariya and currently DON’T have a home charger (yet — waiting on solar which is another boondoggle for another post). As we live in a town with abundant EVGo chargers (and the Ariya came with 1 yr of free EVGo charging), we thought we could manage.

When it works, its amazing. But it doesn’t … a frustrating proportion of the time. And, as a result, we’ve become oddly superstitious about which chargers we go to and when.

I’m glad the charging companies are aware and are trying to address the problem. As someone who’s had to ship and support product, I also recognize that creating charging infrastructure in all kinds of settings which need to handle all kinds of electric vehicles is not trivial.

But, it’s damn frustrating to not be able to count on these (rest assured, we will be installing our own home charger soon), so I do hope that future Federal monies will have strict uptime requirements and penalties. Absent this, vehicle electrification becomes incredibly difficult outside of the surburban homeowner market.

Reading Excerpt • Canary Media

”

“J.D. Power reported in August that 20 percent of all non-Tesla EV drivers in its most recent study said they visited a charger but did not charge their vehicle, whether because the charger was inoperable or because of long wait times to use it, up from 15 percent in the first quarter of 2021. Fear of inadequate public charging has now overtaken “range anxiety” as the chief concern about EVs among the car-buying public, according to J.D. Power. “Although the majority of EV charging occurs at home” — about 80 percent of it, according to industry data — “public charging needs to provide a much better experience across the board, not just for the users of today, but also to alleviate the concerns of skeptical future customers,” said Brent Gruber, executive director of J.D. Power’s global automotive practice.”

EV chargers have a big reliability problem. Can the government fix it?

Jeff St. John | Canary Media

Will Hong Kong Put the Fear Into China’s Property Builders?

Tue, 28 Nov 2023 00:00:00 GMT

The collapse of China’s massive property bubble is under way and it is wreaking havoc as significant amounts of the debt raised by Chinese property builders is from offshore investors.

Because of (well-founded) concerns on how Chinese Mainland courts would treat foreign concerns, most of these agreements have historically been conducted under Hong Kong law. As a result, foreign creditors have (understandably) hauled their deadbeat Chinese property builder debtors to court there.

While the judgements (especially from Linda Chan, the subject of this Bloomberg article) are unsurprisingly against the Chinese property builders (who have been slow to release credible debt restructuring plans), the big question remains whether the Mainland Chinese government will actually enforce these rulings. It certainly would make life harder on (at least until recently very well-connected) Chinese property builders at a moment of weakness in the sector.

But, failure to do so would also hurt the Chinese government’s goal of encouraging more foreign investment: after all, why would you invest in a country where you can’t trust the legal paper?

Reading Excerpt • Bloomberg

”

“Never before has there been such a wave of Chinese corporate defaults on bonds sold to foreign investors. And never in recent memory has a bankruptcy judge in Hong Kong, the de-facto home for such cases, earned a reputation for holding deadbeat companies to account quite like Chan. Chan, 54, has displayed an unwavering determination to give creditors a fair shot at recouping as much of their money as they can. One morning in early May, she shocked the packed courtroom by suddenly ordering the liquidation of Jiayuan. She had peppered the company’s lawyers that day as they tried, unsuccessfully, to explain why they needed more time to iron out their debt restructuring proposal. And then, late last month, Chan put lawyers for Evergrande, the most indebted developer of them all, on notice: Either turn over a concrete restructuring proposal in five weeks or face the same fate as Jiayuan.”

The Hong Kong Judge Who Puts Fear Into China\u2019s Deadbeat Builders

Dorothy Ma and Alice Huang | Bloomberg

Amazon Delivers More US Packages than UPS and Fedex

Mon, 27 Nov 2023 00:00:00 GMT

It’s both unsurprising but also astonishing at the same time.

Going from Formula One to Odd One Out

Mon, 20 Nov 2023 00:00:00 GMT

Market phase transitions have a tendency to be incredibly disruptive to market participants. A company or market segment used to be the “alpha wolf” can suddenly find themselves an outsider in a short time. Look at how quickly Research in Motion (makers of the Blackberry) went from industry darling to laggard after Apple’s iPhone transformed the phone market.

Something similar is happening in the high performance computing (HPC) world (colloquially known as supercomputers). Built to do the highly complex calculations needed to simulate complex physical phenomena, HPC was, for years, the “Formula One” of the computing world. New memory, networking, and processor technologies oftentimes got their start in HPC, as it was the application that was most in need of pushing the edge (and had the cash to spend on exotic new hardware to do it).

The use of GPUs (graphical processing units) outside of games, for example, was a HPC calling card. NVIDIA’s CUDA framework which has helped give it such a lead in the AI semiconductor race was originally built to accelerate the types of computations that HPC could benefit from.

The success of Deep Learning as the chosen approach for AI benefited greatly from this initial work in HPC, as the math required to make deep learning worked was similar enough that existing GPUs and programming frameworks could be adapted. And, as a result, HPC benefited as well, as more interest and investment flowed into the space.

But, we’re now seeing a market transition. Unlike with HPC which performs mathematical operations requiring every last iota of precision on mostly dense matrices, AI inference works on sparse matrices and does not require much precision at all. This has resulted in a shift in industry away from software and hardware that works for both HPC and AI and towards the much larger AI market specifically.

Couple that with the recent semiconductor shortage (making it harder and more expensive to build HPC system with the latest GPUs) and the fact that research suggests some HPC calculations are more efficiently simulated with AI methods than actually run (in the same way that NVIDIA now uses AI to take a game rendered at a lower resolution and simulate what it would look like at a higher resolution more effectively than actually rendering the game at a higher resolution natively), I think we’re beginning to see traditional HPC shift from “Formula One of computing” to increasingly the “odd one out”.

Reading Excerpt • The Next Platform

”

“The HPC community is used to being first, and we always considered ourselves as the F1 racing team of computing. We invent the turbochargers and fuel injection and the carbon fiber and then we put that into more general purpose vehicles, to use an analogy. I worry that the HPC community has sort of taken the backseat when it comes to AI and is not leading the charge. Like you, I’m seeing a lot of this AI stuff being led out of the hyperscalers and clouds. And we’ve got to find a way to take that back and carve our own use cases. There are a lot more HPC sites around the world than there are cloud sites, and we have got access to all a lot of data.”

Trying to Do More Real HPC in an Increasingly AI World

Timothy Prickett Morgan | The Next Platform

Are Driverless Cars Safer? (This time with data)

Sun, 19 Nov 2023 00:00:00 GMT

I’m over two months late to seeing this study, but a brilliant study design (use insurance data to measure rate of bodily injury and property damage) and strong, noteworthy conclusion (doesn’t matter how you cut it, Waymo’s autonomous vehicle service resulted in fewer injuries per mile and less property damage per mile than human drivers in the same area) make this worthwhile to return to! Short and sweet paper from researchers from Waymo, Swiss Re (the re-insurer), and Stanford that is well worth the 10 minute read!

AI for Defense

Mon, 13 Nov 2023 00:00:00 GMT

My good friend Danny Goodman (and Co-Founder at Swarm Aero) recently wrote a great essay on how AI can help with America’s defense. He outlines 3 opportunities:

“Affordable mass”: Balancing/augmenting America’s historical strategy of pursuing only extremely expensive, long-lived “exquisite” assets (e.g. F-35’s, aircraft carriers) with autonomous and lower cost units which can safely increase sensor capability &, if it comes to it, serve as alternative targets to help safeguard human operators
Smarter war planning: Leveraging modeling & simulation to devise better tactics and strategies (think AlphaCraft on steroids)
Smarter procurement: Using AI to evaluate how programs and budget line items will actually impact America’s defensive capabilities to provide objectivity in budgeting

Why Childcare is Economic Infrastructure

Mon, 06 Nov 2023 00:00:00 GMT

As a parent myself, few things throw off my work day as much as a wrench in my childcare — like a kid being sick and needing to come home or a school/childcare center being closed for the day. The time required to change plans while balancing work, the desire to check-in on your child throughout the work day to make sure they’re doing okay… and this is as someone with a fair amount of work flexibility, a spouse who also has flexibility, and nearby family who can pitch in.

Childcare, while expensive, is a vital piece of the infrastructure that makes my and my spouse’s careers possible — and hence the (hopefully positive 😇) economic impact we have possible. It’s made me very sympathetic to the notion that we need to take childcare policy much more seriously — something that I think played out for millions of households when COVID disrupted schooling and childcare plans.

The Washington Post’s Catherine Rampell also lays this out clearly in an Opinion piece, tracking how the closure of one Wisconsin day-care had cascading impacts on the affected parents and then their employers.

Reading Excerpt • Washington Post

”

“Census data suggest that, as things are, the child-care industry nationwide has been operating in the red for two straight years. Now, as programs still stressed by the pandemic lose a major source of public funds, many programs around the country are considering closure. When these businesses do shut down, they can send shock waves throughout their local economies. The shuttered child-care business sheds jobs; parents that relied on that business lose care arrangements for their kids, which in turn disrupts parents’ ability to work; and the employers of those parents must then scramble to adjust for lost workforce hours. While each of those can feel like an individual misfortune, they are all part of a larger system of how our country cares for our young while adults work — or fails to do so. And the ripple effects can be enormous. Here’s one story of what happened downstream when a single day-care center in Wisconsin shut its doors.”

Opinion | What happened to this Wisconsin day-care should concern us all

Catherine Rampell | Washington Post

Good Windows on ARM at last?

Wed, 01 Nov 2023 00:00:00 GMT

Silicon nerd 🤓 that I am, I have gone through multiple cycles of excited-then-disappointed for Windows-on-ARM, especially considering the success of ChromeOS with ARM, the Apple M1/M2 (Apple’s own ARM silicon which now powers its laptops), and AWS Graviton (Amazon’s own ARM chip for its cloud computing services).

I may just be setting myself up for disappointment here but these (admittedly vendor-provided) specs for their new Snapdragon X (based on technology they acquired from Nuvia and are currently being sued for by ARM) look very impressive. Biased as they may be, the fact that these chips are performing in the same performance range as Intel/AMD/Apple’s silicon on single-threaded benchmarks (not to mention the multi-threaded applications which work well with the Snapdragon X’s 12 cores) hopefully bodes well for the state of CPU competition in the PC market!

Single-threaded CPU performance (Config A is a high performance tuned offering, Config B is a “thin & light” configuration)

Multi-threaded CPU performance (Config A is a high performance tuned offering, Config B is a “thin & light” configuration)

Reading Excerpt • Anandtech

”

“Overall, Qualcomm’s early benchmark disclosure offers an interesting first look at what to expect from their forthcoming laptop SoC. While the competitive performance comparisons are poorly-timed given that next-generation hardware is just around the corner from most of Qualcomm’s rivals, the fact that we’re talking about the Snapdragon X Elite in the same breath as the M2 or Raptor Lake is a major achievement for Qualcomm. Coming from the lackluster Snapdragon 8cx SoCs, which simply couldn’t compete on performance, the Snapdragon X Elite is clearly going to be a big step up in virtually every way.”

Qualcomm Snapdragon X Elite Performance Preview: A First Look at What\u2019s to Come

Ryan Smith | Anandtech

Complex operations for gene editing therapies

Mon, 30 Oct 2023 00:00:00 GMT

Gene editing makes possible new therapies and actual cures (not just treatments) that were previously not. But, one thing that doesn’t get discussed a great deal is how these new gene editing-based therapies throw the “take two and call me in the morning” model out the window.

This interesting piece in BioPharmaDive gives a fascinating look at all the steps for a gene editing therapy for sickle cell disease that Vertex Pharmaceuticals is awaiting FDA approval for. The steps include:

referral by hematologist (not to mention insurance approval!)
collection of cells (probably via bone marrow extraction)
(partial) myeloablation of the patient
shipping the cells to a manufacturing facility
manufacturing facility applies gene editing on the cells
shipping of cells back
infusion of the gene edited cells to the patient (so they hopefully engraft back in their bone marrow)

Each step is complicated and has their own set of risks. And, while there are many economic aspects of this that are similar to more traditional drug regimens (high price points, deep biological understanding of disease, complicated manufacturing [esp for biologicals], medical / insurance outreach, patient education, etc.), gene editing-based therapies (which can also include CAR-T therapy) now require a level of ongoing operational complexity that the biotech/pharmaceutical industries will need to adapt to if we want to bring these therapies to more people.

Reading Excerpt • BioPharma Dive

”

“To make and administer the therapy is laborious, first requiring a referral from a hematologist. If the patient is eligible, their cells are collected and shipped to a manufacturing facility where they’re genetically edited to express a form of an essential protein called hemoglobin. The cells are then shipped back to a treatment facility that infuses them into the patient’s bone marrow. But to make sure there’s enough room for these new cells, patients first undergo myeloablation — a chemotherapy regimen that can be very difficult on their bodies and comes with the risk of infertility. Older patients may not be healthy enough to receive this treatment. “This is an extensive and expensive process,” Arbuckle said.”

Vertex gearing up for launch as sickle cell therapy review advances

Jacob Bell | BioPharma Dive

The World Runs on Excel… and its Mistakes

Mon, 23 Oct 2023 00:00:00 GMT

The 2022 CHIPS and Science Act earmarked hundreds of billions in subsidies and tax credits to bolster a U.S. domestic semiconductor (and especially semiconductor manufacturing) industry. If it works, it will dramatically reposition the U.S. in the global semiconductor value chain (especially relative to China).

With such large amounts of taxpayer money practically “gifted” to large (already very profitable) corporations like Intel, the U.S. taxpayer can reasonably assume that these funds should be allocated carefully and thoughtfully and with processes in place to make sure every penny furthered the U.S.’s strategic goals.

But, when the world’s financial decisions are powered by Excel spreadsheets, even the best laid plans can go awry.

The team behind the startup Rowsie created a large language model (LLM)-powered tool which can understand Excel spreadsheets and answer questions posed to it. They downloaded a spreadsheet that the US government provided as an example of the information and calculations they want applicants fill out in order to qualify. They then applied their AI tool to the spreadsheet to understand it’s structure and formulas.

Interestingly, Rowsie was able to find a single-cell spreadsheet error (see images below) which resulted in a $178 million understatement of interest payments!

The Control Panel tab in the Example Pre-App-Simple-Financial-Model spreadsheet from the CHIPS Act funding application website. Notice cells C51 and D51 corresponding to the senior debt interest rate (5%) and the subordinated debt interest rate (8%)

The Assumptions Processing tab in the Example Pre-App-Simple-Financial-Model spreadsheet from the CHIPS Act funding application website. Notice row 50. Despite the section being about Subordinated Debt (see Cell B50), they’re using cell C51 from the Control Panel tab (which points to the Senior Debt rate of 5%) rather than the correct cell of D51 (which points to the Subordinated Debt rate of 8%).

To be clear, this is not a criticism of the spreadsheet’s architects. In this case, what seems to have happened, is that the spreadsheet creator copied an earlier row (row 40) and forgot to edit the formula to account for the fact that row 50 is about subordinated debt and row 40 is about senior debt. It’s a familiar story to anyone who’s ever been tasked with doing something complicated in Excel. Features like copy and paste and complex formulas are very powerful, but also make it very easy for a small mistake to cascade. It’s also remarkably hard to catch!

Hopefully the Department of Commerce catches on and fixes this little clerical mishap, and that applicants are submitting good spreadsheets, free of errors. But, this case underscores how (1) so many of the world’s financial and policy decisions rest on Excel spreadsheets and you just have to hope 🤞🏻 no large mistakes were made, and (2) the potential for tools like Rowsie to be tireless proofreaders and assistants who can help us avoid mistakes and understand those critical spreadsheets quickly.

If you’re interested in checking out Rowsie, check it out at <https://www.rowsie.ai/>!

DISCLAIMER: I happen to be friends with the founders of Rowsie which is how I found out about this

The Problem with the MCU

Fri, 13 Oct 2023 00:00:00 GMT

Something is wrong with the state of the Marvel Cinematic Universe (MCU).

In 2019, Disney/Marvel topped off an amazing decade-plus run of films with Avengers: Endgame, becoming (until Avatar was re-released in China) the highest grossing film of all time. This was in spite of an objectively complicated plot which required a deep understanding of all of Marvel Cinematic Universe continuity to follow.

And yet critics and fans (myself included! 🙋🏻‍♂️) loved it! It seemed like Marvel could do no wrong.

It doesn’t feel that way anymore. While I’ve personally enjoyed Black Panther: Wakanda Forever and Shang-Chi, this Time article does a good job of critiquing how complicated the MCU has become, so much so that a layperson can’t just watch one casually.

But it misses one additional thing which I think gets to the heart of why the MCU just doesn’t feel right anymore. The MCU is now so commercially large, that the scripts feel like they’re written by a committee of businesspeople (oh make sure you’re setting up this other show/movie! let’s get in an action scene with some kind of viral quip!) rather than writers/directors trying to tell an entertaining story for the sake of the story.

And until they get to that, I’m not sure even Marvel’s plans to cut down on the number of productions will deliver.

The Parents Trying to Pass Down a Language They Hardly Speak

Mon, 09 Oct 2023 00:00:00 GMT

This article resonates with me on so many levels: both as the child who came to the US and saw his language skills deteriorate as he assimilated and as the parent trying to preserve his kids’ connection to their cultural heritage

Pixel’s Parade of AI

Sat, 07 Oct 2023 00:00:00 GMT

I am a big Google Pixel fan, being an owner and user of multiple Google Pixel line products. As a result, I tuned in to the recent MadeByGoogle stream. While it was hard not to be impressed with the demonstrations of Google’s AI prowess, I couldn’t help but be a little baffled…

What was the point of making everything AI-related?

Given how low Pixel’s market share is in the smartphone market, you’d think the focus ought to be on explaining why “normies” should buy the phone or find the price tag compelling, but instead every feature had to tie back to AI in some way.

Don’t get me wrong, AI is a compelling enabler of new technologies. Some of the call and photo functionalities are amazing, both as technological demonstrations but also in terms of pure utility for the user.

But, every product person learns early that customers care less about how something gets done and more about whether the product does what they want it too. And, as someone who very much wants a meaningful rival to Apple and Samsung, I hope Google doesn’t forget that either.

Artificial Wombs

Mon, 02 Oct 2023 00:00:00 GMT

Anyone who’s ever seen a NICU or statistics on the health outcomes for extremely premature babies understands how artificial womb technology could be incredibly impactful.

At the same time, it poses some tricky ethical challenges:

How can you get informed, ethical consent to trial this? It’s nigh impossible to predict who will need a premature delivery.
In places like the US with limitations / controversy on reproductive rights, how do you push this forward without impacting assessments of “fetal viability” that may impact the abortion debate?

Great piece in Nature News below 👇🏻

Reading Excerpt • **[start soon. Here\u2019s what you need to know](https://www.nature.com/articles/d41586-023-02901-1)**

”

“Now, the researchers at CHOP are seeking approval for the first human clinical trials of the device they’ve been testing, named the Extra-uterine Environment for Newborn Development, or EXTEND. The team has emphasized that the technology is not intended — or able — to support development from conception to birth. Instead, the scientists hope that simulating some elements of a natural womb will increase survival and improve outcomes for extremely premature babies. In humans, that’s anything earlier than 28 weeks of gestation — less than 70% of the way to full term, which is typically between 37 and 40 weeks. The CHOP group has made bold predictions about the technology’s potential. In another 2017 video describing the project, Alan Flake, a fetal surgeon at CHOP who has been leading the effort, said: “If it’s as successful as we think it can be, ultimately, the majority of pregnancies that are predicted at-risk for extreme prematurity would be delivered early onto our system rather than being delivered premature onto a ventilator.” In 2019, several members of the CHOP team joined a start-up company, Vitara Biomedical in Philadelphia, which has since raised US$100 million to develop EXTEND. (Flake declined to comment for this article, citing “conflicts of interest” and “restrictions on proprietary information”. His co-authors on the 2017 paper did not respond to Nature’s request for comment.)”

Human trials of artificial wombs could

**[start soon. Here\u2019s what you need to know](https://www.nature.com/articles/d41586-023-02901-1)**

The “Large Vision Model” (LVM) Era is Upon Us

Thu, 28 Sep 2023 00:00:00 GMT

Unless you’ve been under a rock, you’ll know the tech industry has been rocked by the rapid advance in performance by large language models (LLMs) such as ChatGPT. By adapting self-supervised learning methods, LLMs “learn” to sound like a human being by learning how to fill in gaps in language and, by doing so, become remarkably adept at solving not just language problems but understanding & creativity.

Interestingly, the same is happening in imaging, as models largely trained to fill in “gaps” in images are becoming amazingly adept. A friend of mine, Pearse Keane’s group at University College of London, for instance, just published a model trained using self-supervised learning methods on ophthalmological images which is capable of not only diagnosing diabetic retinopathy and glaucoma relatively accurately, but relatively good at predicting cardiovascular events and Parkinson’s.

At a talk, Andrew Ng captured it well, by pointing out the parallels between the advances in language modeling that happened after the seminal Transformer paper and what is happening in the “large vision model” world with this great illustration.

From Andrew Ng (Image credit: EETimes)

I want your market and you to pay for it

Mon, 25 Sep 2023 00:00:00 GMT

I have followed TSMC very closely since I started my career in the semiconductor industry. A brilliant combination of bold business bet (by founder Morris Chang), industry tailwinds (with the rise of fabless semiconductor model), forward-thinking from the Taiwanese government (who helped launch TSMC), and technological progress, it’s been fascinating to see the company enter the public consciousness.

In hearing about TSMC’s investment in the very aptly-named ESMC (European Semiconductor Manufacturing Company), I can’t help but think this is another brilliant TSMC-esque play. TSMC gets:

Guarantee outsized market share in leading edge semiconductor technology in Europe
Paid for in part by some of their largest customers (Infineon, Bosch, and NXP) who will likely commit / guarantee some of their volumes to fill this new manufacturing facility
AND (likely) additional subsidies / policy support from the European Union government (who increasingly doesn’t want to be left out of advanced chip manufacturing given Asia’s current dominance and the US’s Inflation Reduction Act push)

TSMC has managed to turn what could have been a disaster for them (growing nationalism in semiconductor manufacturing) into a subsidized, volume-committed factory.

The cable bundle of the future

Sat, 23 Sep 2023 00:00:00 GMT

Charter and Disney recently made peace over the recent ESPN carriage fee dispute.

Three things are happening in video delivery world that are colliding here:

People are “cutting the cord” as they become less dependent on cable for high quality content (due to things like YouTube and Netflix)
Because you’ve lost the “cable bundle” economics (where cable subscribers would cross-subsidize each other’s viewing — you pay because you really want ESPN & I pay because I really want HBO and, as a result, we both end up paying less for more content), video streaming services like Disney+ inevitably increase prices & introduce ad models to cover their (very high) cost (of content production). This naturally means new bundles will emerge as consumers look to find ways to pay less for more content.
High speed internet today is largely subsidized by the investments from cable industry to deliver video. If ‘cord cutting’ (as in canceling cable) continues, then eventually the cost of high speed internet will go up as it becomes the “main event” for the company’s financials. Given (2), I think this likely means “cable companies” will increasingly become “bundled internet + streaming service” companies soon.

All this is ironically not that different from the original cable bundle, only this time we have a few new logos (i.e. Netflix) and a little more price transparency since you can see what the unsubsidized streaming video service cost (i.e. Disney+, Hulu, etc.) would be outside of the bundle.

Yummy AND durable 🍅

Wed, 20 Sep 2023 00:00:00 GMT

One reason much of the modern produce we buy tastes so bland is because our agricultural system has bred modern varieties for ship-ability and the ability of produce to be picked by machine, rather than flavor.

While this has expanded the access to produce (both geographically but also in terms of cost due to the ability to use automation), it’s meant that consumers often have to choose between shelf life and good taste.

But, advances in plant genetics could change that. If we could understand the genes that are responsible for durability, that could inform how we breed or gene-edit varieties that can combine desirable taste attributes with durability.

Researchers in China published a paper in Nature Plants identifying the gene (fs8.1) responsible for making roma tomatoes elongated and crush-resistant enough to be machine-harvestable and even demonstrated that it would work in alternative varieties without changing their taste.

Both the paper and the Science article on it are worth checking out.

UAW Strike vs. the Detroit Three EV Transition

Mon, 18 Sep 2023 00:00:00 GMT

Bloomberg had a great article over the weekend about the events leading up to the (partial) UAW strike against the Detroit Three (GM, Ford, and Stellantis [fka Chrysler]).

I’m not surprised (nor should anyone) by the strike. New UAW management, years-long grievances, and tight labor market favoring workers means a strike was almost certainly always going to be the first move by the UAW.

The bigger question is what this will do to the Detroit Three’s electrification push. The UAW’s challenge here (as it has been since the 80s) is that the Detroit Three are in a weak and uncertain position as it relates to foreign auto makers and new EV giants like Tesla. While the Inflation Reduction Act may give US auto makers the US EV market, going into a technology transition with a large labor cost & agility disadvantage is a surefire way to (continue to) cede the much larger global market which, in the end, hurts all of the US auto industry (not to mention the Biden administration’s hopes that this creates new jobs and centers green manufacturing in the US).

Reading Excerpt • Bloomberg News

”

“[UAW President] Fain will need to deliver agreements he can leverage into a more compelling case for workers at carmakers and battery companies the union hasn’t organized. Otherwise, Detroit risks starting the EV age in a position similar to the one they were in when the oil crisis hit half a century ago: stuck with labor costs that put Motor City companies at an untenable disadvantage. “Breathing down the neck of the Big Three are the so-called foreign transplants and Tesla, which are eating up a bigger part of the market and remain unorganized,” said William Gould, a chair of the National Labor Relations Board under President Bill Clinton and professor emeritus at Stanford Law School. “That’s the Achilles’ heel of organized labor.””

How Auto Executives Misread the UAW Ahead of Historic Strike

David Welch, Keith Naughton, Gabrielle Coppola, and Josh Eidelson | Bloomberg News

Hopelessness in China’s Youth

Tue, 22 Aug 2023 00:00:00 GMT

This article in the Economist paints a dismal picture of the state of life for the youth in China: youth unemployment so high the government has stopped reporting on it (as if that was going to change anything…), housing and childcare costs so high that young people have given up on having traditional families, a government and state-run media that actively scolds them for being soft and pampered, and the best and brightest fleeing to Singapore…

How’s that “Chinese dream 中國夢” going?

Smart Home Manifesto

Sat, 19 Aug 2023 00:00:00 GMT

This is an older piece (written in 2016) but remarkable in how well it resonates even 7 years later.

Given that I have a home server and opinions on Matter/Thread, it shouldn’t be a surprise that I have many smart home gadgets in my house. And while I’ve made many purchasing and configuration choices in the spirit of this manifesto, it still boggles my mind that I still fall short of the seamless vision the writer (Paulus Schoutsen, founder of Home Assistant) lays out.

Psychedelics in the Clinic

Fri, 18 Aug 2023 00:00:00 GMT

When I first heard about the use of psychedelics (like ketamine and psilocybin) for treatment of mental illness, I was skeptical. It just seemed too ripe for abuse.

But, there is a growing body of credible academic work suggesting that psychedelics when dosed properly and used in conjunction with therapy / other drugs can be a gamechanger — especially for treatment-resistant depression and suicidality — and that is incredibly exciting.

At the same time, as a former telemedicine startup operator, this makes me more alarmed by the numerous companies working to commercialize these. In the bid for venture-style growth, it’s all too easy to lose track of the “when dosed properly and used in conjunction with therapy / other drugs” part.

In any event, this article from Medicine at Michigan is a good overview of the recent research highlights in the field and why so many clinicians and scientists are excited.

Hawaiian Electric having a PG&E Moment

Thu, 17 Aug 2023 00:00:00 GMT

The fires in Maui have had a devastating human toll (111 dead, 1000 missing as of this writing). It is not surprising that it’s raising some questions about the role of Hawaii’s utility (Hawaiian Electric/HECO) played in the disaster.

While it will take time to sort out the investigation and the class action lawsuit, it’s clear that investors and Hawaiian Electric management are scrambling, with the WSJ reporting that Hawaiian Electric is now talking to restructuring advisors to explore their next steps, in a crisis that very much parallels the series of wildfires that were ultimately blamed on Northern California utility PG&E and resulted in bankruptcy proceedings.

Utilities now face three simultaneous problems (arguably of their own making):

climate change escalating the risks of catastrophic wildfires and storms
utilities across the country having aging energy infrastructure
homeownership patterns, disaster insurance coverage & premiums, and utility risk management plans built for a pre-climate-change risk environment

The smart ones will be proactively overhauling their processes and infrastructure to cope with this. The less smart ones will potentially be dragged kicking and screaming into this world in much the same way that PG&E and Hawaiian Electric currently are.

Setting Up Pihole, Nginx Proxy, and Twingate with OpenMediaVault

Thu, 10 Aug 2023 00:00:00 GMT

(Note: this is part of my ongoing series on cheaply selfhosting)

I recently shared how I set up a (OpenMediaVault) home server on a cheap mini-PC. After posting it, I received a number of suggestions that inspired me to make a few additional tweaks to improve the security and usability of my server.

Read more if you’re interested in setting up (on an OpenMediaVault v6 server):

Pihole, a “DNS filter” that blocks ads / trackers
using Pihole as a local DNS server to have custom web addresses for software services running on your network and Nginx to handle port forwarding
Twingate (a better alternative to opening up a port and setting up Dynamic DNS to grant secure access to your network)

Pihole

Pihole is a lightweight local DNS server (it gets its name from the Raspberry Pi, a <$100 device popular with hobbyists, that it can run fully on).

A DNS (or Domain Name Server) converts human readable addresses (like www.google.com) into IP addresses (like 142.250.191.46). As a result, every piece of internet-connected technology is routinely making DNS requests when using the internet. Internet service providers typically offer their own DNS servers for their customers. But, some technology vendors (like Google and CloudFlare) also offer their own DNS services with optimizations on speed, security, and privacy.

A home-grown DNS server like Pihole can layer additional functionality on top:

DNS “filter” for ad / tracker blocking: Pihole can be configured to return dummy IP addresses for specific domains. This can be used to block online tracking or ads (by blocking the domains commonly associated with those activities). While not foolproof, one advantage this approach has over traditional ad blocking software is that, because this blocking happens at the network level, the blocking extends to all devices on the network (such as internet-connected gadgets, smart TVs, and smartphones) without needing to install any extra software.
DNS caching for performance improvements: In addition to the performance gains from blocking ads, Pihole also boosts performance by caching commonly requested domains, reducing the need to “go out to the internet” to find a particular IP address. While this won’t speed up a video stream or download, it will make content from frequently visited sites on your network load faster by skipping that internet lookup step.

To install Pihole using Docker on OpenMediaVault:

If you haven’t already, make sure you have OMV Extras and Docker Compose installed (refer to the section Docker and OMV-Extras in my previous post) and have a static local IP address assigned to the server.
Login to your OpenMediaVault web admin panel, go to [Services > Compose > Files], and press the button. Under Name put down [PIHOLE] and under File, adapt the following (making sure the number of spaces are consistent)
```
version: "3"  
services:  
   pihole:  
     container_name: pihole  
     image: pihole/pihole:latest  
     ports:  
       - "53:53/tcp"  
       - "53:53/udp"  
       - "8000:80/tcp"  
     environment:  
       TZ: 'America/Los_Angeles'  
       WEBPASSWORD: '&lt;Password for the web admin panel&gt;'  
       FTLCONF_LOCAL_IPV4: '&lt;your server IP address&gt;'  
     volumes:  
       - '&lt;absolute path to shared config folder&gt;/pihole:/etc/pihole'  
       - '&lt;absolute path to shared config folder&gt;/dnsmasq.d:/etc/dnsmasq.d'  
     restart: unless-stopped
```
You’ll need to replace <Password for the web admin panel> with the password you’ll want to use to be access the Pihole web configuration interface, <your server IP address> with the static local IP address for your server, and <absolute path to shared config folder> with the absolute path to the config folder where you want Docker-installed applications to store their configuration information (accessible by going to [Storage > Shared Folders] in the administrative panel).

I live in the Bay Area so I set timezone TZ to America/Los_Angeles. You can find yours here.

Under Ports, I’ve kept the port 53 reservation (as this is the standard port for DNS requests) but I’ve chosen to map the Pihole administrative console to port 8000 (instead of the default of port 80 to avoid a conflict with the OpenMediaVault admin panel default). Note: This will prevent you from using Pihole’s default pi.hole domain as a way to get to the Pihole administrative console out-of-the-box. Because standard web traffic goes to port 80 (and this configuration has Pihole listening at port 8080), pi.hole would likely just direct you to the OpenMediaVault panel. While you could let pi.hole take over port 80, you would need to move OpenMediaVault’s admin panel to a different port (which itself has complexity). I ultimately opted with keeping OpenMediaVault at port 80 knowing that I could configure Pihole and Nginx proxy (see below) to redirect pi.hole to the right port.

You’ll notice this configures two volumes, one for dnsmasq.d, which is the DNS service, and one for [PIHOLE] which provides an easy way to configure dnsmasq.d and download blocklists.

Note: the above instructions assume your home network, like most, is IPv4 only. If you have an IPv6 network, you will need to add an IPv6: True line under environment: and replace the FTLCONF_LOCAL_IPV4:'<server IPv4 address>' with FTLCONF_LOCAL_IPV6:'<server IPv6 address>'. For more information, see the official Pihole Docker instructions.

Once you’re done, hit Save and you should be returned to your list of Docker compose files for the next step. Notice that the new [PIHOLE] entry you created has a Down status, showing the container has yet to be initiated.
Disabling systemd-resolved: Most modern Linux operating systems include a built-in DNS resolver that listens on port 53 called systemd-resolved. Prior to initiating the Pihole container, you’ll need to disable this to prevent that port conflict. Use WeTTy (refer to the section Docker and OMV-Extras in my previous post) or SSH to login as the root user to your OpenMediaVault command line. Enter the following command:
```
nano /etc/systemd/resolved.conf
```
Look for the line that says #DNSStubListener=yes and replace it with DNSStubListener=no, making sure to remove the # at the start of the line. (Hit Ctrl+X to exit, Y to save, and Enter to overwrite the file). This configuration will tell systemd-resolved to stop listening to port 53.

To complete the configuration change, you’ll need to edit the symlink /etc/resolv.conf to point to the file you just edited by running:
```
sh -c 'rm /etc/resolv.conf && ln -s /etc/systemd/resolved.conf /etc/resolv.conf'
```
Now all that remains is to restart systemd-resolved:
```
systemctl restart systemd-resolved
```
How to start / update / stop / remove your Pihole container: You can manage all of your Docker Compose files by going to [Services > Compose > Files] in the OpenMediaVault admin panel. Click on the [PIHOLE] entry (which should turn it yellow) and press the (up) button. This will create the container, download any files needed, and, if you properly disabled systemd-resolved in the last step, initiate Pihole.

And that’s it! To prove it worked, go to your-server-ip:8000 in a browser and you should see the login for the Pihole admin webpage (see below).

From time to time, you’ll want to update the container. OMV makes this very easy. Every time you press the (pull) button in the [Services > Compose > Files] interface, Docker will pull the latest version (maintained by the Pihole team).

Now that you have Pihole running, it is time to enable and configure it for your network.

Test Pihole from a computer: Before you change your network settings, it’s a good idea to make sure everything works.
- On your computer, manually set your DNS service to your Pihole by putting in your server IP address as the address for your computer’s primary DNS server (Mac OS instructions; Windows instructions; Linux instructions). Be sure to leave any alternate / secondary addresses blank (many computers will issue DNS requests to every server they have on their list and if an alternative exists you may not end up blocking anything).
- (Temporarily) disable any ad blocking service you may have on your computer / browser you want to test with (so that this is a good test of Pihole as opposed to your ad blocking software). Then try to go to https://consumerproductsusa.com/ — this is a URL that is blocked by default by Pihole. If you see a very spammy website promising rewards, either your Pihole does not work or you did not configure your DNS correctly.
- Finally login to the Pihole configuration panel (your-server-ip:8000) using the password you set up during installation. From the dashboard click on the Queries Blocked box at the top (your colors may vary but it’s the red box on my panel, see below).
  
  On the next screen, you should see the domain consumerproductsusa.com next to the IP address of your computer, confirming that the address was blocked.
  
  You can now turn your ad blocking software back on!
- You should now set the DNS service on your computer back to “automatic” or “DHCP” so that it will inherit its DNS settings from the network/router (and especially if this is a laptop that you may use on another network).
Configure DNS on router: Once you’ve confirmed that the Pihole service works, you should configure the default DNS settings on your router to make Pihole the DNS service for your entire network. The instructions for this will vary by router manufacturer. If you use Google Wifi as I do, here are the instructions.

Once this is completed, every device which inherits DNS settings from the router will now be using Pihole for their DNS requests.

Note: one downside of this approach is that the Pihole becomes a single point of failure for the entire network. If the Pihole crashes or fails, for any reason, none of your network’s DNS requests will go through until the router’s settings are changed or the Pihole becomes functional again. Pihole generally has good reliability so this is unlikely to be an issue most of the time, but I am currently using Google’s DNS as a fallback on my Google Wifi (for the times when something goes awry with my server) and I would also encourage you to know how to change the DNS settings for your router in case things go bad so that your access to the internet is not taken out unnecessarily.
Configure Pihole: To get the most out of Pihole’s ad blocking functionality, I would suggest three things
- Select Good Upstream DNS Servers: From the Pihole administrative panel, click on Settings. Then select the DNS tab. Here, Pihole allows you to configure which external DNS services the DNS requests on your network should go to if they aren’t going to be blocked and haven’t yet been cached. I would recommend selecting the checkboxes next to Google and Cloudflare given their reputations for providing fast, secure, and high quality DNS services (and selecting multiple will provide redundancy).
- Update Gravity periodically: Gravity is the system by which Pihole updates its list of domains to block. From the Pihole administrative panel, click on [Tools > Update Gravity] and click the Update button. If there are any updates to the blocklists you are using, these will be downloaded and “turned on”.
- Configure Domains to block/allow: Pihole allows administrators to granularly customize the domains to block (blacklist) or allow (whitelist). From the Pihole administrative panel, click on Domains. Here, an admin can add a domain (or a regular expression for a family of domains) to the blacklist (if it’s not currently blocked) or the whitelist (if it currently is) to change what happens when a user on the network accesses the DNS.
  
  I added whitelist exclusions for link.axios.com to let me click through links from the Axios email newsletters I receive and www.googleadservices.com to let my wife click through Google-served ads. Pihole also makes it easy to manually take a domain that a device on your network has requested to block/allow. Tap on Total Queries from the Pihole dashboard, click on the IP address of the device making the request, and you’ll see every DNS request (including those which were blocked) with a link beside them to add to the domain whitelist or blacklist.
  
  Pihole will also allow admins to configure different rules for different sets of devices. This can be done by calling out clients (which can be done by clicking on Clients and picking their IP address / MAC address / hostnames), assigning them to groups (which can be defined by clicking on Groups), and then configuring domain rules to go with those groups (in Domains). Unfortunately because Google Wifi simply forwards DNS requests rather than distributes them, I can only do this for devices that are configured to directly point at the Pihole, but this could be an interesting way to impose parental internet controls.

Now you have a Pihole network-level ad blocker and DNS cache!

Local DNS and Nginx proxy

As a local DNS server, Pihole can do more than just block ads. It also lets you create human readable addresses for services running on your network. In my case, I created one for the OpenMediaVault admin panel (omv.home), one for WeTTy (wetty.home), and one for Ubooquity (ubooquity.home).

If your setup is like mine (all services use the same IP address but different ports), you will need to set up a proxy as DNS does not handle port forwarding. Luckily, OpenMediaVault has Nginx, a popular web server with a performant proxy, built-in. While many online tutorials suggest installing Nginx Proxy Manager, that felt like overkill, so I decided to configure Nginx directly.

To get started:

Configure the A records for the domains you want in Pihole: Login to your Pihole administrative console (your-server-ip:8000) and click on [Local DNS > DNS Records] from the sidebar. Under the section called Add a new domain/IP combination, fill out the Domain: you want for a given service (like omv.home or wetty.home) and the IP Address: (if you’ve been following my guides, this will be your-server-ip). Press the Add button and it will show up below. Repeat for all the domains you want. If you have a setup similar to mine, you will see many domains pointed at the same IP address (because the different services are simply different ports on my server).

To test if these work, enter any of the domains you just put in to a browser and it should take you to the login page for the OpenMediaVault admin panel (as currently they are just pointing at your server IP address).

Note 1: while you can generally use whatever domains you want, it is suggested that you don’t use a TLD that could conflict with an actual website (i.e. .com) or that are commonly used by networking systems (i.e. .local or .lan). This is why I used .home for all of my domains (the IETF has a list they recommend, although it includes .lan which I would advise against as some routers such as Google Wifi use this)

Note 2: Pihole itself automatically tries to forward pi.hole to its web admin panel, so you don’t need to configure that domain. The next step (configuring proxy port forwarding) will allow pi.hole to work.
Edit the Nginx proxy configuration: Pihole’s Local DNS server will send users looking for one of the domains you set up (i.e. wetty.home) to the IP address you configured. Now you need your server to forward that request to the appropriate port to get to the right service.

You can do this by taking advantage of the fact that Nginx, by default, will load any .conf file in the /etc/nginx/conf.d/ directory as a proxy configuration. Pick any file name you want (I went with dothome.conf because all of my service domains end with .home) and after using WeTTy or SSH to login as root, run:
```
nano /etc/nginx/conf.d/&lt;your file name&gt;.conf
```
The first time you run this, it will open up a blank file. Nginx looks at the information in this file for how to redirect incoming requests. What we’ll want to do is tell Nginx that when a request comes in for a particular domain (i.e. ubooquity.home or pi.hole) that request should be sent to a particular IP address and port.

Manually writing these configuration files can be a little daunting and, truth be told, the text file I share below is the result of a lot of trial and error, but in general there are 2 types of proxy commands that are relevant for making your domain setup work.

One is a proxy_pass where Nginx will basically take any traffic to a given domain and just pass it along (sometimes with additional configuration headers). I use this below for wetty.home, pi.hole, ubooquityadmin.home, and ubooquity.home. It worked without the need to pass any additional headers for WeTTy and Ubooquity, but for pi.hole, I had to set several additional proxy headers (which I learned from this post on Reddit).

The other is a 301 redirect where you tell the client to simply forward itself to another location. I use this for ubooquityadmin.home because the actual URL you need to reach is not / but /admin/ and the 301 makes it easy to setup an automatic forward. I then use the regex match ~ /(.*)$ to make sure every other URL is proxy_pass‘d to the appropriate domain and port.

You’ll notice I did not include the domain I configured for my OpenMediaVault console (omv.home). That is because omv.home already goes to the right place without needing any proxy to port forward.
```
server {  
        listen        80;  
        server_name   pi.hole;  
        location / {  
                proxy_pass        http://&lt;your-server-ip&gt;:8000;  
                proxy_set_header  Host            $host;  
                proxy_set_header  X-Real-IP       $host;  
                proxy_set_header  X-ForwardedFor  $proxy_add_x_forwarded_for;             
                proxy_hide_header X-Frame-Options;  
                proxy_set_header  X-Frame-Options "SAMEORIGIN";  
                proxy_read_timeout 90;  
       }  
}  
server {  
        listen        80;  
        server_name   wetty.home;  
        location / {  
                proxy_pass       http://&lt;your-server-ip&gt;:2222;  
                proxy_set_header  Host            $host;  
                proxy_set_header  X-Real-IP       $host;  
                proxy_set_header  X-ForwardedFor  $proxy_add_x_forwarded_for;             
        }  
}  
server {  
        listen        80;  
        server_name   ubooquity.home;  
        location / {  
                proxy_pass       http://&lt;your-server-ip&gt;:2202;  
        }  
}  
server {  
        listen        80;  
        server_name   ubooquityadmin.home;  
        location =/ {  
                return 301 http://ubooquityadmin.home/admin;  
        }  
        location ~ /(.*)$ {  
                proxy_pass        http://&lt;your-server-ip&gt;:2203/$1;  
        }  
}
```
If you are using other domains, ports, or IP addresses, adjust accordingly. Be sure all your curly braces have their mates ({}) and that each line ends with a semicolon (;) or Nginx will crash. I use Tab‘s between statements (i.e. between listen and 80) to format them more nicely but Nginx will accept any number or type of whitespace.

To test if your new configuration worked, save your changes (hit Ctrl+X to exit, Y to save, and Enter to overwrite the file if you are editing a pre-edited one). In the command line, run the following command to restart Nginx with your new configuration loaded.
```
systemctl restart nginx
```
Try to login to your OpenMediaVault administrative panel in a browser. If that works, it means Nginx is up and running and you at least didn’t make any obvious syntax errors!

Next try to access one of the domains you just configured (for instance pi.hole) to test if the proxy was configured correctly.

If either of those steps failed, use WeTTy or SSH to log back in to the command line and use the command above to edit the file (you can delete everything if you want to start fresh) and rerun the restart command after you’ve made changes to see if that fixes it. It may take a little bit of doing if you have a tricky configuration but once you’re set, everyone on the network can now use your configured addresses to access the services on your network.

Twingate

In my previous post, I set up Dynamic DNS and a Wireguard VPN to grant secure access to the network from external devices (i.e. a work computer, my smartphone when I’m out, etc.). While it worked, the approach had two flaws:

The work required to set up each device for Wireguard is quite involved (you have to configure it on the VPN server and then pass credentials to the device via QR code or file)
It requires me to open up a port on my router for external traffic (a security risk) and maintain a Dynamic DNS setup that is vulnerable to multiple points of failure and could make changing domain providers difficult.

A friend of mine, after reading my post, suggested I look into Twingate instead. Twingate offers several advantages, including:

Simple graphical configuration of which resources should be made available to which devices
Easier to use client software with secure (but still easy to use) authentication
No need to configure Dynamic DNS or open a port
Support for local DNS rules (i.e. the domains I configured in Pihole)

I was intrigued (it didn’t hurt that Twingate has a generous free Starter plan that should work for most home server setups). To set up Twingate to enable remote access:

Create a Twingate account and Network: Go to their signup page and create an account. You will then be asked to set up a unique Network name. The resulting address, <yournetworkname>.twingate.com, will be your Network configuration page from where you can configure remote access.
Add a Remote Network: Click the Add button on the right-hand-side of the screen. Select On Premise for Location and enter any name you choose (I went with Home network).
Add Resources: Select the Remote Network you just created (if you haven’t already) and use the Add Resource button to add an individual domain name or IP address and then grant access to a group of users (by default, it will go to everyone).

With my configuration, I added 5 domains (pi.hole + the four .home domains I configured through Pihole) and 1 IP address (for the server, to handle the ubooquityadmin.home forwarding and in case there was ever a need to access an additional service on my server that I had not yet created a domain for).
Install Connector Docker Container: To make the selected network resources available through Twingate requires installing a Twingate Connector to something internet-connected on the network.

Press the Deploy Connector button on one of the connectors on the right-hand-side of the Remote Network page (mine is called flying-mongrel). Select [DOCKER] in Step 1 to get Docker instructions (see below). Then press the Generate Tokens button under Step 2 to create the tokens that you’ll need to link your Connector to your Twingate network and resources.

With the Access Token and Refresh Token saved, you are ready to configure Docker to install. Login to the OpenMediaVault administrative panel and go to [Services > Compose > Files] and press the button. Under Name put down Twingate Connector and under File, enter the following (making sure the number of spaces are consistent)
```
services:  
   twingate_connector:  
     container_name: twingate_connector  
     restart: unless-stopped  
     image: "twingate/connector:latest"  
     environment:  
       - SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt  
       - TWINGATE_API_ENDPOINT=/connector.stock  
       - TWINGATE_NETWORK=&lt;your network name&gt;  
       - TWINGATE_ACCESS_TOKEN=&lt;your connector access token&gt;  
       - TWINGATE_REFRESH_TOKEN=&lt;your connector refresh token&gt;  
       - TWINGATE_LOG_LEVEL=7
```
You’ll need to replace <your network name> with the name of the Twingate network you created, <your connector access token> and <your connector refresh token> with the access token and refresh token generated from the Twingate website. Do not add any single or double quotation marks around the network name or the tokens as they will result in a failed authentication with Twingate (as I was forced to learn through experience).

Once you’re done, hit Save and you should be returned to your list of Docker compose files. Click on the entry for Twingate Connector you just created and then press the (up) button to initialize the container.

Go back to your Twingate network page and select the Remote Network your Connector is associated with. If you were successful, within a few moments, the Connector’s status will reflect this (see below for the before and after).

If, after a few minutes there is still no change, you should check the container logs. This can be done by going to [Services > Compose > Services] in the OpenMediaVault administrative panel. Select the Twingate Connector container and press the (logs) button in the menubar. The TWINGATE_LOG_LEVEL=7 setting in the Docker configuration file sets the Twingate Connector to report all activities in great detail and should give you (or a helpful participant on the Twingate forum) a hint as to what went wrong.
Add Users and Install Clients: Once the configuration is done and the Connector is set up, all that remains is to add user accounts and install the Twingate client software on the devices that should be able to access the network resources.

Users can be added (or removed) by going to your Twingate network page and clicking on the Team link in the menu bar. You can Add User (via email) or otherwise customize Group policies. Be mindful of the Twingate Starter plan limit to 5 users…

As for the devices, the client software can be found at <https://get.twingate.com/>. Once installed, to access the network, the user will simply need to authenticate.
Remove my old VPN / Dynamic DNS setup. This is not strictly necessary, but if you followed my instructions from before, you can now undo those by:
- Closing the port you opened from your Router configuration
- Disabling Dynamic DNS setup from your domain provider
- “Down”-ing and deleting the container and configuration file for DDClient (you can do this by going to [Services > Compose > Files] from OpenMediaVault admin panel)
- Deleting the configured Wireguard clients and tunnels (you can do this by going to [Services > Wireguard] from the OpenMediaVault admin panel) and then disabling the Wireguard plugin (go to [System > Plugins])
- Removing the Wireguard client from my devices

And there you have it! A secure means of accessing your network while retaining your local DNS settings and avoiding the pitfalls of Dynamic DNS and opening a port.

Resources

There were a number of resources that were very helpful in configuring the above. I’m listing them below in case they are helpful:

(If you’re interested in how to setup a home server on OpenMediaVault or how to self-host different services, check out all my posts on the subject)

Why Thread is Matter’s biggest problem right now

Tue, 08 Aug 2023 00:00:00 GMT

Stop me if you’ve heard this one before… Adoption of a technology is being impeded by too many standards. The solution? A new standard, of course, and before you know it, we now have another new standard to deal with.

The smart home industry needs to figure out how to properly embrace Thread (and Matter). It (or something like it) will be necessary for broader smart home / Internet of Things adoption.

The next big thing on ESPN is… Microsoft Excel

Fri, 04 Aug 2023 00:00:00 GMT

As a former “Excel monkey”, I am extremely tickled by the fact that what used to be analyst bravado about Excel skills (being able to create big spreadsheet models without touching the mouse was a big thing) is now a sport with viewers.

But it’s a testament to how powerful and versatile spreadsheets are. And how many people know what it is.

The joke in SaaS is that every SaaS product is basically competing with Excel. Well, apparently, Excel’s competing with e-sports and games now too!

Flow Batteries Resurgent?

Wed, 02 Aug 2023 00:00:00 GMT

I’ve been pitched by numerous flow battery companies in my days as a deeptech/climatetech investor. The promise of the technology has always been:

Long cycle life (the number of charge-discharge cycles you can do before the performance degrades)
Easy to scale: you want 2x the storage? Just get 2x the electrolyte!
Low fire risk: most flow batteries use water-based electrolytes which won’t ignite in the air (the way the lithium in lithium-ion batteries do)

Despite compelling benefits, this category never achieved the level of success or scale as lithium-ion did. This was due in part to a variety of technological limitations (poor energy density, lower cycle efficiency, concerns around the amount of Vanadium-containing electrolyte “lying around” in a system, etc). But, the main cause was the breath-taking progress lithium-ion batteries have made in cost, energy density, and safety driven first by consumer electronics demand and then by electric vehicle demand.

This C&EN article covers the renewed optimism the flow battery world is experiencing as market interest in the technology revitalizes.

My hot-take🔥: while technological improvements play a part, once again, what is driving the flow battery market is what’s happening in lithium-ion world. There simply is too much demand for energy storage and growing uncertainty about the ability of lithium-ion to handle it in the face of the conflict between the West and China (the leading supplier of lithium ion batteries) and supply chain concerns about critical minerals for lithium ion batteries (like nickel and cobalt). Grid storage players have to look elsewhere. (Electric vehicle companies would probably like to but do not have the option!)

Considering the importance of grid energy storage in electrifying our world and onboarding new renewable generation, I think having and seeing more options is a good thing. So I, too, am optimistic here 👍🏻

Note: this is the first in a (hopefully ongoing) series of posts called “What I’m Reading” where I’ll share & comment on an interesting article I’ve come across!

Reading Excerpt • C&EN

”

“Redox flow batteries have a reputation of being second best. Less energy intensive and slower to charge and discharge than their lithium-ion cousins, they fail to meet the performance requirements of snazzy, mainstream applications, such as cars and cell phones. There’s no such thing as a flow-battery Tesla. But the companies at the International Flow Battery Forum in Prague in late June were adamant that flow batteries are now cheaper, more reliable, and safer than lithium ion in a growing number of real-world stationary energy applications. Flow-battery makers say their technology—and not lithium ion—should be the first choice for capturing excess renewable energy and returning it when the sun is not out and the wind is not blowing.”

Flow batteries, the forgotten energy storage device

Alex Scott | C&EN

Setting Up an OpenMediaVault Home Server with Docker, Plex, Ubooquity, and WireGuard

Wed, 26 Jul 2023 00:00:00 GMT

(Note: this is part of my ongoing series on cheaply selfhosting)

I spent a few days last week setting up a cheap home server which now serves my family as:

a media server — stores and streams media to phones, tablets, computers, and internet-connected TVs (even when I’m out of the house!)
network-attached storage (NAS) — lets computers connected to my home network store and share files
VPN — lets me connect to my storage and media server when I’m outside of my home

Until about a week ago, I had run a Plex media server on my aging (8 years old!) NVIDIA SHIELD TV. While I loved the device, it was starting to show it’s age – it would sometimes overheat and not boot for several days. My home technology setup had also shifted. I bought the SHIELD all those years ago to put Android TV functionality onto my “dumb” TV.

But, about a year ago, I upgraded to a newer Sony TV which had it built-in. Now, the SHIELD felt “extra” and the media server felt increasingly constrained by what it could not do (e.g., slow network access speeds, can only run services that are Android apps, etc.)

I considered buying a high-end consumer NAS from Synology or QNAP (which would have been much simpler!), but decided to build my own to both get better hardware for less money but also as a fun project which would teach me more about servers and let me configure everything to my heart’s content.

If you’re interested in doing something similar, let me walk you through my hardware choices and the steps I took to get to my current home server setup.

Note: on the recommendation of a friend, I’ve since reconfigured how external access works to not rely on a VPN with an open port and Dynamic DNS and instead use Twingate. For more information, refer to my post on Setting Up Pihole, Nginx Proxy, and Twingate with OpenMediaVault

Hardware

I purchased a Beelink EQ12 Mini, a “mini PC” (fits in your hand, power-efficient, but still capable of handling a web browser, office applications, or a media server), during Amazon’s Prime Day sale for just under $200.

Beelink EQ12 Mini (Image Source: Chigz Tech Review)

While I’m very happy with the choice I made, for those of you contemplating something similar, the exact machine isn’t important. Many of the mini PC brands ultimately produce very similar hardware, and by the time you read this, there will probably be a newer and better product. But, I chose this particular model because:

It was from one of the more reputable Mini PC brands which gave me more confidence in its build quality (and my ability to return it if something went wrong). Other reputable vendors beyond Beelink include Geekom, Minisforum, Chuwi, etc.
It had a USB-C port which helps with futureproofing, and the option to convert this into something else useful if this server experiment doesn’t work out.
It had an Intel CPU. While AMD makes excellent CPUs, the benefit of going with Intel is support for Intel Quick Sync, which allows for hardware accelerated video transcode (converting video and audio streams to different formats and resolutions – so that other devices can play them – without overwhelming the system or needing a beefy graphics card). Many popular media servers support Intel Quick Sync-powered transcode.
It was not a i3/5/7/9 chip. Intel’s higher end chips have names that include “i3” or “i5” or “i7”. Those are generally overkill on performance, power consumption, and price for a simple file and media server. All I needed for my purposes was a lower-end Celeron-type device.
It was the most advanced Intel architecture I could find for ≤$200. While I didn’t need the best performance, there was no reason to avoid more advanced technology. Thankfully, the N100 chip in the EQ12 Mini uses Intel’s 12th Generation Core architecture (Alder Lake). Many of the other mini-PCs at this price range had older (10th and 11th generation) CPUs.
I went with the smallest RAM and onboard storage option. I wasn’t planning on putting much on the included storage (because you want to isolate the operating system for the server away from the data) nor did I expect to tax the computer memory for my use case.

I also considered purchasing a Raspberry Pi, a <$100 low-power device popular with hobbyists, but the lack of transcode and the non-x86 architecture (Raspberry Pi’s use ARM CPUs and won’t be compatible with all server software) pushed me towards an Intel-based mini PC.

In addition to the mini-PC, I also needed:

Storage: a media server / NAS without storage is not very useful. I had a 4 TB USB hard drive (previously connected to my SHIELD TV) which I used here, and I also bought a 4 TB SATA SSD (for ~$150) to mount inside the mini-PC.
- Note 1: if you decide to go with OpenMediaVault as I have, install the Linux distribution before you install the SATA drive. The installer (foolishly) tries to install itself to the first drive it finds, so don’t give it any bad options.
- Note 2: most Mini PC manufacturers say their systems only support additional drives up to 2 TB. This appears to be mainly the manufacturers being overly conservative. My 4 TB SATA SSD works like a charm.
A USB stick: Most Linux distributions (especially those that power open source NAS solutions) are installed from a bootable USB stick. I used one that was lying around that had 2 GB on it.
Ethernet cables and a “dumb” switch: I use Google Wifi in my home and I wanted to connect both my TV and my new media server to the router in my living room. To do that, I bought a simple Ethernet switch (you don’t need anything fancy because it’s just bridging several devices) and 3 Ethernet cables to tie it all together (one to connect the router to the switch, one to connect the TV to the switch, and one to connect the server to the switch). Depending on your home configuration, you may want something different.
A Monitor & Keyboard: if you decide to go with OpenMediaVault as I have, you’ll only need this during the installation phase as the server itself is controllable through a web interface. So, I used an old keyboard and monitor (that I’ve since given away).

OpenMediaVault

There are a number of open source home server / NAS solutions you can use. But I chose to go with OpenMediaVault because it’s:

built on Debian (a well-supported, widely used flavor of Linux)
still being actively developed (last update as of this writing was literally today)
considered easy to use
lightweight enough to be able to run on a Raspberry Pi
backed by a solid community forum and subreddit (something I knew I would need!)

To install OpenMediaVault on the mini PC, you just need to:

Download the installation image ISO and burn it to a bootable USB stick (if you use Windows, you can use Rufus to do so)
Plug the USB stick into the mini PC (and make sure to connect the monitor and keyboard) and then turn the machine on. If it goes to Windows (i.e. it doesn’t boot from your USB stick), you’ll need to restart and go into BIOS (you can usually do this by pressing Delete or F2 or F7 after turning on the machine) to configure the machine to boot from a USB drive.
Follow the on-screen instructions.
- You should pick a good root password and write it down (it gates administrative access to the machine, and you’ll need it to make some of the changes below).
- You can pick pretty much any name you want for the hostname and domain name (it shouldn’t affect anything but it will be what your machine calls itself).
- Make sure to select the right drive for installation
And that should be it! After you complete the installation, you will be prompted to enter the root password you created to login.

Unfortunately for me, OpenMediaVault did not recognize my mini PC’s ethernet ports or wireless card. If it detects your network adapter just fine, you can skip this next block of steps. But, if you run into this, select the “does not have network card” option and “minimal setup” options during install. You should still be able to get the end of the process. Then, once the OpenMediaVault operating system installs and reboots:

Login by entering the root password you picked during the installation and make sure your system is plugged in to your router via ethernet. Note: Linux is known to have issues recognizing some wireless cards and it’s considered best practice to run a media server off of Ethernet rather than WiFi.
In the command line, enter omv-firstaid. This is a gateway to a series of commonly used tools to fix an OpenMediaVault install. In this case, select the Configure Network Interface option and say yes to all the IPv4 DHCP options (you can decide if you want to set up IPv6).
Step 2 should fix the issue where OpenMediaVault could not see your internet connection. To prove this, you should try two things:
- Enter ping google.com -c 3 in the command line. You should see 3 lines with something like 64 bytes from random-url.blahurl.net showing that your system could reach Google (and thus the internet). If it doesn’t work, try again in a few minutes (sometimes it takes some time for your router to register a new system).
- Enter ip addr in the command line. Somewhere on the screen, you should see something that probably looks like inet 192.168.xx.xx/xx. That is your local IP address and it’s a sign that the mini PC has connected to your router.
Now you need to update the Linux operating system so that it knows where to look for updates to Debian. As of this writing, the latest version of OpenMediaVault (6) is based on Debian 11 (codenamed Bullseye), so you may need to replace bullseye with <name of Debian codename that your OpenMediaVault is based on> in the text below if your version is based on a different version of Debian (i.e. Bookworm, Trixie, etc.).

In the command line, enter nano /etc/apt/sources.list. This will let you edit the file that contains all the information on where your Linux operating system will find valid software updates. Enter the text below underneath all the lines that start with # (replacing bullseye with the name of the Debian version that underlies your version of OpenMediaVault if needed).
```
deb http://deb.debian.org/debian bullseye main   
deb-src http://deb.debian.org/debian bullseye main   
deb http://deb.debian.org/debian-security/ bullseye-security main   
deb-src http://deb.debian.org/debian-security/ bullseye-security main   
deb http://deb.debian.org/debian bullseye-updates main   
deb-src http://deb.debian.org/debian bullseye-updates main
```
Then press Ctrl+X to exit, press Y when asked if you want to save your changes, and finally Enter to confirm that you want to overwrite the existing file.
To prove that this worked, in the command line enter apt-get update and you should see some text fly by that includes some of the URLs you entered into sources.list. Next enter apt-get upgrade -y, and this should install all the updates the system found.

Congratulations, you’ve installed OpenMediaVault!

Setting up the File Server

You should now connect any storage (internal or USB) that you want to use for your server. You can turn off the machine if you need to by pulling the plug, or holding the physical power button down for a few seconds, or by entering shutdown now in the command line. After connecting the storage, turn the system back on.

Once setup is complete, OpenMediaVault can generally be completely controlled and managed from the web. But to do this, you need your server’s local IP address. Log in (if you haven’t already) using the root password you set up during the installation process. Enter ip addr in the command line. Somewhere on the screen, you should see something that looks like inet 192.168.xx.xx/xx. That set of numbers connected by decimal points but before the slash (for example: 192.168.444.23) is your local IP address. Write that down.

Now, go into any other computer connected to the same network (i.e. on WiFi or plugged into the router) as the media server and enter the local IP address you wrote down into the address bar of a browser. If you configured everything correctly, you should see something like this (you may have to change the language to English by clicking on the globe icon in the upper right):

The OpenMediaVault administrative panel login

Congratulations, you no longer need to connect a keyboard or mouse to your server, because you can manage it from any other computer on the network!

Login using the default username admin and default password openmediavault. Below are the key things to do first. (Note: after hitting Save on a major change, as an ~~annoying~~ extra precaution, OpenMediaVault will ask you to confirm the change again with a bright yellow confirmation banner at the top. You can wait until you have several changes, but you need to make sure you hit the check mark at least once or your changes won’t be reflected):

Change your password: This panel controls the configuration for your system, so it’s best not to let it be the default. You can do this by clicking on the (user settings) icon in the upper-right and selecting Change Password
Some useful odds & ends:
- Make auto logout (time before the panel logs you out automatically) longer. You can do this by going to [System > Workbench] in the menu and changing Auto logout to something like 60 minutes
- Set the system timezone. You can do this by going to [System > Date & Time] and changing the Time zone field.
Update the software: On the left-hand side, select [System > Update Management > Updates]. Press the button to search for new updates. If any show up press the button to install everything on the list that it can. (see below, Image credit: OMV-extras Wiki)
Mount your storage:
- From the menu, select [Storage > Disks]. The table that results (see below) shows everything OpenMediaVault sees connected to your server. If you’re missing anything, time to troubleshoot (check the connection and then make sure the storage works on another computer).
- It’s a good idea (although not strictly necessary) to reformat any un-empty disks before using them with OpenMediaVault for performance. You can do this by selecting the disk entry (marking it yellow) and then pressing the (Wipe) button
- Go to [Storage > File Systems]. This shows what drives (and what file systems) are accessible to OpenMediaVault. To properly mount your storage:
  - Press the button for every unformatted drive added you may want to mount to OpenMediaVault. This will add a disk with an existing file system to the purview of your file server.
  - Press the button in the upper-left (just to the right of the triangular button) to add a drive that’s just been formatted. Of the file system options that come up, I would choose EXT4 (it’s what modern Linux operating systems tend to use). This will result in your chosen file system being added to the drive before it’s ultimately mounted.
Set up your File Server: Ok, you’ve got storage! Now you want to make it available for the computers on your network. To do this, you need to do three things:
- Enabling SMB/CIFS: Windows, Mac OS, and Linux systems tend to work pretty well with SMB/CIFS for network file shares. From the menu, select [Services > SMB/CIFS > Settings].
  
  Check the Enabled box. If your LAN workgroup is something other than the default WORKGROUP you should enter it. Now any device on your network that supports SMB/CIFS will be able to see the folders that OpenMediaVault shares. (see below, Image credit: OMV-extras Wiki)
- Selecting folders to share: On the left-hand-side of the administrative panel, select [Storage > Shared Folders]. This will list all the folders that can be shared.
  
  To make a folder available to your network, select the button in the upper-left, and fill out the Name (what you want the folder to be called when other’s access it) and select the File System you’ve previously mounted that the folder will connect to. You can write out the name of the directory you want to share and/or use the directory folder icon to the right of the Relative Path field to help select the right folder. Under Permissions, for simplicity I would assign Everyone: read/write. (see below, Image credit: OMV-extras Wiki)
  
  Hit Save to return to the list of folder shares (see below for what a completed entry looks like, Image credit: OMV-extras Wiki). Repeat the process to add as many Shared Folders as you’d like.
- Make the shared folders available to SMB/CIFS: To do this go to [Services > SMB/CIFS > Shares]. Hit the button and, in, Shared Folder, select the Shared Folder you configured from the dropdown. Under Public, select Guests allowed – this will allow users on the network to access the folder without supplying a username or password. Check the Inherit Permissions, Extended attributes, and Store DOS attributes boxes as well and then hit Save. Repeat this for all the shared folders you want to make available. (Image credit: OMV-extras Wiki)
Set a static local IP: Home networks typically dynamically assign IP addresses to the devices on the network (something called DHCP). As a result, the IP address for your server may suddenly change. To give your server a consistent address to connect to, you should configure your router to assign a static IP to your server. The exact instructions will vary by router so you’ll need to consult your router’s documentation. In my household, we use Google Wifi and, if you do too, here are the instructions for doing so. (Make sure to write down the static IP you assign to the server as you will need it later. If you change the IP from what it already was, make sure to log into the OpenMediaVault panel from that new address before proceeding.)
Check that the shared folders show up on your network: Linux, Mac OS, and Windows all have separate ways of mounting a SMB/CIFS file share. The steps above hopefully simplify this by:
- letting users connect as a Guest (no extra authentication needed)
- providing a Static IP address for the file share

Docker and OMV-Extras

Once upon a time, setting up other software you might want to run on your home server required a lot of command line work. While efficient, it made worse the consequences of entering the wrong command or having two applications with conflicting dependencies. After all, a certain blogger accidentally deleted his entire blog because he didn’t understand what he was doing.

Enter containers. Containers are “portable environments” for software, first popularized by the company Docker, that gives software a predictable background to run on. This makes it easier to run applications reliably, regardless of machine (because the application only sees what the container shows it). It also means a greatly reduced risk of a misconfigured app affecting another since the application “lives” in its own container.

While this has tremendous implications for software in general, for our purposes, this just makes it a lot easier to install software … provided you have Docker installed. For OpenMediaVault, the best way to get Docker is to install OMV-extras.

If you know how to use ssh, go ahead and use it to access your server’s IP address, login as the root user, and skip to Step 4. But, if you don’t, the easiest way to proceed is to set up WeTTY (Steps 1-3):

Install WeTTY: Go to [System > Plugins] and search or scroll until you find the row for openmediavault-wetty. Click on it to mark it yellow and then press the button to install it. WeTTY is a web-based terminal which will let you access the server command line from a browser.
Enable WeTTY: Once the install is complete, go to [Services > WeTTY], check the Enabled box, and hit Save. You’ll be prompted by OpenMediaVault to confirm the pending change.
Press Open UI button on the page to access WeTTY: It should open up a new tab that takes you to your-ip-address:2222 which should open up a black screen which is basically the command line for your server! Enter root when prompted for your username and then your root password that you configured during installation.
Enter this into the command line:
```
wget -O - https://github.com/OpenMediaVault-Plugin-Developers/packages/raw/master/install | bash
```
Installation will take a while but once it’s complete, you can verify it by going back to your administrative panel, refreshing the page, and seeing if there is a new menu item [System > omv-extras].
Enable the Docker repo: From the administrative panel, go to [System > omv-extras] and check the Docker repo box. Press the apt clean button once you have.
Install the Docker-compose plugin: Go to [System > Plugins] and search or scroll down until you find the entry for openmediavault-compose. Click on it to mark it yellow and then press the button on the upper-left to install it. To confirm that it’s been installed, you should see a new menu item [Services > Compose]
Update the System: As before, select [System > Update Management > Updates]. Press the button to search for new updates. Press the button which will automatically install everything.
Create three shared folders: compose, containers, and config: Just as with setting up the network folder shares, you can do this by going to [Storage > Shared Folders] and pressing the button in the upper left. You can generally pick any location you’d like, but make sure it’s on a file system with a decent amount of storage as media server applications can store quite a bit of configuration and temporary data (e.g. preview thumbnails).

compose and containers will be used by Docker to store the information it needs to set up and operate the containers you’ll want.

I would also recommend sharing config on the local network to make it easier to see and change the application configuration files (go to [Services > SMB/CIFS > Shares] and add it in the same way you did for the File Server step). Later below, I use this to add a custom theme to Ubooquity.
Configure Docker Compose: Go to [Services > Compose > Settings]. Where it says Shared folder under Compose Files, select the compose folder you created in Step 8. Where it says Docker storage under [DOCKER] , copy the absolute path (not the relative path) for the containers folder (which you can get from [Storage > Shared Folders]). Once that’s all set. Press Reinstall Docker.
Set up a User for Docker: You’ll need to create a separate user for Docker as it is dangerous to give any application full access to your root user. Go to [Users > Users] (yes, that is Users twice). Press the button to create a new user. You can give it whatever name (i.e. dockeruser) and password you want, but under Groups make sure to select both [DOCKER] and users. Hit Save and once you’re set you should see your new user on the table. Make a note of the UID and GID (they’ll probably be 1000 and 100, respectively, if this is your first user other than the root) as you’ll need it when you install applications.

That was a lot! But, now you’ve set up Docker Compose. Now let’s use it to install some applications!

Setting up Media Server(s)

Before you set up the applications that access your data, you should make sure all of that data (i.e. photos you’ve taken, music you’ve downloaded, movies you’ve ripped / bought, PDFs you’d like to make available, etc.) are on your server and organized.

My suggestion is to set up a shared folder accessible to the network (mine is called Media) and have subdirectories in that folder corresponding for the different types of files that you may want your media server(s) to handle (for example: Videos, Photos, Files, etc). Then, use the network to move the files over (you should get comparable, if not faster, speeds as a USB transfer on a local area network).

The two media servers I’ve set up on my system are Plex (to serve videos, photos, and music) and Ubooquity (to serve files and especially ePUB/PDFs). There are other options out there, many of which can be similarly deployed using Docker compose, but I’m just going to cover my setup with Plex and Ubooquity below.

Plex

Why I chose it:
- I’ve been using Plex for many years now, having set up clients on virtually all of my devices (phones, tablets, computers, and smart TVs).
- I bought a lifetime Plex Pass a few years back which gives me access to even more functionality (including Intel Quick Sync transcode).
- It has a wealth of automatic features (i.e. automatic video detection and tagging, authenticated access through the web without needing to configure a VPN, etc.) that have worked reliably over the years.
- With a for-profit company backing it, (I believe) there’s a better chance that the platform will grow (they built a surprisingly decent free & ad-sponsored Live TV offering a few years ago) and be supported over the long-term
How to set up Docker Compose: Go to [Services > Compose > Files] and press the button. Under Name put down Plex and under File, paste the following (making sure the number of spaces are consistent)
```
version: "2.1"  
services:  
  plex:  
    image: lscr.io/linuxserver/plex:latest  
    container_name: plex  
    network_mode: host  
    environment:  
      - PUID=&lt;UID of Docker User&gt;  
      - PGID=&lt;GID of Docker User&gt;  
      - TZ=America/Los_Angeles  
      - VERSION=docker  
    devices:  
      - /dev/dri/:/dev/dri/  
    volumes:  
      - &lt;absolute path to shared config folder&gt;/plex:/config  
      - &lt;absolute path to Media folder&gt;:/media  
    restart: unless-stopped
```
You need to replace <UID of Docker User> and <GID of Docker User> with the UID and GID of the Docker user you created when you set up Docker Compose (Step 10 above), which will likely be 1000 and 100 if you followed the steps I laid out.

You can get the the absolute paths to your config folder and the location of your media files by going to [Storage > Shared Folders] in the administrative panel. I added a /plex to the config folder path under volumes:. This way you can install as many apps through Docker as you want and consolidate all of their configuration files in one place, while still keeping them separate.

If you have an Intel QuickSync CPU, the two lines that start with devices: and /dev/dri/ will allow Plex to use it (provided you also paid for a Plex Pass). If you don’t have a chip with Intel QuickSync, haven’t paid for Plex Pass, or don’t want it, leave out those two lines.

I live in the Bay Area so I set timezone TZ to America/Los_Angeles. You can find yours here.

Once you’re done, hit Save and you should be returned to your list of Docker compose files for the next step. Notice that the new Plex entry you created has a Down status, showing the container has yet to be initiated.
How to start / update / stop / remove your Plex container: You can manage all of your Docker Compose files by going to [Services > Compose > Files]. Click on the Plex entry (which should turn it yellow) and press the (up) button. This will create the container, download any files needed, and run it.

And that’s it! To prove it worked, go to http://your-ip-address:32400/web in a browser and you should see a login screen (see image below)

From time to time, you’ll want to update your software. Docker makes this very easy. Because of the image: lscr.io/linuxserver/plex:latest line, every time you press the (pull) button, Docker will pull the latest version from linuxserver.io (a group that maintains commonly used Linux containers) and, usually, you can get away with an update without needing to stop or restart your container.

Similarly, to stop the Plex container, simply tap the (stop) button. And to delete the container, tap the (down) button.
Getting started with Plex: There are great guides that have been written on the subject but my main recommendations are:
- Do the setup wizard. It has good default settings (automatic library scans, remote access, etc.) — and I haven’t had to make many tweaks.
- Take advantage of remote access — You can access your Plex server even when you’re not at home just by going to plex.tv and logging in.
- Install Plex clients everywhere — It’s available on pretty much everything (Web, iOS, Android) and, with remote access, becomes a pretty easy way to get access to all of your content
- I hide most of Plex’s default content in the Plex clients I’ve setup. While their ad-sponsored offerings are actually pretty good, I’m rarely consuming those. You can do this by configuring which things are pinned, and I pretty much only leave the things on my media server up.

Ubooquity

Why I chose it: Ubooquity has, sadly, not been updated in almost 5 years as of this writing. But, I still chose it for two reasons. First, unlike many alternatives, it does not require me to create a new file organization structure or manually tag my old files to work. It simply shows me my folder structure, lets me open the files one page at a time, maintains read location across devices, and lets me have multiple users.

Second, it’s available as a container on linuxserver.io (like Plex) which makes it easy to install and means that the infrastructure (if not the application) will continue to be updated as new container software comes out.

I may choose to switch (and the beauty of Docker is that it’s very easy to just install another content server to try it out) but for now Ubooquity made the most sense.
How to set up the Docker Compose configuration: Like with Plex, go to [Services > Compose > Files] and press the button. Under Name put down Ubooquity and under File, paste the following
```
---  
 version: "2.1"  
 services:  
   ubooquity:  
     image: lscr.io/linuxserver/ubooquity:latest  
     container_name: ubooquity  
     environment:  
       - PUID=&lt;UID of Docker User&gt;  
       - PGID=&lt;GID of Docker User&gt;  
       - TZ=America/Los_Angeles  
       - MAXMEM=512  
     volumes:  
       - &lt;absolute path to shared config folder&gt;/ubooquity:/config  
       - &lt;absolute path to shared Media folder&gt;/Books:/books  
       - &lt;absolute path to shared Media folder&gt;/Comics:/comics  
       - &lt;absolute path to shared Media folder&gt;/Files:/files  
     ports:  
       - 2202:2202  
       - 2203:2203  
     restart: unless-stopped
```
You need to replace <UID of Docker User> and <GID of Docker User> with the UID and GID of the Docker user you created when you set up Docker Compose (Step 10 above), which will likely be 1000 and 100 if you followed the steps I laid out.

You can get the the absolute paths to your config folder and the location of your media files by going to [Storage > Shared Folders] in the administrative panel. I added a /ubooquity to the config folder path under volumes:. This way you can install as many apps through Docker as you want and consolidate all of their configuration files in one place, while still keeping them separate.

I live in the Bay Area so I set timezone TZ to America/Los_Angeles. You can find yours here.

Once you’re done, hit Save and you should be returned to your list of Docker compose files for the next step. Notice that the Ubooquity entry you created has a Down status, showing it has yet to be initiated.
How to start / update / stop / remove your Ubooquity container: You can manage all of your Docker Compose files by going to [Services > Compose > Files]. Click on the Ubooquity entry (which should turn it yellow) and press the (up) button. This will create the container, download any files needed, and run the system.

And that’s it! To prove it worked, go to your-ip-address:2202/ubooquity in a browser and you should see the user interface (image credit: Ubooquity)

From time to time, you’ll want to update your software. Docker makes this very easy. Because of the image: lscr.io/linuxserver/ubooquity:latest line, every time you press the (pull) button, Docker will pull the latest version from linuxserver.io (a group that maintains commonly used Linux containers) and, usually, you can get away with an update without needing to stop or restart your container.

Similarly, to stop the Ubooquity container, simply tap the (stop) button. And to remove the container, tap the (down) button.
Getting started with Ubooquity: While Ubooquity will more or less work out of the box, if you want to really configure your setup you’ll need to go to the admin panel at your-ip-address:2203/ubooquity/admin (you will be prompted to create a password the first time)
- In the General tab, you can see how many files are tracked in the table at the top, configure how frequently Ubooquity scans your folders for new files under Automatic scan period, manually launch a scan if you just added files with Launch New Scan, and select a theme for the interface.
- If you want to create User accounts to have separate read state management or to segment which users can access specific content, you can create these users in the Security tab of the administrative panel. By doing so, you’ll need to manually go into the content type tabs (i.e. Comics, Books, Raw Files) and manually configure which users have access to which shared folders.
- The base Ubooquity interface is pretty dated so I am using a Plex-inspired theme.
  
  The easiest way to do this is to download the ZIP file at the link I gave. Unzip it on your computer (in this case it will result in the creation of a directory called plextheme-reading). Then, assuming the config shared folder you set up previously is shared across the network, take the unzipped directory and put it into the /ubooquity/themes subdirectory of the config folder.
  
  Lastly, go back to the General tab in Ubooquity admin and, next to Current theme select plextheme-reading
- Edit (10-Aug-2023): I’ve since switched to using a Local DNS service powered by Pihole to access Ubooquity using a human readable web address ubooquity.home that every device on my network can access. For information on how to do this, refer to my post on Setting Up Pihole, Nginx Proxy, and Twingate with OpenMediaVault
  ~~Because entering in a local ip address and remembering 2202 or 2203 and the folders afterwards is a pain, I created keyword shortcuts for these in Chrome. The instructions for doing this will vary by browser, but to do this in Chrome, go to chrome://settings/searchEngines. There is a section of the page called Site search. Press the Add button next to it. Even though the dialog box says Add Search Engine, in practice you can use this to add keywords to any URL, just put a name for the shortcut in the Search Engine field, the shortcut you want to use in Shortcut (I used ubooquity for the core application and ubooquityadmin for the administrative console) and the URLs in URL with %s in place of query (i.e. http://your-ip-address:2202/ubooquity and http://your-ip-address:2203/ubooquity/admin).
  
  Now to get to Ubooquity, I simply type in ubooquity in the Chrome address bar rather than a hodge podge of numbers and slashes that I’ll probably forget~~

External Access

One of Plex’s best features is making it very easy to access your media server even when you’re not on your home network. Having experienced that, I wanted the same level of access when I was out of the house to my network file share and applications like Ubooquity.

Edit (10-Aug-2023): I’ve since switched my method of granting external access to Twingate. This provides secure access to network resources without needing to configure Dynamic DNS, a VPN, or open up a port. For more information on how to do this, refer to my post on Setting Up Pihole, Nginx Proxy, and Twingate with OpenMediaVault

There are a few ways to do this, but the most secure path is through a VPN (virtual private network). VPNs are secure connections between computers that mimic actually being directly networked together. In our case, it lets a device securely access local network resources (like your server) even when it’s not on the home network.

OpenMediaVault makes it relatively easy to use Wireguard, a fast and popular VPN technology with support for many different types of devices. To set up Wireguard for your server for remote access, you’ll need to do six things:

~~Get a domain name and enable Dynamic DNS on it Most residential internet customers do not have a static IP. This means that the IP address for your home, as the rest of the world sees it, can change without warning. This makes it difficult to access externally (in much the same way that DHCP makes it hard to access your home server internally).

To address this, many domain providers offer Dynamic DNS, where a domain name (for example: myurl.com) can point to a different IP address depending on when you access it, so long as the domain provider is told what the IP address should be whenever it changes.

The exact instructions for how to do this will vary based on who your domain provider is. I use Namecheap and took an existing domain I owned and followed their instructions for enabling Dynamic DNS on it. I personally configured mine to use my vpn. subdomain, but you should use the setup you’d like, so long as you make a note of it for step 3 below.

If you don’t want to buy your own domain and are comfortable using someone else’s, you can also sign up for Duck DNS which is a free Dynamic DNS service tied to a Duck DNS subdomain.~~
~~Set up DDClient. To update the IP address your domain provider maps the domain to, you’ll need to run a background service on your server that will regularly check its IP address. One common way to do this is a software package called DDClient.

Thankfully, setting up DDClient is fairly easy thanks (again!) to a linuxserver.io container. Like with Plex & Ubooquity, go to [Services > Compose > Files] and press the button. Under Name put down DDClient and under File, paste the following
```
---  
 version: "2.1"  
 services:  
   ddclient:  
     image: lscr.io/linuxserver/ddclient:latest  
     container_name: ddclient  
     environment:  
       - PUID=&lt;UID of Docker User&gt;  
       - PGID=&lt;GID of Docker User&gt;  
       - TZ=America/Los_Angeles  
     volumes:  
       - &lt;absolute path to shared config folder&gt;/ddclient:/config  
     restart: unless-stopped
```
You need to replace <UID of Docker User> and <GID of Docker User> with the UID and GID of the Docker user you created when you set up Docker Compose (Step 10 above), which will likely be 1000 and 100 if you followed the steps I laid out.

You can get the the absolute path to your config folder by going to [Storage > Shared Folders] in the administrative panel. I added a /ddclient to the config folder path. This way you can install as many apps through Docker as you want and consolidate all of their configuration files in one place, while still keeping them separate.

I live in the Bay Area so I set timezone TZ to America/Los_Angeles. You can find yours here.

Once you’re done, hit Save and you should be returned to your list of Docker compose files. Click on the DDClient entry (which should turn it yellow) and press the (up) button. This will create the container, download any files needed, and run DDClient. Now, it is ready for configuration.~~
~~Configure DDClient to work with your domain provider. While the precise configuration of DDClient will vary by domain provider, the process will always involve editing a text file. To do this, login to your server using SSH or WeTTy (see the section above on Installing OMV-Extras) and enter into the command line:
```
nano &lt;absolute path to shared config folder&gt;/ddclient/ddclient.conf
```
Remember to substitute <absolute path to shared config folder> with the absolute path to the config folder you set up for your applications (which you can access by going to [Storage > Shared Folders] in the administrative panel).

This will open up Linux’s native text editor. Scroll to the very bottom and enter the configuration information that your domain provider requires for DynamicDNS to work. As I use Namecheap, I followed these instructions. In general, you’ll need to supply some type of information about the protocol, the server, your login / password for the domain provider, and the subdomain you intend to map to your IP address.

Then press Ctrl+X to exit, press Y when asked if you want to save, and finally Enter to confirm that you want to overwrite the old file.~~
~~Set up Port Forwarding on your router. Dynamic DNS gives devices outside of your network a consistent “address” to get to your server but it won’t do any good if your router doesn’t pass those external requests through. In this case, you’ll need to tell your router to let incoming UDP requests from port 51820 through to your server to line up with Wireguard’s defaults.

The exact instructions will vary by router so you’ll need to consult your router’s documentation. In my household, we use Google Wifi and, if you do too, here are the instructions for doing so.~~
~~Enable Wireguard. If you installed OMV-Extras above as I suggested, you’ll have access to a Plugin that turns on Wireguard. Go to [System > Plugins] on the administrative panel and then search or scroll down until you find the entry for openmediavault-wireguard. Click on it to mark it yellow and then press the button to install it.

Now go to [Services > Wireguard > Tunnels] and press the (create) button to set up a VPN tunnel. You can give it any Name you want (i.e. omv-vpn). Select your server’s main network connection for Network adapter. But, most importantly, under Endpoint, add the domain you just configured for DynamicDNS/DDClient (for example, vpn.myurl.com). Press Save~~
~~Set up Wireguard on your devices. With a Wireguard tunnel configured, your next step is to set up the devices (called clients or peers) to connect. This has two parts.

First, install the Wireguard applications on the devices themselves. Go to wireguard.com/install and download or set up the Wireguard apps. There are apps for Windows, MacOS, Android, iOS, and many flavors of Linux

Then, go back into your administrative panel and go to [Services > Wireguard > Clients] and press the (create) button to create a valid client for the VPN. Check the box next to Enable, select the tunnel you just created under Tunnel number, put a name for the device you’re going to connect under Name, and assign a unique (or it will not work) client number in Client Number . Press Save and you’ll be brought back to the Client list. Make sure to approve the change and then press the (client config) button. What you should do next depends on what kind of client device you’re configuring.

If the device you’re configuring is not a smartphone (i.e. a computer), copy the text that shows up in the Client Config popup that comes up and save that as a .conf file (for example: work_laptop_wireguard.conf). Send that file to the device in question as that file will be used by the Wireguard app on that device to configure and access the VPN. Hit Close when you’re done

If the device you’re configuring is a smartphone, hit Close button on the Client Config popup that comes up as you will be presented with a QR code that your smartphone Wireguard app can capture to configure the VPN connection.

Now go into your Wireguard app on the client device and use it to either take a picture of the QR code when prompted or load the .conf file. Your device is now configured to connect to your server securely no matter where you are. A good test of this is to disconnect a set up smartphone from your home WiFi and enable the VPN. Since you’re no longer on WiFi you should not be on the same network as your server. If you can enter http://your-ip-address in this mode into a browser and still reach the administrative panel for OpenMediaVault, you’re home free!

One additional note: by default, Wireguard also acts as a proxy, meaning all internet traffic you send from the device will be routed through the server. This can be valuable if you’re trying to access a blocked website or pretend to be from a different location, but it can also be unnecessarily slow (and bandwidth consuming). I have my Wireguard configured to only route traffic that is going to my server’s local IP address through Wireguard. You can do this by configuring your client device’s Allowed IPs to your-ip-address (for example: 192.168.99.99) from the Wireguard app.~~

Congratulations, you have now configured a file server and media server that you can securely access from anywhere!

Concluding Thoughts

A few concluding thoughts:

This was probably way too complicated for most people. Believe it or not, what was written above is a shortened version of what I went through. Even holding aside that use of the command line and Docker automatically makes this hard for many consumers, I still had to deal with missing drivers, Linux not recognizing my USB drive through the USB C port (but through the USB A one?), puzzling over different external access configurations (VPN vs Let’s Encrypt SSL on my server vs self-sign certificate), and minimal feedback when my initial attempts to use Wireguard failed. While I learned a great deal, for most people, it makes more sense to go completely third party (i.e. use Google / Amazon / Apple for everything) or, if you have some pain tolerance, with a high-end NAS.
Docker/containerization is extremely powerful. Prior to this, I had thought of Docker as just a “flavor” of virtual machine, a software technology underlying cloud computing which abstracts server software from server hardware. And, while there is some overlap, I completely misunderstood why containers were so powerful for software deployment. By using 3 fairly simple blocks of text, I was able to deploy 3 complicated applications which needed different levels of hardware and network access (Ubooquity, DDClient, Plex) in minutes without issue.
I was pleasantly surprised by how helpful the blogs and forums were. While the amount of work needed to find the right advice can be daunting, every time I ran into an issue, I was able to find some guidance online (often in a forum or subreddit). While there were certainly … abrasive personalities, by in large many of the questions being asked were by non-experts and they were answered by experts showing patience and generosity of spirit. Part of the reason I wrote this is to pay this forward for the next set of people who want to experiment with setting up their own server.
I am excited to try still more applications. Lists about what hobbyists are running on their home servers like this and this and this make me very intrigued by the possibilities. I’m currently considering a network-wide adblocker like Pi-Hole and backup tools like BorgBackup. There is a tremendous amount of creativity out there!

For more help on setting any of this stuff up, here are a few additional resources that proved helpful to me:

OMV Extras Wiki — probably the single best source of help I turned to in setting up OpenMediaVault, Docker Compose, and Wireguard
The OpenMediaVault Forum and r/OpenMediaVault have an amazing wealth of information
linuxserver.io documentation — Nice listing of available images and basics on how to run containers and configure those text files
Ubooquity website — sadly hasn’t been updated in a while but still useful information
the Debian SourcesList documentation page — this saved me when I was trying to figure out how to get Debian to update itself
Plex Support site
Wireguard website

(If you’re interested in how to setup a home server on OpenMediaVault or how to self-host different services, check out all my posts on the subject)

It’s not just the GOP who misunderstands Section 230

Sun, 07 Feb 2021 00:00:00 GMT

Source: NPR

Section 230 of the Communications Decency Act has been rightfully called “the twenty-six words that created the Internet.” It is a valuable legal shield which allows internet hosts and platforms the ability to distribute user-generated content and practice moderation without unreasonable fear of being sued, something which forms the basis of all social media, user review, and user forum, and internet hosting services.

In recent months, as big tech companies have drawn greater scrutiny for the role they play in shaping our discussions, Section 230 has become a scapegoat for many of the ills of technology. Until 2021, much of that criticism has come from the Republican Party who argue incorrectly that it promotes bias on platforms with President Trump even vetoing unrelated defense legislation because it did not repeal Section 230.

So, it’s refreshing (and distressing) to see the Democrats now take their turn in misunderstanding what Section 230 does for the internet. This critique is based mainly on Senator Mark Warner’s proposed changes to Section 230 and the FAQ his office posted about the SAFE TECH act he (alongside Senators Hirono and Klobuchar) is proposing but apply to many commentators from the Democratic Party and the press which seems to have misunderstood the practical implications and have received this positively.

While I think it’s reasonable to modify Section 230 to obligate platforms to help victims of clearly heinous acts like cyberstalking, swatting, violent threats, and human rights violations, what the Democratic Senators are proposing goes far beyond that in several dangerous ways.

First, Warner and his colleagues have proposed carving out from Section 230 all content which accompanies payment (see below). While I sympathize with what I believe was the intention (to put a different bar on advertisements), this is remarkably short-sighted, because Section 230 applies to far more than companies with ad / content moderation policies Democrats dislike such as Facebook, Google, and Twitter.

Source: Mark Warner’s “redlines” of Section 230; highlighting is mine

It also encompasses email providers, web hosts, user generated review sites, and more. Any service that currently receives payment (for example: a paid blog hosting service, any eCommerce vendor who lets users post reviews, a premium forum, etc) could be made liable for any user posted content. This would make it legally and financially untenable to host any potentially controversial content.

Secondly, these rules will disproportionately impact smaller companies and startups. This is because these smaller companies lack the resources that larger companies have to deal with the new legal burdens and moderation challenges that such a change to Section 230 would call for. It’s hard to know if Senator Warner’s glip answer in his FAQ that people don’t litigate small companies (see below) is ignorance or a willful desire to mislead, but ask tech startups how they feel about patent trolls and whether or not being small protects them from frivolous lawsuits

Source: Mark Warner’s FAQ on SAFE TECH Act; highlighting mine

Third, the use of the language “affirmative defense” and “injunctive relief” may have far-reaching consequences that go beyond minor changes in legalese (see below). By reducing Section 230 from an immunity to an affirmative defense, it means that companies hosting content will cease to be able to dismiss cases that clearly fall within Section 230 because they now have a “burden of [proof] by a preponderance of the evidence.”

Source: Mark Warner’s “redlines” of Section 230; highlighting is mine

Similarly, carving out “injunctive relief” from Section 230 protections (see below) means that Section 230 doesn’t apply if the party suing is only interested in taking something down (but not financial damages)

Source: Mark Warner’s “redlines” of Section 230

I suspect the intention of these clauses is to make it harder for large tech companies to dodge legitimate concerns, but what this practically means is that anyone who has the money to pursue legal action can simply tie up any internet company or platform hosting content that they don’t like.

That may seem like hyperbole, but this is what happened in the UK until 2014 where libel / slander laws making it easy for wealthy individuals and corporations to sue anyone for negative press due to weak protections. Imagine Jeffrey Epstein being able to sue any platform for carrying posts or links to stories about his actions or any individual for forwarding an unflattering email about him.

There is no doubt that we need new tools and incentives (both positive and negative) to tamp down on online harms like cyberbullying and cyberstalking, and that we need to come up with new and fair standards for dealing with “fake news”. But, it is distressing that elected officials will react by proposing far-reaching changes that show a lack of thoughtfulness as it pertains to how the internet works and the positives of existing rules and regulations.

It is my hope that this was only an early draft that will go through many rounds of revisions with people with real technology policy and technology industry expertise.

It’s Even Easier Now to Set up Let’s Encrypt on WordPress with AWS Lightsail

Sun, 31 Jan 2021 00:00:00 GMT

Amazon’s Lightsail service has made it remarkably cheap and simple for people hosting webpages (including this blog) and simple web applications to get access to high quality virtual private servers (VPS). Beyond the ability to do really stupid things with your VPS powers, my only real complaint with the experience thus far has been that getting HTTPS working with Let’s Encrypt’s free SSL certificates has been a a hassle, requiring a lot more DNS configuration and manual command-line tweaking than is really necessary.

Because of my recent server mishap, I discovered that Bitnami, creator of the WordPress “package” which Lightsail uses to get users up and running quickly with WordPress, has made it much simpler.

Whereas the previous experience using Let’s Encrypt’s certbot required:

Manually adjusting your DNS configuration with TXT records to confirm domain ownership (each renewal)
Manually copying of certificate files to the right directory
Requires you to manually setup URL redirection (i.e. moving your HTTP URLs to HTTPS, figuring out how to handle “www.” subdomain, etc)
Manual renewal every cycle (Let’s Encrypt certificates expire every 90 days)

Bitnami’s new command line tool (bncert) is vastly simpler to use:

Assuming you’ve already linked the domain you wish to enable HTTPS for to the public IP address for your server, the domain certification is handled automatically without any effort (if it’s not, you may need to add TXT records)
Automatically configures URL redirection for you based on what you enter when prompted
Automatically puts the certificates where they need to be
Automatically renews your certificates in 80 days (right before they expire)

The only downside I can see from the new bncert tool is that it does not support a wildcard certificate (which would work with any subdomain the way that certbot did), so if you’re planning on using multiple subdomains, it may be worth first planning out which domains you intend to use prior to using bncert (or to continue with certbot)

If you’re using a recent Bitnami WordPress image, bncert should already be present (and you can simply follow the instructions on the Lightsail documentation page and skip Step 4) — if not, you can either install the tool (Step 4 of the instructions) or (and this is the path I’d recommend) create a new Lightsail image and migrate your WordPress blog over. It’s a bit of a pain but it’s worth the effort as if you’ve been using an older version of Bitnami WordPress, there’s a good chance you’re using an out of date PHP installation. For help, here is a guide I wrote to how to move an old WordPress blog to a new WordPress setup.

Then the only remaining step is to change your WordPress Domain name in the wp-config.php file (so that your WordPress install thinks of itself as an https rather than an http domain). To do this, use the Lightsail command line interface and enter

sudo nano /opt/bitnami/wordpress/wp-config.php

This opens up a text editor (nano). Use the down key until you see a pair of lines that say:

define('WP_SITEURL', 'http://' . $_SERVER['HTTP_HOST'] . '/');
define('WP_HOME', 'http://' . $_SERVER['HTTP_HOST'] . '/');Code language: PHP (php)

Add an ‘s’ to the ‘http’ so that you end up with:

define('WP_SITEURL', 'https://' . $_SERVER['HTTP_HOST'] . '/');
define('WP_HOME', 'https://' . $_SERVER['HTTP_HOST'] . '/');Code language: PHP (php)

Hit Ctrl+X to exit and when it prompts you to “Save modified buffer?”, tap “Y” (for yes) and you are done.

Is Gamestop a Win for the Little Guy?

Thu, 28 Jan 2021 00:00:00 GMT

If you’ve been exposed to any financial news in the last few days, you’ll have heard of Gamestop, the mostly brick and mortar video gaming retailer who’s stock has been caught between many retail investors on the subreddit r/WallstreetBets and hedge fund Melvin Capital which had been actively betting against the company. The resulting short squeeze (where a rising stock price forces investors betting against a company to buy shares to cover their own potential losses — which itself can push the stock price even higher) has been amazing to behold with the worth of Gamestop shares increasing over 10-fold in a matter of months.

Source: Yahoo Finance (pulled 28 Jan 2021)

While it’s hard not to get swept up in the idea of “the little guy winning one over on a hedge fund”, the narrative that this is Main Street winning over Wall Street is overblown.

Chasing the big short 💎✊🏼 – #GME #HoldTheLine #wallstreetbets pic.twitter.com/E2J2IxE4GQ

— Synon (@nonysv) January 28, 2021

A brief sampling of #HoldTheLine on Twitter

First, speaking practically, it’s hard to argue that giving one hedge fund a black eye by making Gamestop executives & directors and large investment funds holding $100M’s of Gamestop prior to the increase wealthier is anyone winning anything over on Wall Street. And that’s not even accounting for the fact that hedge funds are usually managing a significant amount of money on behalf of pension funds and foundation / university endowments.

Winning one over on Wall Street? Created using Imgflip

Second, while the paper value of recent investments in Gamestop has clearly jumped through the roof, what these investors will actually “win” is unclear. Even holding aside short-term capital gains taxes that many retail investors are unclear on, the reality is that, to make money on an investment, you not only have to buy low, you have to successfully sell high. By definition, any company experiencing a short-squeeze is volume-limited — meaning that it’s the lack of sellers that is causing the increase in price (the only way to get someone to sell is to offer them a higher price). If the stock price changes direction, it could trigger a flood of investors flocking to sell to try to hold on to their gains which can create the opposite problem: too many people trying to sell relative to people trying to buy which can cause the price to crater.

Buy high sell low? Created using Imgflip

Regulatory and legal experts are better suited to weigh in on whether or not this constitutes market manipulation that needs to be regulated. For whatever it’s worth, I personally feel that Redditors egging each other on is no different than an institutional investor hyping their investments on cable TV.

But what is not in doubt is that these sorts of trades are extremely risky for all parties involved — whether you’re betting against a popular stock or trying to “hold the line” on a short-squeeze. For that reason, I’m sympathetic to the brokerages which are limiting investor activity in some of these speculative tickers.

While many retail investors view these restrictions as a move by Wall Street to screw the little guy, there’s a practical reality here that the brokerages are probably fearful of:

Lawsuits from investors, some of whom will eventually lose quite a bit of money here
SEC actions and punishments due to eventual outcry from investors losing money

This is the third reason I’m worried the Gamestop story will ultimately be a bad thing for Main Street. If the resulting lawsuits and/or regulatory actions cause brokerages to put more restrictions on investors, this could put additional friction on investors in terms of how they can participate in long-term wealth creation, something more households need given the paltry state of retirement savings.

I love stories of hedge funds facing the consequences of the risks they take on — but the idea that this is a clear win for Main Street is suspect (as is the idea that the right answer for most retail investors is to HODL through thick and through thin).

Thought this was interesting or helpful? Check out some of my other pieces on investing / finance.

Mea Culpa

Sun, 17 Jan 2021 00:00:00 GMT

Mea culpa.

I’ve been a big fan of moving my personal page over to AWS Lightsail. But, if I had one complaint, it’s the dangerous combination of (1) their pre-packaged WordPress image being hard to upgrade software on and (2) the training-wheel-lacking full root access that Lightsail gives to its customers. That combination led me to make some regrettable mistakes yesterday which resulted in the complete loss of my old blog posts and pages.

It’s the most painful when you know your problems are your own fault. Thankfully, with the very same AWS Lightsail, it’s easy enough to start up a new WordPress instance. With the help of site visit and search engine analytics, I’ve prioritized the most popular posts and pages to resurrect using Google’s cache.

Unfortunately, that process led to my email subscribers receiving way too many emails from me as I recreated each post. For that, I’m sorry — mea culpa — it shouldn’t happen again.

I’ve come to terms with the fact that I’ve lost the majority of the 10+ years of content I’ve created. But, I’ve now learned the value of systematically backing up things (especially my AWS Lightsail instance), and hopefully I’ll write some good content in the future to make up for what was lost.

Visualizing How Market Volatility Impacts Risk and Returns

Wed, 19 Aug 2020 00:00:00 GMT

S&P500 Performance for 2020 (Yahoo Finance), pulled Jan 17, 2021

2020 has seen the greatest market volatility in history for American stocks. The roller-coaster ride investors have experienced over the last 6 months included a steep ~33% single-month drop followed by a four-month bull market run taking the S&P500 back roughly to where it started.

While not usually so dramatic, volatility is a fact of life for investors. In researching how to create a long-term investment strategy that can cope with volatility, I found a lot of the writing on the subject unsatisfying for two reasons:

First, much of the writing on investment approaches leans heavily on historical comparisons (or “backtesting”). While it’s important to understand how a particular approach would play out in the past, it is dangerous to assume that volatility will always play out in the same way. For example, take a series of coin tosses. It’s possible that during the most recent 100 flips, the coin came up heads 10 times in a row. Relying mainly on backtesting this particular sequence of coin tosses could lead to conclusions that rely on a long sequences of heads always coming up. In a similar way, investment strategies that lean heavily on backtesting recent history may be well-situated for handling the 2008 crash and the 2010-2019 bull market but fall apart if the next boom or bust happens in a different way.

Second, much of the analysis on investment allocation is overly focused on arithmetic mean returns rather than geometric means. This sounds like a minor technical distinction, but to illustrate why it’s significant, imagine that you’ve invested $1,000 in a stock that doubled in the first year (annual return: 100%) and then halved the following year (annual return: -50%). Simple math shows that, since you’re back where you started, you experienced a return over those two years (in this case, the geometric mean return) of 0%. The arithmetic mean, on the other hand, comes back with a market-beating 25% return [1/2 x (100% + -50%)]! One of these numbers suggests this is an amazing investment and the other correctly calls it as a terrible one! Yet despite the fact that the arithmetic mean always overestimates the (geometric mean) return that an investor experiences, much of the practice of asset allocation and portfolio theory is still focused on arithmetic mean returns because they are easier to calculate and build precise analytical solutions around.

Visualizing a 40-Year Investment in the S&P500

To overcome these limitations, I used Monte Carlo simulations to visualize what volatility means for investment returns and risk. For simplicity, I assumed an investment in the S&P500 would see annual returns that look like a normal distribution based on how the S&P500 has performed from 1928 – 2019. I ran 100,000 simulations of 40 years of returns and looked at what sorts of (geometric mean) returns an investor would see.

This first chart below is a heatmap showing the likelihood that an investor will earn a certain return in each year (the darker the shade of blue, the more simulations wound up with that geometric return in that year).

Density Map of 40-Year Returns for Investment in S&P500
Densities are log (base 10)-adjusted; Assumes S&P500 returns are normally distributed (clipped from -90% to +100%) based on 1928-2019 annual returns; Years go from 0-39 (rather than 1-40)

This second chart below is a different view of the same data, calling out what the median return (the light blue-green line in the middle; where you have a 50-50 shot at doing better or worse) looks like. Going “outward” from the median line are lines representing the lower and upper bounds of the middle 50%, 70%, and 90% of returns.

Confidence Interval Map of 40-Year Return for Investment in S&P500
(from outside to middle) 90%, 70%, and 50% confidence interval + median investment returns. Assumes S&P500 returns are normally distributed (clipped from -90% to +100%) based on 1928-2019 annual returns

Finally, the third chart below captures the probability that an investment in the S&P500 over 40 years will result not in a loss (the darkest blue line at the top), will beat 5% (the second line), will beat 10% (the third line), and will beat 15% (the lightest blue line at the bottom) returns.

Probability 40-Year Investment in S&P500 will Exceed 0%, 5%, 10%, and 15% Returns
(from top to bottom/darkest to lightest) Probability that 40-year S&P500 returns simulation beat 0%, 5%, 10%, and 15% geometric mean return. Assumes S&P500 returns are normally distributed (clipped from -90% to +100%) based on 1928-2019 annual returns

The charts are a nice visual representation of what uncertainty / volatility mean for an investor and show two things.

First, the level of uncertainty around what an investor will earn declines the longer they can go without touching the investment. In the early years, there is a much greater spread in returns because of the high level of volatility in any given year’s stock market returns. From 1928 – 2019, stock markets saw returns ranging from a 53% increase to a 44% drop. Over time, however, reversion to the mean (a fancy way of saying a good or bad year is more likely to be followed by more normal looking years) narrows the variation an investor is likely to see. As a result, while the median return stays fairly constant over time (starting at ~11.6% in year 1 — in line with the historical arithmetic mean return of the market — but dropping slowly to ~10% by year 10 and to ~9.8% starting in year 30), the “spread” of returns narrows. In year 1, you would expect a return between -21% and 44% around 90% of the time. But by year 5, this narrows to -5% to 25%. By year 12, this narrows further to just above 0% to 19.4% (put another way, the middle 90% of returns does not include a loss). And at year 40, this narrows to 4.6% to 15%.

Secondly, **the risk an investor faces depends on the return threshold they “need”.**As the probability chart shows, if the main concern is about losing money over the long haul, then the risk of that happening starts relatively low (~28% in year 1) and drops rapidly (~10% in year 7, ~1% in year 23). If the main concern is about getting at least a 5% return, this too drops from ~37% in year 1 to ~10% by year 28. However, if one needs to achieve a return greater than the median (~9.8%), then the probability gets worse over time and gets worse the greater the return threshold needed. To beat a 15% return, in year 1, there is a ~43% chance that this will happen. But this rapidly shrinks to ~20% by year 11, ~10% by year 24, and ~5% by year 40.

The Impact of Increasing Average Annual Return

These simulations are a useful way to explore how long-term returns vary. Let’s see what happens if we increase the (arithmetic) average annual return by 1% from the S&P500 historical average.

As one might expect, the heatmap for returns (below) generally looks about the same:

Density Map of 40-Year Returns for Higher Average Annual Return Investment
Densities are log (base 10)-adjusted; Assumes an asset with normally distributed annual returns (clipped from -90% to +100%) based on 1928-2019 S&P500 annual returns but with 1% higher mean. Years go from 0-39 (rather than 1-40)

Looking more closely at the contour lines and overlaying them with the contour lines of the original S&P500 distribution (below, green is the new, blue the old), it looks like all the lines have roughly the same shape and spread, but have just been shifted upward by ~1%.

Confidence Interval Map of 40-Year Return for Higher Average Return Investment (Green) vs. S&P500 (Blue)
(from outside to middle/darkest to lightest) 90%, 50% confidence interval, and median investment returns for S&P500 (blue lines; assuming normal distribution clipped from -90% to +100% based on 1928-2019 annual returns) and hypothetical investment with identical variance but 1% higher mean (green lines)

This is reflected in the shifts in the probability chart (below). The different levels of movement correspond to the impact an incremental 1% in returns makes to each scenario. For fairly low returns (i.e. the probability of a loss), the probability will not change much as it was low to begin with. Similarly, for fairly high returns (i.e., 15%), adding an extra 1% is unlikely to make you earn vastly above the median. On the other hand, for returns that are much closer to the median return, the extra 1% will have a much larger relative impact on an investment’s ability to beat those moderate return thresholds.

Probability Higher Average Return Investment (Green) and S&P500 (Blue) will Exceed 0%, 5%, 10%, and 15% Returns
(from top to bottom/darkest to lightest) Probability that 40-year S&P500 returns simulation beat 0%, 5%, 10%, and 15% geometric mean return. Assumes S&P500 returns are normally distributed (clipped from -90% to +100%) based on 1928-2019 annual returns. Higher average return investment is a hypothetical asset with identical variance but 1% higher mean

Overall, there isn’t much of a surprise from increasing the mean: returns go up roughly in line with the change and the probability that you beat different thresholds goes up overall but more so for moderate returns closer to the median than the extremes.

What about volatility?

The Impact of Decreasing Volatility

Having completed the prior analysis, I expected that tweaking volatility (in the form of adjusting the variance of the distribution) would result in preserving the basic distribution shape and position but narrowing or expanding it’s “spread”. However, I was surprised to find that adjusting the volatility didn’t just impact the “spread” of the distribution, it impacted the median returns as well!

Below is the returns heatmap for an investment that has the same mean as the S&P500 from 1928-2019 but 2% lower variance. A quick comparison with the first heat/density map shows that, as expected, the overall shape looks similar but is clearly narrower.

Density Map of 40-Year Returns for Low Volatility Investment
Densities are log (base 10)-adjusted; Assumes S&P500 returns are normally distributed (clipped from -90% to +100%) based on 1928-2019 annual returns but with 2% lower variance. Years go from 0-39 (rather than 1-40)

Looking more closely at the contour lines (below) of the new distribution (in red) and comparing with the original S&P500 distribution (in blue) reveals, however, that the difference is more than just in the “spread” of returns, but in their relative position as well! The red lines are all shifted upward and the upward shift seems to increase over time. It turns out a ~2% decrease in variance appears to buy a 1% increase in the median return and a 1.5% increase in the lower bound of the 50% confidence interval at year 40!

Confidence Interval Map of 40-Year Return for Low Volatility Investment (Red) vs. S&P500 (Blue)
Left: (from outside to middle/darkest to lightest) 90%, 50% confidence interval, and median investment returns for S&P500 (blue lines; assuming normal distribution clipped from -90% to +100% based on 1928-2019 annual returns) and hypothetical investment with identical mean but 2% lower variance (red lines).
Right: Zoomed-in look just at the median lines and 50% confidence interval bounds

The probability comparison (below) makes the impact of this clear. With lower volatility, not only is an investor better able to avoid a loss / beat a moderate 5% return (the first two red lines having been meaningfully shifted upwards from the first two blue lines), but by raising the median return, the probability of beating a median-like return (10%) gets better over time as well! The one area the lower volatility distribution under-performs the original is in the probability of beating a high return (15%). This too makes sense — because the hypothetical investment experiences lower volatility, it becomes less likely to get the string of high returns needed to consistently beat the median over the long term.

Probability Low Volatility Investment (Red) and S&P500 (Blue) will Exceed 0%, 5%, 10%, and 15% Returns
(from top to bottom/darkest to lightest) Probability that 40-year S&P500 returns simulation beat 0%, 5%, 10%, and 15% geometric mean return. Assumes S&P500 returns are normally distributed (clipped from -90% to +100%) based on 1928-2019 annual returns. Low volatility investment is a hypothetical asset with identical mean but 2% lower variance

The Risk-Reward Tradeoff

Unfortunately, it’s not easy to find a “S&P500 but less volatile” or a “S&P500 but higher return”. In general, higher returns tend to go with greater volatility and vice versa.

While the exact nature of the tradeoff will depend on the specific numbers, to see what happens when you combine the two effects, I charted out the contours and probability curves for two distributions with roughly the same median return (below): one investment with a higher return (+1%) and higher volatility (+2% variance) than the S&P500 and another with a lower return (-1%) and lower volatility (-2% variance) than the S&P500:

Confidence Interval Map for Low Volatility/Low Return (Purple) vs. High Volatility/High Return (Gray)
Left: (from outside to middle/darkest to lightest) 90%, 50% confidence interval, and median investment returns for hypothetical investment with 1% higher mean and 2% higher variance than S&P500 (gray) and one with 1% lower mean and 2% lower variance than S&P500 (purple). Both returns assume normal distribution clipped from -90% to +100% with mean/variance based on 1928-2019 annual returns for S&P500.
Right: Zoomed-in look just at the median lines and 50% confidence interval bounds

Probability Low Volatility/Low Return (Purple) vs. High Volatility/High Return (Gray) Exceed 0%, 5%, 10%, and 15% Returns
(from top to bottom/darkest to lightest) Probability that 40-year returns simulation for hypothetical investment with 1% higher mean and 2% higher variance than S&P500 (gray) and one with 1% lower mean and 2% lower variance than S&P500 (purple) beat 0%, 5%, 10%, and 15% geometric mean return. Both returns assume normal distribution clipped from -90% to +100% with mean/variance based on 1928-2019 annual returns for S&P500.

The results show how two different ways of targeting the same long-run median return compare. The lower volatility investment, despite the lower (arithmetic) average annual return, still sees a much improved chance of avoiding loss and clearing the 5% return threshold. On the other hand, the higher return investment has a distinct advantage at outperforming the median over the long term and even provides a consistent advantage in beating the 10% return threshold close to the median.

Takeaways

The simulations above made it easy to profile unconventional metrics (geometric mean returns and the probability to beat different threshold returns) across time without doing a massive amount of hairy, symbolic math. By charting out the results, they also helped provide a richer, visual understanding of investment risk that goes beyond the overly simple and widely held belief that “volatility is the same thing as risk”:

**Time horizon matters as uncertainty in returns decreases with time:**As the charts above showed, “reversion to the mean” reduces the uncertainty (or “spread”) in returns over time. What this means is that the same level of volatility can be viewed wildly differently by two different investors with two different time horizons. An investor who needs the money in 2 years could find one level of variance unbearably bumpy while the investor saving for a goal 20 years away may see it very differently.
The investment return “needed” is key to assessing risk: An investor who needs to avoid a loss at all costs should have very different preferences and assessments of risk level than an investor who must generate higher returns in order to retire comfortably, even at the same time. The first investor should prioritize lower volatility investments and longer holding periods, while the latter should prioritize higher volatility investments and shorter holding periods. It’s not just a question of personal preferences about gambling & risk, as much of the discussion on risk tolerance seems to suggest, because the same level of volatility should rationally be viewed differently by different investors with different financial needs.
**Volatility impacts long-run returns:**Higher volatility decreases long-term median returns, and lower volatility increases long-term returns. From some of my own testing, this seems to happen at roughly a 2:1 ratio (where a 2% increase in variance decreases median returns by 1% and vice versa — at least for values of return / variance near the historical values for S&P500). The result is that understanding volatility is key to formulating the right investment approach, and it creates an interesting framework with which to evaluate how much to hold of lower risk/”riskless” things like cash and government bonds.

What’s Next

Having demonstrated how simulations can be applied to get a visual understanding of investment decisions and returns, I want to apply this analysis to other problems. I’d love to hear requests for other questions of interest, but for now, I plan to look into:

Diversification
Rebalancing
Withdrawal levels
Dollar cost averaging
Asset allocation
Alternative investment return distributions

Thought this was interesting or helpful? Check out some of my other pieces on investing / finance.

What I’ve Changed My Mind on Over the 2010s

Wed, 01 Jan 2020 00:00:00 GMT

It me. (Source: They Can Talk)

I’ve been reading a lot of year-end/decade-end reflections (as one does this time of year) — and while a part of me wanted to #humblebrag about how I got a 🏠/💍/👶🏻 this decade 😇 — I thought it would be more interesting & profound to instead call out 10 worldviews & beliefs I had going into the 2010s that I no longer hold.

Sales is an unimportant skill relative to hard work / being smart
As a stereotypical “good Asian kid” 🤓, I was taught to focus on nailing the task. I still think that focus is important early in one’s life & career, but this decade has made me realize that everyone, whether they know it or not, has to sell — you sell to employers to hire you, academics/nonprofits sell to attract donors and grant funding, even institutional investors have to sell to their investors/limited partners. Its a skill at least as important (if not more so).
Marriage is about finding your soul-mate and living happily ever after
Having been married for slightly over half the decade, I’ve now come to believe that marriage is less about finding the perfect soul-mate (the “Hollywood version”) as it is about finding a life partner who you can actively choose to celebrate (despite and including their flaws, mistakes, and baggage). Its not that passionate love is unimportant, but its hard to rely on that alone to make a lifelong partnership work. I now believe that really boring-sounding things like how you make #adulting decisions and compatibility of communication style matter a lot more than things usually celebrated in fiction like the wedding planning, first dates, how nice your vacations together are, whether you can finish each other’s sentences, etc.
Industrial policy doesn’t work
I tend to be a big skeptic of big government policy — both because of unintended consequences and the risks of politicians picking winners. But, a decade of studying (and working with companies who operate in) East Asian economies and watching how subsidies and economies of scale have made Asia the heart of much of advanced manufacturing have forced me to reconsider. Its not that the negatives don’t happen (there are many examples of China screwing things up with heavy-handed policy) but its hard to seriously think about how the world works without recognizing the role that industrial policy played. For more on how land management and industrial policies impacted economic development in different Asian countries, check out Joe Studwell’s book How Asia Works
Obesity & weight loss are simple — its just calories in & calories out
From a pure physics perspective, weight gain is a “simple” thermodynamic equation of “calories in minus calories out”. But in working with companies focused on dealing with prediabetes/obesity, I’ve come to appreciate that this “logic” not only ignores the economic and social factors that make obesity a public health problem, it also overlooks that different kinds of foods drive different physiological responses. As an example that just begins to scratch the surface, one very well-controlled study (sadly, a rarity in the field) published in July showed that, even after controlling for exercise and calories, carbs, fat, fiber, and other nutrients present in a meal, diets consisting of processed foods resulted in greater weight-gain than a diet consisting of unprocessed foods
Revering luminaries & leaders is a good thing
Its very natural to be so compelled by an idea / movement that you find yourself idolizing the people spearheading it. The media feeds into this with popular memoirs & biographies and numerous articles about how you can think/be/act more like [Steve Jobs/Jeff Bezos/Warren Buffett/Barack Obama/etc]. But, over the past decade, I’ve come to feel that this sort of reverence leads to a pernicious laziness of thought. I can admire Steve Jobs for his brilliance in product design but do I want to copy his approach to management or his use of alternative medicine to treat his cancer or condoning how he treated his illegitimate daughter. I think its far better to appreciate an idea and the work of the key people behind it than to equate the piece of work with the person and get sucked in to that cult of personality.
Startups are great place for everyone
Call it being sucked into the Silicon valley ethos but for a long time I believed that startups were a great place for everyone to build a career: high speed path to learning & responsibility, ability to network with other folks, favorable venture funding, one of the only paths to getting stock in rapidly growing companies, low job seeking risk (since there’s an expectation that startups often fail or pivot). Several years spent working in VC and startups later, and, while I still agree with my list above, I’ve come to believe that startups are really not a great place for most people. The risk-reward is generally not great for all but the earliest of employees and the most successful of companies, and the “startups are great for learning” Kool-aid is oftentimes used to justify poor management and work practices. I still think its a great place for some (i.e. people who can tolerate more risk [b/c of personal wealth or a spouse with a stable high-paying job], who are knowingly optimizing for learning & responsibility, or who are true believers in a startup’s mission), but I frankly think most people don’t fit the bill.
Microaggressions are just people being overly sensitive
I’ve been blessed at having only rarely faced overt racism (telling me to go back to China 🙄 / or that I don’t belong in this country). It’s a product of both where I’ve spent most of my life (in urban areas on the coasts) and my career/socioeconomic status (it’s not great to be overtly racist to a VC you’re trying to raise money from). But, having spent some dedicated time outside of those coastal areas this past decade and speaking with minorities who’ve lived there, I’ve become exposed to and more aware of “microaggressions”, forms of non-overt prejudice that are generally perpetrated without ill intent: questions like ‘so where are you really from?’ or comments like ‘you speak English really well!’. I once believed people complaining about these were simply being overly sensitive, but I’ve since become an active convert to the idea that, while these are certainly nowhere near as awful as overt hate crimes / racism, they are their own form of systematic prejudice which can, over time, grate and eat away at your sense of self-worth.
The Western model (liberal democracy, free markets, global institutions) will reign unchallenged as a model for prosperity
I once believed that the Western model of (relatively) liberal democracy, (relatively) free markets, and US/Europe-led global institutions was the only model of prosperity that would reign falling the collapse of the Soviet Union. While I probably wouldn’t have gone as far as Fukuyama did in proclaiming “the end of history”, I believed that the world was going to see authoritarian regimes increasingly globalize and embrace Western institutions. What I did not expect was the simultaneous rise of different models of success by countries like China and Saudi Arabia (who, frighteningly, now serve as models for still other countries to embrace), as well as a lasting backlash within the Western countries themselves (i.e. the rise of Trump, Brexit, “anti-globalism”, etc). This has fractured traditional political divides (hence the soul-searching that both major parties are undergoing in the US and the UK) and the election of illiberal populists in places like Mexico, Brazil, and Europe.
Strategy trumps execution
As a cerebral guy who spent the first years of his career in the last part of the 2000s as a strategy consultant, it shouldn’t be a surprise that much of my focus was on formulating smart business strategy. But having spent much of this decade focused on startups as well as having seen large companies like Apple, Amazon, and Netflix brilliantly out-execute companies with better ‘strategic positioning’ (Nokia, Blackberry, Walmart, big media), I’ve come around to a different understanding of how the two balance each other.
We need to invent radically new solutions to solve the climate crisis
Its going to be hard to do this one justice in this limited space — especially since I net out here very differently from Bill Gates — but going into this decade, I never would have expected that the cost of new solar or wind energy facilities could be cheaper than the cost of operating an existing coal plant. I never thought that lithium batteries or LEDs would get as cheap or as good as they are today (with signs that this progress will continue) or that the hottest IPO of the year would be an alternative food technology company (Beyond Meat) which will play a key role in helping us mitigate food/animal-related emissions. Despite the challenges of being a cleantech investor for much of the decade, its been a surprising bright spot to see how much pure smart capital and market forces have pushed many of the technologies we need. I still think we will need new policies and a huge amount of political willpower — I’d also like to see more progress made on long-duration energy storage, carbon capture, and industrial — but whereas I once believed that we’d need radically new energy technologies to thwart the worst of climate change, I am now much more of an optimist here than I was when the decade started.

Here’s to more worldview shifts in the coming decade!

Calculating the Financial Returns to College

Mon, 13 May 2019 00:00:00 GMT

Despite the recent spotlight on the staggering $1.5 trillion in student debt that 44 million Americans owe in 2019, there has been surprisingly little discussion on how to measure the value of a college education relative to its rapidly growing price tag (which is the reason so many take on debt to pay for it).

Source: US News

While it’s impossible to quantify all the intangibles of a college education, the tools of finance offers a practical, quantitative way to look at the tangible costs and benefits which can shed light on (1) whether to go to college / which college to go to, (2) whether taking on debt to pay for college is a wise choice, and (3) how best to design policies around student debt.

The below briefly walks through how finance would view the value of a college education and the soundness of taking on debt to pay for it and how it can help guide students / families thinking about applying and paying for colleges and, surprisingly, how there might actually be too little college debt and where policy should focus to address some of the issues around the burden of student debt.

The Finance View: College as an Investment

Through the lens of finance, the choice to go to college looks like an investment decision and can be evaluated in the same way that a company might evaluate investing in a new factory. Whereas a factory turns an upfront investment of construction and equipment into profits on production from the factory, the choice to go to college turns an upfront investment of cash tuition and missed salary while attending college into higher after-tax wages.

Finance has come up with different ways to measure returns for an investment, but one that is well-suited here is the internal rate of return (IRR). The IRR boils down all the aspects of an investment (i.e., timing and amount of costs vs. profits) into a single percentage that can be compared with the rates of return on another investment or with the interest rate on a loan. If an investment’s IRR is higher than the interest rate on a loan, then it makes sense to use the loan to finance the investment (i.e., borrowing at 5% to make 8%), as it suggests that, even if the debt payments are relatively onerous in the beginning, the gains from the investment will more than compensate for it.

To gauge what these returns look like, I put together a financial model in Google Sheets which generated the figures and charts below (this article in Investopedia explains the math in greater detail). I used publicly available data around wages (from the 2017 Current Population Survey, GoBankingRate’s starting salaries by school, and National Association of Colleges and Employer’s starting salaries by major), tax brackets (using the 2018 income tax), and costs associated with college (from College Board’s statistics [PDF] and the Harvard admissions website). To simplify the comparisons, I assumed a retirement age of 65, and that nobody gets a degree more advanced than a Bachelor’s.

Interactive Spreadsheet Model

College Degree Financial Returns & IRR Model

Interactive discounted cash flow workbook modeling college as a capital investment: incorporates CPS wage distributions, age-matched wage trajectories, federal income tax brackets, tuition net costs, and loan amortization.

Tabs: Overview & Scenarios CPS Wage Tables (2017) Net Tuition & Costs IRR & Payback Sensitivity

To give an example: if Sally Student can get a starting salary after college in line with the average salary of an 18-24 year old Bachelor’s degree-only holder ($47,551), would have earned the average salary of an 18-24 year old high school diploma-only holder had she not gone to college ($30,696), and expects wage growth similar to what age-matched cohorts saw from 1997-2017, then the IRR of a 4-year degree at a non-profit private school if Sally pays the average net (meaning after subtracting grants and tax credits) tuition, fees, room & board ($26,740/yr in 2017, or a 4-year cost of ~$106,960), the IRR of that investment in college would be 8.1%.

How to Benchmark Rates of Return

Is that a good or a bad return? Well, in my opinion, 8.1% is pretty good. Its much higher than what you’d expect from a typical savings account (~0.1%) or a CD or a Treasury Bond (as of this writing), and is also meaningfully higher than the 5.05% rate charged for federal subsidized loans for 2018-2019 school year — this means borrowing to pay for college would be a sensible choice. That being said, its not higher than the stock market (the S&P500 90-year total return is ~9.8%) or the 20% that you’d need to get into the top quartile of Venture Capital/Private Equity funds [PDF].

What Drives Better / Worse Rates of Return

Playing out different scenarios shows which factors are important in determining returns. An obvious factor is the cost of college:

T&F: Tuition & Fees; TFR&B: Tuition, Fees, Room & Board
List: Average List Price; Net: Average List Price Less Grants and Tax Benefits
Blue: In-State Public; Green: Private Non-Profit; Red: Harvard

As evident from the chart, there is huge difference between the rate of return Sally would get if she landed the same job but instead attended an in-state public school, did not have to pay for room & board, and got a typical level of financial aid (a stock-market-beating IRR of 11.1%) versus the world where she had to pay full list price at Harvard (IRR of 5.3%). In one case, attending college is a fantastic investment and Sally borrowing money to pay for it makes great sense (investors everywhere would love to borrow at ~5% and get ~11%). In the other, the decision to attend college is less straightforward (financially), and it would be very risky for Sally to borrow money at anything near subsidized rates to pay for it.

Some other trends jump out from the chart. **Attending an in-state public university improves returns for the average college wage-earner by 1-2%**compared with attending private universities (comparing the blue and green bars). Getting an average amount of financial aid (paying net vs list) also seems to improve returns by 0.7-1% for public schools and 2% for private.

As with college costs, the returns also understandably vary by starting salary:

There is a night and day difference between the returns Sally would see making $40K per year (~$10K more than an average high school diploma holder) versus if she made what the average Caltech graduate does post-graduation (4.6% vs 17.9%), let alone if she were to start with a six-figure salary (IRR of over 21%). If Sally is making six figures, she would be making better returns than the vast majority of venture capital firms, but if she were starting at $40K/yr, her rate of return would be lower than the interest rate on subsidized student loans, making borrowing for school financially unsound.

Time spent in college also has a big impact on returns:

Graduating sooner not only reduces the amount of foregone wages, it also means earning higher wages sooner and for more years. As a result, if Sally graduates in two years while still paying for four years worth of education costs, she would experience a higher return (12.6%) than if she were to graduate in three years and save one year worth of costs (11.1%)! Similarly, if Sally were to finish school in five years instead of four, this would lower her returns (6.3% if still only paying for four years, 5.8% if adding an extra year’s worth of costs). The result is that an extra / less year spent in college is a ~2% hit / boost to returns!

Finally, how quickly a college graduate’s wages grow relative to a high school diploma holder’s also has a significant impact on the returns to a college education:

Census/BLS data suggests that, between 1997 and 2017, wages of bachelor’s degree holders grew faster on an annualized basis by ~0.7% per year than for those with only a high school diploma (6.7% vs 5.8% until age 35, 4.0% vs 3.3% for ages 35-55, both sets of wage growth appear to taper off after 55).

The numbers show that if Sally’s future wages grew at the same rate as the wages of those with only a high school diploma, her rate of return drops to 5.3% (just barely above the subsidized loan rate). On the other hand, if Sally’s wages end up growing 1% faster until age 55 than they did for similar aged cohorts from 1997-2017, her rate of return jumps to a stock-market-beating 10.3%.

Lessons for Students / Families

What do all the charts and formulas tell a student / family considering college and the options for paying for it?

First, college can be an amazing investment, well worth taking on student debt and the effort to earn grants and scholarships. While there is well-founded concern about the impact that debt load and debt payments can have on new graduates, in many cases, the financial decision to borrow is a good one. Below is a sensitivity table laying out the rates of return across a wide range of starting salaries (the rows in the table) and costs of college (the columns in the table) and color codes how the resulting rates of return compare with the cost of borrowing and with returns in the stock market (red: risky to borrow at subsidized rates; white: does make sense to borrow at subsidized rates but it’s sensible to be mindful of the amount of debt / rates; green: returns are better than the stock market).

Except for graduates with well below average starting salaries (less than or equal to $40,000/yr), most of the cells are white or green. At the average starting salary, except for those without financial aid attending a private school, the returns are generally better than subsidized student loan rates. For those attending public schools with financial aid, the returns are better than what you’d expect from the stock market.

Secondly, there are ways to push returns to a college education higher. They involve effort and sometimes painful tradeoffs but, financially, they are well worth considering. Students / families choosing where to apply or where to go should keep in mind costs, average starting salaries, quality of career services, and availability of financial aid / scholarships / grants, as all of these factors will have a sizable impact on returns. After enrollment, student choices / actions can also have a meaningful impact: graduating in fewer semesters/quarters, taking advantage of career resources to research and network into higher starting salary jobs, applying for scholarships and grants, and, where possible, going for a 4th/5th year masters degree can all help students earn higher returns to help pay off any debt they take on.

Lastly, use the spreadsheet*! The figures and charts above are for a very specific set of scenarios and don’t factor in any particular individual’s circumstances or career trajectory, nor is it very intelligent about selecting what the most likely alternative to a college degree would be. These are all factors that are important to consider and may dramatically change the answer.

*To use the Google Sheet, you must be logged into a Google account; use the “Make a Copy” command in the File menu to save a version to your Google Drive and edit the tan cells with red numbers in them to whatever best matches your situation and see the impact on the yellow highlighted cells for IRR and the age when investment pays off

Implications for Policy on Student Debt

Given the growing concerns around student debt and rising tuitions, I went into this exercise expecting to find that the rates of return across the board would be mediocre for all but the highest earners. I was (pleasantly) surprised to discover that a college graduate earning an average starting salary would be able to achieve a rate of return well above federal loan rates even at a private (non-profit) university.

While the rate of return is not a perfect indicator of loan affordability (as it doesn’t account for how onerous the payments are compared to early salaries), the fact that the rates of return are so high is a sign that, contrary to popular opinion, there may actually be too little student debt rather than too much, and that the right policy goal may actually be to find ways to encourage the public and private sector to make more loans to more prospective students.

As for concerns around affordability, while proposals to cancel all student debt plays well to younger voters, the fact that many graduates are enjoying very high returns suggests that such a blanket policy is likely unnecessary, anti-progressive (after all, why should the government zero out the costs on high-return investments for the soon-to-be upper and upper-middle-classes), and fails to address the root cause of the issue (mainly that there shouldn’t be institutions granting degrees that fail to be good financial investments). Instead, a more effective approach might be:

Require all institutions to publish basic statistics (i.e. on costs, availability of scholarships/grants, starting salaries by degree/major, time to graduation, etc.) to help students better understand their own financial equation
Hold educational institutions accountable when too many students graduate with unaffordable loan burdens/payments (i.e. as a fraction of salary they earn and/or fraction of students who default on loans) and require them to make improvements to continue to qualify for federally subsidized loans
Making it easier for students to discharge student debt upon bankruptcy and increasing government oversight of collectors / borrower rights to prevent abuse
Government-supported loan modifications (deferrals, term changes, rate modifications, etc.) where short-term affordability is an issue (but long-term returns story looks good); loan cancellation in cases where debt load is unsustainable in the long-term (where long-term returns are not keeping up) or where debt was used for an institution that is now being denied new loans due to unaffordability
Making the path to public service loan forgiveness (where graduates who spend 10 years working for non-profits and who have never missed an interest payment get their student loans forgiven) clearer and addressing some of the issues which have led to 99% of applications to date being rejected

Special thanks Sophia Wang, Kathy Chen, and Dennis Coyle for reading an earlier version of this and sharing helpful comments!

Thought this was interesting or helpful? Check out some of my other pieces on investing / finance.

Lyft vs Uber: A Tale of Two S-1’s

Sun, 14 Apr 2019 00:00:00 GMT

You can learn a great deal from reading and comparing the financial filings of two close competitors. Tech-finance nerd that I am, you can imagine how excited I was to see Lyft’s and Uber’s respective S-1’s become public within mere weeks of each other.

While the general financial press has covered a lot of the top-level figures on profitability (or lack thereof) and revenue growth, I was more interested in understanding the unit economics — what is the individual “unit” (i.e. a user, a sale, a machine, etc.) of the business and what does the history of associated costs and revenues say about how the business will (or will not) create durable value over time.

For two-sided regional marketplaces like Lyft and Uber, an investor should understand the full economic picture for (1) the users/riders, (2) the drivers, and (3) the regional markets. Sadly, their S-1’s don’t make it easy to get much on (2) or (3) — probably because the companies consider the pertinent data to be highly sensitive information. They did, however, provide a fair amount of information on users/riders and rides and, after doing some quantitative modeling, a couple of interesting findings emerged:

Interactive Spreadsheet Model

Lyft vs. Uber S-1 Marketplace Unit Economics Model

Comparative financial workbook breaking down quarterly S-1 metrics from 2016 to 2018: bookings per ride, monthly rides per user, revenue monetization rates, and take-rate trajectories.

Tabs: Lyft Quarterly Data Uber Quarterly Data Comparative Take Rates