🍔🧠 How DoorDash Replaced Elasticsearch for 50% Faster Searches

Happy Monday! ☀️

Welcome to the 738 new hungry minds who have joined us since last Monday!

If you aren't subscribed yet, join smart, curious, and hungry folks by subscribing here.

📚 Software Engineering Articles

• The last AI masterclass you’ll ever need (it’s free with my link!)

• Master domain-driven design to bridge the technical and business worlds

• Learn 15 proven techniques to scale your database effectively

• Discover hidden git secrets that will supercharge your workflow

• Visual patterns that make code hard to read

• Seven key behaviors distinguishing junior from senior engineers

🗞️ Tech and AI Trends

• Anthropic's valuation soars to $61.5B in latest funding round

• NASA achieves historic first GPS navigation on the moon

• Google tests AI-only search results, marking a major shift

👨🏻‍💻 Coding Tip

• Use torch.autograd.functional.jacobian in PyTorch to compute derivatives for neural networks

Time-to-digest: 5 minutes

Big thanks to our partners for keeping this newsletter free.

If you have a second, clicking the ad below helps us a ton—and who knows, you might find something you love. 💚

Make 2025 your Best Year With This $399 FREE AI Masterclass

AI is reshaping the world— and you need to master it before it masters you!

A 3-hour power-packed workshop covering 25+ AI tools, mastering prompts, and revealing hacks, strategies, and secrets of the top 1%.

Save your seat now (Offer valid for 24 hours only ⏰)

Here’s why you can’t miss this:

✅ Learn 30+ cutting-edge AI tools 
✅ Automate your workflows and reclaim 20+ hours a week
✅ Create an AI-powered version of YOU

⏲️ Time: 10 AM EST (Tomorrow)

Register here (first 100 people get it for free + $500 bonus) 🎁

DoorDash Built Their Own Search Engine and Crushed It 🔍️ 

In 2022, DoorDash faced a critical scaling problem: their Elasticsearch-based global search couldn't keep up with growth, especially as they expanded from store-only to hybrid item-and-store search.
Rather than patching the existing system, they boldly created their own search engine from scratch using Apache Lucene as the foundation.

The challenge:

Elasticsearch's document-replication mechanism and lack of support for complex document relationships were becoming major bottlenecks, with no built-in capabilities for query understanding and ranking.

Implementation highlights:

• Separated indexing and searching traffic with a segment-replication model: indexer for updates, searcher for queries

• Designed a broker service to fan out queries across index shards and merge results

• Created a flexible schema model with support for indexed fields, computed fields, and query planning pipelines

• Implemented tenant isolation through "search stacks" to prevent issues in one index from affecting others

• Built a control plane for easy deployment of new versions without worrying about backward compatibility

Results and learnings:

• 50% reduction in p99.9 latency and 75% decrease in hardware costs

• Tight control over index structure and query flow enables DoorDash-specific optimizations

Turns out the best way to fix a tech scaling problem is sometimes to just build the damn thing yourself 🌯

ESSENTIAL (disagree to agree)
How Not To Disagree

ESSENTIAL (good ideas gone bad)
Systems Ideas That Sound Good But Almost Never Work

GITHUB REPO (time-traveling database)
SpacetimeDB

GITHUB REPO (AI's web buddy)
browser-use

ARTICLE (next-level traffic jam)
How much traffic can a pre-rendered Next.js site really handle?

ARTICLE (chatbot in your browser)
Fully In-Browser Graph RAG with Kuzu-Wasm

ARTICLE (FAANG no more)
I quit my FAANG job because it'll be automated by the end of 2025

ARTICLE (code readability 101)
What Makes Code Hard To Read: Visual Patterns of Complexity

ARTICLE (LLMs to the rescue)
Here's how I use LLMs to help me write code

ARTICLE (junior vs senior showdown)
The 7 Behaviors That Separate Juniors From Seniors

Want to reach 150,000+ engineers?

Let’s work together! Whether it’s your product, service, or event, we’d love to help you connect with this awesome community.

💡 Alphabet's Taara Chip Revolutionizes Internet Access with Light Beams (2 min)

Brief: Alphabet's Taara chip utilizes light beams to deliver high-speed internet at 20 Gbps, promising rapid installation and connectivity in remote areas.

🚀 Amazon-Backed Anthropic Hits $61.5 Billion Valuation After Funding Round (3 min)

Brief: Anthropic secures a $3.5 billion funding round, boosting its valuation to $61.5 billion, as it aims to enhance AI capabilities and expand into Asia and Europe.

🍏 Apple Unveils M3-Powered iPad Air with Enhanced Features (2 min)

Brief: Apple introduces the M3-powered iPad Air, featuring a spec bump with improved performance, a new Magic Keyboard, and starting prices of $599 for the 11-inch model.

🌕 NASA Uses GPS on the Moon for the First Time (3 min)

Brief: NASA's Blue Ghost lander successfully demonstrated the use of Earth-based GPS signals on the moon, enhancing navigation for future lunar missions.

🤖 Google Tests AI-Only Search Results in Major Shift (3 min)

Brief: Google is experimenting with AI-only search results using its new Gemini 2.0 model, potentially transforming how users interact with search by providing direct answers instead of traditional links.

📝 Mistral's New OCR API Transforms PDFs into AI-Ready Markdown Files (2 min)

Brief: Mistral unveils an OCR API that converts PDF documents into AI-compatible Markdown, streamlining the process for developers and enhancing document accessibility.

This week’s coding challenge:

This week’s tip:

In Python, when working with PyTorch, you can use torch.autograd.functional.jacobian to compute the Jacobian matrix of a function efficiently, which is particularly useful for advanced optimization or sensitivity analysis in neural networks.
This method computes partial derivatives of outputs with respect to inputs in a single call, avoiding manual gradient computation loops.

Wen?

• Neural Network Sensitivity Analysis: Use it to understand how small changes in inputs affect outputs, crucial for debugging or interpreting model behavior in deep learning.

• Custom Optimization Algorithms: Apply it when implementing algorithms requiring higher-order derivatives, like Newton’s method, where the Jacobian informs step directions.

• Physics-Informed Neural Networks (PINNs): Essential for computing derivatives of predictions with respect to inputs (e.g., time or space) to enforce physical constraints in the loss function.

"The most common way people give up their power is by thinking they don't have any."
Alice Walker

That’s it for today! ☀️

Enjoyed this issue? Send it to your friends here to sign up, or share it on Twitter!

If you want to submit a section to the newsletter or tell us what you think about today’s issue, reply to this email or DM me on Twitter! 🐦

Thanks for spending part of your Monday morning with Hungry Minds.
See you in a week — Alex.

Icons by Icons8.

*I may earn a commission if you get a subscription through the links marked with “aff.” (at no extra cost to you).