• Hungry Minds
  • Posts
  • 🍔🧠 Data Streams 101: How to Handle Petabytes of Data

🍔🧠 Data Streams 101: How to Handle Petabytes of Data

PLUS: How to Design WhatsApp ⚡, Apple's Leak on Home Device 🍎, How to Scale Databases 🚀

Author

Alex Zajac
July 8th, 2024

In partnership with

Happy Monday! ☀️

Welcome to the 1168 new hungry minds who have joined us since last Monday!
If you aren’t subscribed yet, join smart, curious, and hungry folks by subscribing here.

This week I’m off to Italy for a road trip from Milan, to Verona, to Florence, Tuscany, Cinque Terra, Pisa, and Rome 🇮🇹

As such today’s episode is going to be a bit lighter, but here’s a picture of me enjoying a Hungry Figgacia in Verona yesterday 🥪

🍔 THIS WEEK’S MENU 🥗

  • 📚 Data Streams 101, WhatsApp System Design, Quantitative Finance Toolkit

  • 🗞️ Moshi Wants to Take Down GPT-4o, Grok 2 Confirmed for August, Robot Controlled by Parts of Human Brain

  • 👨🏻‍💻 Quick byte: Dynamic queries in MongoDB for flexible search criteria.

Reading time: 5 minutes

A few words from our sponsor this week:

FREE AI & ChatGPT Masterclass to automate 50% of your workflow

More than 300 Million people use AI across the globe, but just the top 1% know the right ones for the right use-cases.

Join this free masterclass on AI tools that will teach you the 25 most useful AI tools on the internet – that too for $0 (they have 100 free seats only!)

This masterclass will teach you how to:

  • Build business strategies & solve problems like a pro

  • Write content for emails, socials & more in minutes

  • Build AI assistants & custom bots in minutes

  • Research 10x faster, do more in less time & make your life easier

You’ll wish you knew about this FREE AI masterclass sooner 😉

Food for Thought
A mindset, an example, and an action item to start the week
❝

"The function of good software is to make the complex appear to be simple.”

Grady Booch

Mindset: Simplifying complexity is the key to creating value and driving user adoption.

Example: Apple's iOS interface revolutionized smartphones by making advanced technology accessible to all.

Action item: Identify one complex process in your work and brainstorm ways to simplify it for end-users.

The Rabbit Hole
Deep dives, trends, and resources curated to stay ahead

LUC #58: Understanding Data Streams — The Solution to Handling Continuous Flows of Big Data

Plus, concurrency is NOT parallelism, explaining SOLID principles, and API design best practices

How Uber Minimizes Flaky Tests

We'll delve into the architecture of Testopedia, a service Uber uses to find and eliminate flaky tests. Plus, job market insights from Hacker News, switching to engineering management and more.

How Halo Scaled to 11.6 Million Users Using the Saga Design Pattern 🎮

#51: Break Into Saga Design Pattern (4 Minutes)

System Design: How to Scale a Database

#18 Scaling a Database

How to pass data engineer interviews in 2024

Data engineering interviews can feel unpredictable and you don’t quite know what to expect! In this article, I’ll be going over the main types of interviews to expect as you go through out your data engineer journey. The main interviews are: SQL interview (

How Indexing Information Can Make You a Better Engineer

Switching to a new team as an engineer can be challenging, especially when it comes to finding the information needed to be productive. Rather than memorizing vast amounts of data, index it.

3 Telltale Signs to Quit Your Software Engineering Job

The importance of knowing when to leave your job can’t be overstated. Making the tough decision to leave companies thus far has always netted me increased financial stability, better career opportunities, and expedited learning.

How I plan my week as a Senior Engineer in Big Tech

My exact process for achieving my goals and getting sh*t done

💾 SIDE DISHES 💾

ARTICLE (js bloat)
JavaScript Bloat in 2024

The Weekly Digest
Software, AI, and startup news worth your time

Brief: Kyutai launches Moshi, a real-time multimodal AI model challenging OpenAI with real-time interaction, support for consumer-grade hardware, and innovative training techniques.

Brief: Amazon accelerates AI endeavors by onboarding top talent from AI startup Adept and securing technology licensing to bolster its AI services amid fierce industry competition.

Brief: Apple's backend code leak reveals development of a new mysterious "Home Accessory" linked to the A18 chip, possibly running a variant of tvOS like the HomePod, hinting at upcoming Apple innovations.

Brief: Elon Musk reveals plans to launch the Grok-2 AI model in August, boasting enhanced features, reduced reliance on external training data, and improved response quality compared to its predecessor.

Brief: Chinese researchers create a brain-powered robot by connecting human brain organoids to a humanoid body, aiming to explore brain-computer interfaces and potential brain repair applications.

The Quick Byte
One coding tip because you’re technical after all

This week’s coding challenge:

Coding Challenge #67 - YAML Parser

This challenge is to build your own YAML Parser.

This week’s tip:

Using NoSQL databases like MongoDB, you can dynamically build queries based on various runtime conditions. This approach helps in handling varying query parameters efficiently without hard-coding every possible query variation.

Wen?

  • Flexible Search Criteria: Useful when implementing advanced search functionalities where users can set different filters.

  • Real-Time Data Processing: Efficient in scenarios where the query parameters frequently change based on user input or other runtime conditions.

Why?

  • Scalability and Flexibility: Allows the application to scale by adapting to different search criteria without multiple code changes.

  • Performance Optimization: Reduces overhead by only querying necessary data, which optimizes the performance and response time of database operations.

Burp-A-Laugh
The most important meal of your day

That’s it for today! ☀️

Enjoyed this issue? Send it to your friends here to sign up, or share it on Twitter!

If you want to submit a section to the newsletter or tell us what you think about today’s issue, reply to this email or DM me on Twitter! 🐦

Thanks for spending part of your Monday morning with Hungry Minds.
See you in a week — Alex.

Icons by Icons8.

*I may earn a commission if you get a subscription through the links marked with “aff.” (at no extra cost to you).