Why Are the New AI Agents Choosing Markdown Over HTML?

AI agents are taking over the world, marking the next big step in AI evolution 🦖. So, what do all these agents have in common? They use Markdown instead of raw HTML when processing content on web pages ⛓️. Curious to know why?

This blog post will show you how this simple trick can save you up to 99% in tokens and money!

AI Agents and Data Processing: An Introduction

AI agents are software systems that harness the power of artificial intelligence to accomplish tasks and pursue goals on behalf of users. Equipped with reasoning, planning, and memory, these agents can make decisions, learn, and adapt—all on their own. 🤯

In recent months, AI agents have taken off, especially in the world of browser automation. These AI agent browsers enable you to use LLMs to control browsers programmatically, automating tasks like adding products to your Amazon cart 🛒.

Ever wondered which libraries and frameworks power AI agents like Crawl4AI, ScrapeGraphAI, and LangChain?

When processing data from web pages, these solutions often convert HTML into Markdown automatically—or offer methods to do so—before sending the data to LLMs. But why do these AI agents favor Markdown over HTML? 🧐

The short answer is: to save tokens and speed up processing! ⏩

Time to dig deeper! But first, let’s take a look at another popular approach AI agents use to reduce data load. 👀

From Data Overload to Clarity: AI Agents’ First Move

Imagine you want your AI agent to:

1. Connect to an e-commerce site (e.g. Amazon)

2. Search for a product (e.g. PlayStation 5)

3. Extract data from that specific product page

That’s a common scenario for an AI agent, as e-commerce scraping is a wild ride 🎢. After all, product pages are a chaotic mess of ever-changing layouts, making programmatic data parsing a nightmare. That’s where AI agents flex their superpowers 💪, leveraging LLMs to extract data seamlessly—no matter how messy the page structure!

Now, let’s say you’re on a mission to grab all the juicy details from the PlayStation 5 product page on Amazon 🎮:

Here’s how you’d command your AI agent browser to make it happen:

Navigate to Amazon's homepage. Search for 'PlayStation 5' and select the top result. 
Extract the product title, price, availability, and customer ratings. 
Return the data in a structured JSON format.

That’s what the AI agent should (hopefully 🤞) do:

1. Open Amazon in the browser 🌍

2. Search for the “PlayStation 5” 🔍

3. Identify the correct product 🎯

4. Extract the product details from the page and return it in JSON 📄

But here’s the real challenge—Step 4. The Amazon PlayStation 5 product page is a beast! The HTML is packed with tons of information, most of which you don’t even need.

Want proof? Copy the page’s full HTML of the page from your browser’s DOM and drop it into a tool like the LLM Token Calculator tool:

🚨 Brace yourself…

896,871 tokens?! 😱 Yeah, you read that right—eight hundred ninety-six thousand, eight hundred seventy-one freaking tokens!

That’s a MASSIVE load of data—aka a ton of money! 💸 (Over $2 per request on GPT-4o! 😬)

As you can imagine, passing all that data to an AI agent comes with major limitations:

1. May require premium/pro plans that support high token usage 💰
2. Costs a fortune—especially if you’re running frequent queries 🤑
3. Slows down responses since the AI has to process a ridiculous amount of info ⏳

The Fix: Trim the Fat

Most AI agents let you specify a CSS selector to extract only relevant sections of a webpage. Others use heuristic algorithms to auto-filter content—like stripping out headers and footers (which usually add no value). ✂️

For example, if you inspect Amazon’s PlayStation 5 product page, you’ll notice that most of the useful content lives inside the HTML element identified by the #ppd CSS selector:

Now, what if you tell your AI agent to focus only on the #ppd element instead of the entire page? Would that make a difference? 🤔

Let’s put it to the test in the head-to-head showdown below! 🔥

Markdown vs HTML in AI Data Processing: A Head-to-Head Comparison

Compare the token usage when processing a portion of a web page directly versus converting it into Markdown.

HTML

In your browser, copy the HTML of the #ppd element, and drop it into an LLM Token Calculator tool:

From 896,871 tokens down to just 309,951—nearly a 65% save!

That’s a huge drop, sure, but let’s be real—it’s still way too many tokens! 😵‍💸

Markdown

Now, let’s replicate the trick that AI agents use by leveraging an HTML-to-Markdown conversion tool online. But first, remember that AI agents perform some preprocessing to remove content-insignificant tags like and and tags const scriptRegex = /

Next, copy the cleaned HTML and convert it into Markdown using an online HTML-to-Markdown conversion tool:

{
  "product_title": "PlayStation®5 console (slim)",
  "price": "$499.00",
  "availability": "In stock",
  "customer_ratings": {
    "rating": 4.6,
    "total_ratings": 5814
  }
}