
Markus_automation
Expert in data parsing and automation
Traditional scrapers break on websites with dynamic layouts: classes are regenerated, XPath expressions stop finding elements, and regular expressions become increasingly complex. An LLM finds data based on meaning rather than an element's location, so layout changes don't affect it. However, if you give the model an entire page, you'll spend more on tokens and get data that needs to be checked again.
In this article, we'll look at when you actually need an LLM, how to prepare HTML to reduce token usage, and how to get responses in a predefined format.
Contents
Maintain your online anonymity with Octo Browser. Your real digital fingerprint cannot be tracked.
Would you like to try Octo Browser at а discount?
Use the promo code OCTOBLOG to get 30% off any subscription. This offer is valid only for new users.
How to prepare HTML before sending it to an LLM
Before sending HTML to an LLM, clean it up, convert it to Markdown, and split it into chunks. This reduces token usage and makes it easier for the model to find the data you need.
First, you need to retrieve the HTML. For many websites, a regular script is enough. But if a website blocks automated requests, you'll need browser profiles with separate fingerprints and proxies, such as those Octo Browser provides. We explain how to launch a profile using the API and connect Playwright to it here:
Raw HTML consists mostly of markup. Here's a typical button:
<button class="bg-blue-500 hover:bg-blue-700 text-white font-bold py-2 px-4 rounded mt-4 w-full flex items-center justify-center" data-id="12345">Buy</button>
<button class="bg-blue-500 hover:bg-blue-700 text-white font-bold py-2 px-4 rounded mt-4 w-full flex items-center justify-center" data-id="12345">Buy</button>
The only useful data here is the word “Buy,” which takes one token. The entire button takes 44 tokens according to the o200k_base tokenizer used by GPT-4o models. If you send this code to the API, you're paying for the CSS classes to be processed, making it harder for the model to focus on the actual data. An entire page, such as document.body.innerHTML, may not even fit within the token limit for a single request.
Preparation involves three steps.
Step 1. Clean up the DOM
Remove everything that carries no useful information: <script>, <style>, <svg>, <iframe>, and <noscript> tags, as well as base64-encoded images. You can do this directly in the browser using page.evaluate() or on the backend with BeautifulSoup or Cheerio. Then remove the class, style, and data-* attributes. If you don't need the page metadata, remove the contents of <head> as well, including stylesheet links, meta tags, and comments.

Page code before cleaning

Page code after removing scripts, styles, and attributes
At this stage, also check the page for duplicates. Many websites display prices and product names twice: one version is visible to users, while the other is hidden and intended for screen readers. The browser displays only one version, but the converter extracts both, merging them into a single line. Remove one of the versions during cleanup, for example, elements with the aria-hidden="true" attribute. Which version to keep depends on the website's markup, so inspect the code of the page you're working with before configuring the cleanup.
Step 2. Convert HTML to Markdown
Markdown preserves the same structure as HTML (headings, lists, and tables) but uses almost no markup overhead. LLMs have been trained on data from GitHub, Reddit, and Wikipedia, where Markdown is widely used, so they handle this format well. You can use libraries such as Turndown (JavaScript) or html2text (Python) for conversion.
The difference is noticeable even in a single table row. In HTML, the row <tr><td>Price</td><td><span>$1,500</span></td></tr> takes 22 tokens, while the Markdown row | Price | $1,500 | takes just 8.
When configuring the converter, set rules for links and images. Keep the href attribute only when scraping contact details or links to product pages; remove it in other cases. For images, keep only the text in the alt attribute. The less markup there is in the text, the less noise there is to distract the model from the data.

Page text after conversion: only the alt text remains for images; prices are duplicated because of hidden duplicates
Step 3. Split the content into chunks
Even if a catalog fits into the model's context window, it's better not to send it all at once. In machine learning, this is known as the Lost in the Middle phenomenon. Research has shown that LLMs are less effective at using information from the middle of a long context than information at the beginning or end. With a large catalog, this can mean that the model skips some product cards in the middle of the list or makes up entries that aren't there.
Don't split text by character count, for example, using markdown_text.substring(0, 5000): the cut may fall in the middle of a word or price. Instead, split it into logical blocks while processing the DOM:
The script finds all product-card containers, for example, using
document.querySelectorAll('.product-card').The array of cards is divided into batches, for example, 30–50 cards per batch.
Each batch is converted to Markdown separately.
The batches are sent to the LLM in parallel requests. The number of concurrent requests depends on your API limits.
The script combines the resulting JSON arrays into one.
This way, the model processes each short fragment in full, every request stays within the token limit, and parallel requests reduce the total processing time.
How to get JSON from an LLM according to a predefined schema
The Structured Outputs mode ensures that JSON is generated according to a predefined schema: the field names and data types in the model's response exactly match the schema. This is known as strict JSON.
The json_object mode, which was the standard for a long time, guarantees only valid syntax: the brackets and quotation marks will be in the right places. Field names and data types can still change. Today, the model might return a price key; tomorrow, it might return price_usd. It might also return the string "$1 500" instead of the number 1500. A script expecting price to be a number will fail when it receives such a response.
Structured Outputs uses constrained decoding. At each generation step, the API checks the possible next tokens against the JSON schema and discards any that would violate it. If the schema requires a number, the model won't be able to put letters in that field. You can conveniently define schemas using Pydantic in Python or Zod in JavaScript.
Let's test this with an example. We'll send the model a prompt that tells it to return the price in words, along with a schema that requires the price to be a number:
import os from pydantic import BaseModel from openai import OpenAI # Read the API key from an environment variable instead of hardcoding it client = OpenAI(api_key=os.environ["OPENAI_API_KEY"]) # Schema: price must be a number (float) class Product(BaseModel): title: str price: float # Adversarial prompt: ask for the price in words html_chunk = "Name: Gaming laptop. Price: one hundred thousand rubles." evil_prompt = "RETURN THE PRICE IN WORDS ONLY! No digits! This is critical!" # Structured Outputs request: the .parse method with the schema in response_format response = client.chat.completions.parse( model="gpt-4o-mini", messages=[ {"role": "system", "content": evil_prompt}, {"role": "user", "content": html_chunk}, ], response_format=Product, temperature=0.0, ) result = response.choices[0].message.parsed print("\n=== RESULT ===") print(result) print(f"Data type of the price field: {type(result.price)}")
import os from pydantic import BaseModel from openai import OpenAI # Read the API key from an environment variable instead of hardcoding it client = OpenAI(api_key=os.environ["OPENAI_API_KEY"]) # Schema: price must be a number (float) class Product(BaseModel): title: str price: float # Adversarial prompt: ask for the price in words html_chunk = "Name: Gaming laptop. Price: one hundred thousand rubles." evil_prompt = "RETURN THE PRICE IN WORDS ONLY! No digits! This is critical!" # Structured Outputs request: the .parse method with the schema in response_format response = client.chat.completions.parse( model="gpt-4o-mini", messages=[ {"role": "system", "content": evil_prompt}, {"role": "user", "content": html_chunk}, ], response_format=Product, temperature=0.0, ) result = response.choices[0].message.parsed print("\n=== RESULT ===") print(result) print(f"Data type of the price field: {type(result.price)}")
The schema guarantees the format of the response, not the accuracy of its values. If the model makes a mistake or invents a price, it will still return it in the correct field and data type. The schema also doesn't protect against responses being cut off due to token limits or the model refusing to answer, so you need to handle these cases in your code.
You can add a description to each schema field using the description parameter:
from typing import Optional from pydantic import BaseModel, Field class Product(BaseModel): title: str = Field(description="Full product name without SEO filler") price: Optional[float] = Field(description="Price as a number only. If there is a discount, use the final price") in_stock: Optional[bool] = Field(description="Whether the product is in stock")
from typing import Optional from pydantic import BaseModel, Field class Product(BaseModel): title: str = Field(description="Full product name without SEO filler") price: Optional[float] = Field(description="Price as a number only. If there is a discount, use the final price") in_stock: Optional[bool] = Field(description="Whether the product is in stock")
The model takes these descriptions into account when filling in each field.
For data extraction, set temperature to 0, or to 0.1 for some local models. The model will select the most likely tokens, making responses to the same HTML almost identical. However, APIs don't guarantee complete reproducibility, and some reasoning models don't let you configure temperature.
An LLM generates text by predicting the most likely continuation. If a smartphone's battery capacity is almost always included in the model's training data, the model may add it to the response even if the page doesn't contain that information. Two techniques can help reduce this risk:
Use optional fields in the schema, such as
Optional[float], as in the example above.Add a
nullrule to the system prompt, for example: "If the data isn't on the page, return null. Don't make up values or use your own knowledge. Extract only what is present in the provided text."
For unusual formats, add one or two input/output examples to the prompt. For instance, a website might say "Remaining: 3 pcs. (1 reserved)," leaving it unclear which number to use. One example showing the expected answer works better than a long rule.
Models that don't support Structured Outputs, including some local models and alternative APIs, may return JSON inside a Markdown code block:
```json { "price": 1500 } ```
```json { "price": 1500 } ```
The json.loads() and JSON.parse() functions will fail on this response. So clean up the response before parsing it, then validate it using the same Pydantic model:
import re # Strip Markdown code fences, then validate against the schema clean_response = re.sub(r"```(?:json)?\s*", "", raw_response).strip() product = Product.model_validate_json(clean_response)
import re # Strip Markdown code fences, then validate against the schema clean_response = re.sub(r"```(?:json)?\s*", "", raw_response).strip() product = Product.model_validate_json(clean_response)
This lets you catch formatting errors immediately rather than discovering them in your database. The more precisely you define the schema and prompt, the less room the model has for interpretation and the more predictable the result.
How to reduce the cost of LLM-based scraping at scale
When processing large volumes of data, it's more cost-effective to use an LLM to support a traditional scraper rather than to process every page with the model. Another way to save money is to run the model on your own server.
Self-healing scraper
Traditional XPath and CSS selectors should do most of the work, with the LLM stepping in only when the scraper fails:
The script scrapes pages using traditional selectors.
The website changes its layout, the selector can no longer find the price, and a
try/exceptblock catches the error.The script sends the LLM the old selector and a fragment of the new HTML, asking: "Here's the old XPath and a fragment of the new DOM. Return a new XPath for the price."
The script tests the new selector on several pages and, if it works, saves it to the configuration and resumes scraping.
Tokens are used only when something breaks, not for every page. Don't skip the validation of the new selector: the LLM might return an XPath that finds the wrong element.
Local models
OpenAI or Anthropic APIs are convenient for prototyping, but processing millions of pages every day—for example, when monitoring marketplaces—can make API costs substantial. One alternative is to run an open-weight model on your own server or a rented GPU server. Models that perform well with code, such as Qwen 3.6, are suitable for scraping. You can run them using Ollama or vLLM. Both tools support generation based on JSON schemas, so the approach described in the previous section works here, too.
In this case, your costs are limited to server rental or electricity, and your data isn't sent to third-party services. Before switching, compare extraction quality on your own pages: a local model may perform worse than a commercial one.
Conclusion
An LLM doesn't replace traditional scraping; it complements it. XPath and CSS selectors handle stable layouts faster and more cheaply, while an LLM is useful when data is scattered across text or the layout changes constantly. To make this approach predictable, remove markup and hidden duplicates from HTML, convert it to Markdown, and split it into logical batches. Request responses according to a JSON schema, and validate both the results and any new selectors in your code.
The entire process starts with retrieving the HTML, and this is where many websites put up a barrier by blocking requests based on IP addresses or browser fingerprints. In Octo Browser, each profile has its own fingerprint and proxy, so a website sees it as a separate user. Profiles can be launched using the API and connected to Playwright or Puppeteer, making them easy to integrate into a scraper. You can then process the retrieved HTML using the steps described in this article.
Maintain your online anonymity with Octo Browser. Your real digital fingerprint cannot be tracked.
Would you like to try Octo Browser at а discount?
Use the promo code OCTOBLOG to get 30% off any subscription. This offer is valid only for new users.
How to prepare HTML before sending it to an LLM
Before sending HTML to an LLM, clean it up, convert it to Markdown, and split it into chunks. This reduces token usage and makes it easier for the model to find the data you need.
First, you need to retrieve the HTML. For many websites, a regular script is enough. But if a website blocks automated requests, you'll need browser profiles with separate fingerprints and proxies, such as those Octo Browser provides. We explain how to launch a profile using the API and connect Playwright to it here:
Raw HTML consists mostly of markup. Here's a typical button:
<button class="bg-blue-500 hover:bg-blue-700 text-white font-bold py-2 px-4 rounded mt-4 w-full flex items-center justify-center" data-id="12345">Buy</button>
The only useful data here is the word “Buy,” which takes one token. The entire button takes 44 tokens according to the o200k_base tokenizer used by GPT-4o models. If you send this code to the API, you're paying for the CSS classes to be processed, making it harder for the model to focus on the actual data. An entire page, such as document.body.innerHTML, may not even fit within the token limit for a single request.
Preparation involves three steps.
Step 1. Clean up the DOM
Remove everything that carries no useful information: <script>, <style>, <svg>, <iframe>, and <noscript> tags, as well as base64-encoded images. You can do this directly in the browser using page.evaluate() or on the backend with BeautifulSoup or Cheerio. Then remove the class, style, and data-* attributes. If you don't need the page metadata, remove the contents of <head> as well, including stylesheet links, meta tags, and comments.

Page code before cleaning

Page code after removing scripts, styles, and attributes
At this stage, also check the page for duplicates. Many websites display prices and product names twice: one version is visible to users, while the other is hidden and intended for screen readers. The browser displays only one version, but the converter extracts both, merging them into a single line. Remove one of the versions during cleanup, for example, elements with the aria-hidden="true" attribute. Which version to keep depends on the website's markup, so inspect the code of the page you're working with before configuring the cleanup.
Step 2. Convert HTML to Markdown
Markdown preserves the same structure as HTML (headings, lists, and tables) but uses almost no markup overhead. LLMs have been trained on data from GitHub, Reddit, and Wikipedia, where Markdown is widely used, so they handle this format well. You can use libraries such as Turndown (JavaScript) or html2text (Python) for conversion.
The difference is noticeable even in a single table row. In HTML, the row <tr><td>Price</td><td><span>$1,500</span></td></tr> takes 22 tokens, while the Markdown row | Price | $1,500 | takes just 8.
When configuring the converter, set rules for links and images. Keep the href attribute only when scraping contact details or links to product pages; remove it in other cases. For images, keep only the text in the alt attribute. The less markup there is in the text, the less noise there is to distract the model from the data.

Page text after conversion: only the alt text remains for images; prices are duplicated because of hidden duplicates
Step 3. Split the content into chunks
Even if a catalog fits into the model's context window, it's better not to send it all at once. In machine learning, this is known as the Lost in the Middle phenomenon. Research has shown that LLMs are less effective at using information from the middle of a long context than information at the beginning or end. With a large catalog, this can mean that the model skips some product cards in the middle of the list or makes up entries that aren't there.
Don't split text by character count, for example, using markdown_text.substring(0, 5000): the cut may fall in the middle of a word or price. Instead, split it into logical blocks while processing the DOM:
The script finds all product-card containers, for example, using
document.querySelectorAll('.product-card').The array of cards is divided into batches, for example, 30–50 cards per batch.
Each batch is converted to Markdown separately.
The batches are sent to the LLM in parallel requests. The number of concurrent requests depends on your API limits.
The script combines the resulting JSON arrays into one.
This way, the model processes each short fragment in full, every request stays within the token limit, and parallel requests reduce the total processing time.
How to get JSON from an LLM according to a predefined schema
The Structured Outputs mode ensures that JSON is generated according to a predefined schema: the field names and data types in the model's response exactly match the schema. This is known as strict JSON.
The json_object mode, which was the standard for a long time, guarantees only valid syntax: the brackets and quotation marks will be in the right places. Field names and data types can still change. Today, the model might return a price key; tomorrow, it might return price_usd. It might also return the string "$1 500" instead of the number 1500. A script expecting price to be a number will fail when it receives such a response.
Structured Outputs uses constrained decoding. At each generation step, the API checks the possible next tokens against the JSON schema and discards any that would violate it. If the schema requires a number, the model won't be able to put letters in that field. You can conveniently define schemas using Pydantic in Python or Zod in JavaScript.
Let's test this with an example. We'll send the model a prompt that tells it to return the price in words, along with a schema that requires the price to be a number:
import os from pydantic import BaseModel from openai import OpenAI # Read the API key from an environment variable instead of hardcoding it client = OpenAI(api_key=os.environ["OPENAI_API_KEY"]) # Schema: price must be a number (float) class Product(BaseModel): title: str price: float # Adversarial prompt: ask for the price in words html_chunk = "Name: Gaming laptop. Price: one hundred thousand rubles." evil_prompt = "RETURN THE PRICE IN WORDS ONLY! No digits! This is critical!" # Structured Outputs request: the .parse method with the schema in response_format response = client.chat.completions.parse( model="gpt-4o-mini", messages=[ {"role": "system", "content": evil_prompt}, {"role": "user", "content": html_chunk}, ], response_format=Product, temperature=0.0, ) result = response.choices[0].message.parsed print("\n=== RESULT ===") print(result) print(f"Data type of the price field: {type(result.price)}")
The schema guarantees the format of the response, not the accuracy of its values. If the model makes a mistake or invents a price, it will still return it in the correct field and data type. The schema also doesn't protect against responses being cut off due to token limits or the model refusing to answer, so you need to handle these cases in your code.
You can add a description to each schema field using the description parameter:
from typing import Optional from pydantic import BaseModel, Field class Product(BaseModel): title: str = Field(description="Full product name without SEO filler") price: Optional[float] = Field(description="Price as a number only. If there is a discount, use the final price") in_stock: Optional[bool] = Field(description="Whether the product is in stock")
The model takes these descriptions into account when filling in each field.
For data extraction, set temperature to 0, or to 0.1 for some local models. The model will select the most likely tokens, making responses to the same HTML almost identical. However, APIs don't guarantee complete reproducibility, and some reasoning models don't let you configure temperature.
An LLM generates text by predicting the most likely continuation. If a smartphone's battery capacity is almost always included in the model's training data, the model may add it to the response even if the page doesn't contain that information. Two techniques can help reduce this risk:
Use optional fields in the schema, such as
Optional[float], as in the example above.Add a
nullrule to the system prompt, for example: "If the data isn't on the page, return null. Don't make up values or use your own knowledge. Extract only what is present in the provided text."
For unusual formats, add one or two input/output examples to the prompt. For instance, a website might say "Remaining: 3 pcs. (1 reserved)," leaving it unclear which number to use. One example showing the expected answer works better than a long rule.
Models that don't support Structured Outputs, including some local models and alternative APIs, may return JSON inside a Markdown code block:
```json { "price": 1500 } ```
The json.loads() and JSON.parse() functions will fail on this response. So clean up the response before parsing it, then validate it using the same Pydantic model:
import re # Strip Markdown code fences, then validate against the schema clean_response = re.sub(r"```(?:json)?\s*", "", raw_response).strip() product = Product.model_validate_json(clean_response)
This lets you catch formatting errors immediately rather than discovering them in your database. The more precisely you define the schema and prompt, the less room the model has for interpretation and the more predictable the result.
How to reduce the cost of LLM-based scraping at scale
When processing large volumes of data, it's more cost-effective to use an LLM to support a traditional scraper rather than to process every page with the model. Another way to save money is to run the model on your own server.
Self-healing scraper
Traditional XPath and CSS selectors should do most of the work, with the LLM stepping in only when the scraper fails:
The script scrapes pages using traditional selectors.
The website changes its layout, the selector can no longer find the price, and a
try/exceptblock catches the error.The script sends the LLM the old selector and a fragment of the new HTML, asking: "Here's the old XPath and a fragment of the new DOM. Return a new XPath for the price."
The script tests the new selector on several pages and, if it works, saves it to the configuration and resumes scraping.
Tokens are used only when something breaks, not for every page. Don't skip the validation of the new selector: the LLM might return an XPath that finds the wrong element.
Local models
OpenAI or Anthropic APIs are convenient for prototyping, but processing millions of pages every day—for example, when monitoring marketplaces—can make API costs substantial. One alternative is to run an open-weight model on your own server or a rented GPU server. Models that perform well with code, such as Qwen 3.6, are suitable for scraping. You can run them using Ollama or vLLM. Both tools support generation based on JSON schemas, so the approach described in the previous section works here, too.
In this case, your costs are limited to server rental or electricity, and your data isn't sent to third-party services. Before switching, compare extraction quality on your own pages: a local model may perform worse than a commercial one.
Conclusion
An LLM doesn't replace traditional scraping; it complements it. XPath and CSS selectors handle stable layouts faster and more cheaply, while an LLM is useful when data is scattered across text or the layout changes constantly. To make this approach predictable, remove markup and hidden duplicates from HTML, convert it to Markdown, and split it into logical batches. Request responses according to a JSON schema, and validate both the results and any new selectors in your code.
The entire process starts with retrieving the HTML, and this is where many websites put up a barrier by blocking requests based on IP addresses or browser fingerprints. In Octo Browser, each profile has its own fingerprint and proxy, so a website sees it as a separate user. Profiles can be launched using the API and connected to Playwright or Puppeteer, making them easy to integrate into a scraper. You can then process the retrieved HTML using the steps described in this article.
Stay up to date with the latest Octo Browser news
By clicking the button you agree to our Privacy Policy.
Stay up to date with the latest Octo Browser news
By clicking the button you agree to our Privacy Policy.
Stay up to date with the latest Octo Browser news
By clicking the button you agree to our Privacy Policy.

Join Octo Browser now
Or contact Customer Service at any time with any questions you might have.

Join Octo Browser now
Or contact Customer Service at any time with any questions you might have.
Join Octo Browser now
Or contact Customer Service at any time with any questions you might have.

