[Bug]: When extracting data with scroll_full_page, only the final elements get parsed

[Popeyef5]

crawl4ai version

0.4.248

Expected Behavior

I'm crawiling Twitter, specifically the "following" section of a profile. I have a css selector for the relevant data (user's names and bios) and set up a JsonCssExtractionStrategy. If I don't use scroll_full_page, I understandably only expect to get the first N user profiles. But if I do enable scroll_full_page, I expect the returned data to contain the list to the fullest extent as visible when browsing manually.

Current Behavior

When not using scroll_full_page, I do get the first 16 profiles in this case. However, when setting scroll_full_page, I only get the LAST 12. It's important to note that there are over 40 profiles listed, so none of the 12 profiles intersect with the first 16. I checked the result's html property and it does only contain information about the last 12. However, strangely the screenshot saved contains all the profiles.

Is this reproducible?

Yes

Inputs Causing the Bug

https://x.com/SomeTwitterProfile/following

Steps to Reproduce

Execute the following snippet with both scoll_full_page on and off.

Code snippets

site_url = "https://x.com/elonmusk/following"

import asyncio
from crawl4ai import AsyncWebCrawler, CacheMode, BrowserConfig, CrawlerRunConfig
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
import json


async def main():
    schema = {
        "name": "Followers",
        "baseSelector": "button[data-testid='UserCell']",
        "fields": [
            {
              "name": "name",
              "selector": "span",
              "type": "text"
	    },
            {
              "name": "handle",
              "selector": 'a[role="link"] > div > div[dir="ltr"]:only-child > span ',
              "type": "text"
	    },
            {
              "name": "bio",
              "selector": "div[dir='auto'] > span",
              "type": "html",
	     }
                        
	 ]
    }
    
    extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)
    
    browser_conf = BrowserConfig(
        extra_args=['--disable-web-security'],
        cookies=[
            {"name": "auth_token", "value": "YOURAUTHTOKEN", "domain": ".x.com", "path": "/"},
        ],
    )
    
    crawler_config = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS, #        cache_mode=CacheMode.BYPASS/DISABLED,
        screenshot=True,
        wait_for="css:button[data-testid='UserCell']:nth-child(1)",
        page_timeout=5000,
        extraction_strategy=extraction_strategy,
        # scan_full_page=True,
        scroll_delay=1.5,
    )

    try:
        async with AsyncWebCrawler(
            config=browser_conf,
            verbose=True,
        ) as crawler:
            result = await crawler.arun(
                url=site_url ,
                config=crawler_config,
            )
                
            from base64 import b64decode
            with open("screenshot.png", "wb") as f:
              f.write(b64decode(result.screenshot))
              
            with open("index.html", "w") as f:
              f.write(result.html)
                                
            data = json.loads(result.extracted_content)
            print(f"Extracted {len(data)} users")
            print(json.dumps(data, indent=2) if data else "No data found")
                    
    except Exception as e:
        print(f"Something happened: {e}")
        raise e

if __name__ == "__main__":
    asyncio.run(main())

OS

Linux

Python version

3.12.9

Browser

Default

Browser version

No response

Error logs & Screenshots (if applicable)

No response

[aravindkarnam]

@Popeyef5 I'm trying to troubleshoot this. Actually running into some wierd CORS errors, even with --disable-web-security passed as extra_args. I'm not sure how you got past that. Anyway we are trying to figure out a solution. I'll keep you updated.

@unclecode This was the issue I referred in our browser profiles chat.

[unclecode]

@Popeyef5 I will be checking your code later, but based on experience, what you’re encountering is likely a common technique used in infinite scrolling websites.

Many modern websites optimize memory usage by keeping a fixed number of UI elements visible, regardless of how much data exists. This prevents excessive memory consumption.

For example:
• If a site has thousands of posts, but a user only sees 10 at a time, the page may only render 30 elements (10 currently visible + 10 before + 10 after).
• As you scroll, these same 30 elements get reused, dynamically replacing old content with new data.

How to Handle It? Since scrolling alone won’t load all items into the DOM, you need to:
1/ Extract the initial HTML using JSON extraction strategy.
2/ Execute JavaScript to scroll and trigger the next batch of items.
3/ Extract HTML again, accumulating new data at each iteration.
4/ Repeat the process until you reach the end.

Btw I’m planning to introduce built-in scroll-based extraction techniques in Crawl4AI, which will:

• Extract while scrolling, rather than jumping too far.
• Not just scroll, but accumulate HTML progressively.

For now, you can verify this behavior using Chrome DevTools—you’ll likely see a fixed number of elements in the DOM that update dynamically as you scroll.

Hope this helps!

[dchang10]

any update on this? @unclecode

[ntohidi]

hi @dchang10 , we're currently working on making infinite scrolling a built-in feature in Crawl4AI. Hopefully, it’ll be available by the end of this quarter.

[gnikoloudis]

Hi
I had the same problem and I did the following, this way you load all the elements by counting the number of classes

while True:
print(f"\n--- Iteration {iteration} ---")

        # --- Parse HTML and count the target class ---
        soup = BeautifulSoup(str(result.html), 'html.parser')
        # Use soup.select() which returns a list of all matching elements
        elements = soup.find_all('a',target_class_selector)
        current_class_count = len(elements)
        
        print(f"Found {current_class_count} elements with selector '{target_class_selector}'.")

        # --- Check if the count has stabilized ---
        if current_class_count == last_class_count:
            print("Element count has stabilized. Reached the end of the scroll.")
            break
        
        # Update the count for the next iteration
        last_class_count = current_class_count

        # --- Scroll down to load more content ---
        print("Scrolling down...")
        scroll_config = CrawlerRunConfig(
            session_id=SESSION_ID,
            js_code="window.scrollTo(0, document.body.scrollHeight-100);",
            js_only=True, # Important: operate on the same page
            delay_before_return_html =5, # Wait 3s for new content. Adjust as needed.
        )
        
        result = await crawler.arun(url, config=scroll_config)
        iteration += 1

then I used a new configuration as the below

crawler_config = CrawlerRunConfig(
session_id=SESSION_ID,
cache_mode=CacheMode.BYPASS,
extraction_strategy=extraction_strategy,
scan_full_page=True

where I enforced the scan_full_page.

Let me know if this works out.

[mkaesler44]

@unclecode @ntohidi Hi all just want to follow up on this, is this feature still in the plans for this quarter? It would be very helpful!

[unclecode]

Thanks for the report! This has been fixed on the develop branch and will be included in the next release.

[IbrahimLaeeq]

hey @unclecode can you confirm when the next release will happen as i need to see if if I can wait for the next release or if I have to do custom code for this.

[ntohidi]

hi @IbrahimLaeeq , We have already shipped the new release, version 0.8.5. It's now available on PyPI, and the Docker image has also been updated.
We will update the main branch by today :)

[IbrahimLaeeq]

@ntohidi

Cool thx .Actually I updated my crawl for AI version to the latest release, the 0.8.5, but still the error was persisting, this exact same error using these exact same parameters. I was kind of confused; I thought maybe the new release hadn't shipped.

I also used Claude code to investigate the commits on the main branch and stuff to find the relevant commit but it wasn't fixed there. Claude code was just saying it found a commit from a couple weeks ago but it was for virtual scroll config and not scan full_page so yeah.