Fix LinkPreview dropping links by truncating before deduplication

[jichaowang02-lang]

Summary

LinkPreview._filter_links applies the max_links limit before
deduplicating, so the limit is spent on duplicate copies of the same URL
instead of distinct URLs.

# Limit number of links
if max_links > 0 and len(filtered_urls) > max_links:
    filtered_urls = filtered_urls[:max_links]      # truncate first ...

# Remove duplicates while preserving order
seen = set(); unique_urls = []
for url in filtered_urls:                            # ... then dedup
    ...

Link extraction routinely produces the same href many times (repeated
nav / footer / CTA links), so the [:max_links] slice keeps duplicate copies
and the later dedup collapses them — yielding far fewer than max_links
unique URLs.

Example

Internal hrefs ["a", "a", "a", "b", "c", "d"] with max_links=3:

	result
expected	["a", "b", "c"] (3 distinct)
actual (before fix)	["a"] (slice takes the 3 as, dedup → 1)

Fix

Deduplicate first, then apply max_links, so the limit counts distinct
URLs. Order is still preserved, and cases where the unique count is already
below max_links are unchanged.

Testing

Adds TestFilterLinksDeduplication to tests/test_merge_head_data_scoring.py:

$ pytest tests/test_merge_head_data_scoring.py -q
13 passed

The dedup-before-limit case fails on the current code and passes with the
fix; the "fewer uniques than the limit" and "cap distinct URLs" cases are
preserved.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.