j8ddlmZddlZddlZddlmZmZddlZddlm Z m Z m Z m Z m Z ddlmZddlZddlmZddlmZddlmZdd lmZej4eZd ed efd Zd eded e fdZ!ejDdejFZ$ded e fdZ%ded efdZ&deded e'fdZ( ddede)de)de ede eded e ee ffdZ*y)) get_browserN)urljoinurlparse)ListDictSetAnyOptional) BeautifulSoup)settings)client)"PRODUCT_LINK_CLASSIFICATION_PROMPT)insert_llm_usagehtmlreturnc g}t|d}|jdddiD]}|jxs|jxsd} t j |}t|tr|n|g}|D]_}t|tst|jdtr|jd |dDO|j|a|S#tj tf$rtjdYwxYw) zEvery JSON-LD object on a page, with @graph wrappers flattened. Must run before the crawler decomposes