LEGAL & SAFETY
What actually matters, explained by the team behind this web scraper API, not legal advice.
It depends on what you scrape, how you use it, and where you and the target site are based: this isn't a single yes/no answer, and nothing here is legal advice. Scraping publicly available data is generally on firmer ground than scraping data behind a login, personal data, or copyrighted content used without permission.
The clearest dividing line is whether the data is publicly accessible, visible to anyone without logging in, or gated behind authentication. Courts in the US have generally leaned toward the position that accessing publicly available data isn't "unauthorized access" under computer-fraud statutes; the hiQ Labs v. LinkedIn line of cases is the most frequently cited example of that reasoning. That doesn't mean public data is automatically fair game for any use, how you use it still matters.
Even where the underlying data is public, a website's terms of service can still create contract-law exposure if you agreed to them (for example, by creating an account) and then violate a no-scraping clause. This is a different legal theory than "unauthorized access," and it can apply even when the data itself was never gated.
If what you're collecting includes personal data, names tied to identifiable individuals, contact details, and similar, privacy law adds real constraints on top of everything above. GDPR in the EU is the most significant example, and it applies based on whose data you're processing, not just where your company is located.
robots.txt is a voluntary signal, not a legal wall, but respecting it (and reasonable rate limits) is good practice regardless of the legal question, since it reduces both the odds of getting blocked and the odds of a dispute escalating.
Jurisdiction varies, case law continues to develop, and the specifics of what you scrape and how you use it matter more than any blanket rule. For casual, low-volume use of clearly public data, the legal risk is generally low. For anything beyond that, commercial use, personal data, or gated content, talk to a lawyer about your specific situation rather than relying on a general FAQ answer.