Search engine crawlers and robots.txt

SEO search engines issues, too much traffic due to crawlers and robots are dealt with using the robots.txt or search engine settings.

In order to configure search engine visibility, you'll want to review our documentation on configuring search engine settings.

To restrict indexers on your production environment, run the following command:

platform environment:info --environment PRODUCTION_ENVIRONMENT_NAME restrict_robots true

Search engine indexers are told to ignore all development environments by default. When you’re ready to go live, you give your production environment a custom domain and then set it to be visible to search engines.

If you're having recurring issues with aggressive crawler, review this community guide, Diagnosing and resolving issues with excessive bot access.