Merge branch 'develop' into issue/1936

1 files changed, 17 insertions, 0 deletions

diff --git a/docs/administration/CLI_tasks/robots_txt.md b/docs/administration/CLI_tasks/robots_txt.md
new file mode 100644
index 000000000..844318cc8
--- /dev/null
+++ b/docs/administration/CLI_tasks/robots_txt.md
@@ -0,0 +1,17 @@
+# Managing robots.txt
+
+{! backend/administration/CLI_tasks/general_cli_task_info.include !}
+
+## Generate a new robots.txt file and add it to the static directory
+
+The `robots.txt` that ships by default is permissive. It allows well-behaved search engines to index all of your instance's URIs.
+
+If you want to generate a restrictive `robots.txt`, you can run the following mix task. The generated `robots.txt` will be written in your instance [static directory](../../../configuration/static_dir/).
+
+```elixir tab="OTP"
+./bin/pleroma_ctl robots_txt disallow_all
+```
+
+```elixir tab="From Source"
+mix pleroma.robots_txt disallow_all
+```