Merge remote-tracking branch 'origin/develop' into global-status-expiration

11 files changed, 285 insertions, 46 deletions

diff --git a/docs/API/admin_api.md b/docs/API/admin_api.md
index 6202c5a1a..639c3224d 100644
--- a/docs/API/admin_api.md
+++ b/docs/API/admin_api.md
@@ -409,6 +409,7 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
 
 ### Get a password reset token for a given nickname
 
+
 - Params: none
 - Response:
 
@@ -427,6 +428,14 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
   - `nicknames`
 - Response: none (code `204`)
 
+## PUT `/api/pleroma/admin/users/disable_mfa`
+
+### Disable mfa for user's account.
+
+- Params:
+  - `nickname`
+- Response: User’s nickname
+
 ## `GET /api/pleroma/admin/users/:nickname/credentials`
 
 ### Get the user's email, password, display and settings-related fields
@@ -502,7 +511,23 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
   - `discoverable`
   - `actor_type`
 
-- Response: none (code `200`)
+- Response:
+
+```json
+{"status": "success"}
+```
+
+```json
+{"errors":
+  {"actor_type": "is invalid"},
+  {"email": "has invalid format"},
+  ...
+ }
+```
+
+```json
+{"error": "Unable to update user."}
+```
 
 ## `GET /api/pleroma/admin/reports`
 
@@ -755,6 +780,17 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
     - 400 Bad Request `"Invalid parameters"` when `status` is missing
   - On success: `204`, empty response
 
+## `GET /api/pleroma/admin/statuses/:id`
+
+### Show status by id
+
+- Params:
+  - `id`: required, status id
+- Response:
+  - On failure:
+    - 404 Not Found `"Not Found"`
+  - On success: JSON, Mastodon Status entity
+
 ## `PUT /api/pleroma/admin/statuses/:id`
 
 ### Change the scope of an individual reported status
diff --git a/docs/API/differences_in_mastoapi_responses.md b/docs/API/differences_in_mastoapi_responses.md
index c099eb1a0..434ade9a4 100644
--- a/docs/API/differences_in_mastoapi_responses.md
+++ b/docs/API/differences_in_mastoapi_responses.md
@@ -6,10 +6,6 @@ A Pleroma instance can be identified by "<Mastodon version> (compatible; Pleroma
 
 Pleroma uses 128-bit ids as opposed to Mastodon's 64 bits. However just like Mastodon's ids they are lexically sortable strings
 
-## Attachment cap
-
-Some apps operate under the assumption that no more than 4 attachments can be returned or uploaded. Pleroma however does not enforce any limits on attachment count neither when returning the status object nor when posting.
-
 ## Timelines
 
 Adding the parameter `with_muted=true` to the timeline queries will also return activities by muted (not by blocked!) users.
@@ -32,12 +28,20 @@ Has these additional fields under the `pleroma` object:
 - `thread_muted`: true if the thread the post belongs to is muted
 - `emoji_reactions`: A list with emoji / reaction maps. The format is `{name: "☕", count: 1, me: true}`. Contains no information about the reacting users, for that use the `/statuses/:id/reactions` endpoint.
 
-## Attachments
+## Media Attachments
 
 Has these additional fields under the `pleroma` object:
 
 - `mime_type`: mime type of the attachment.
 
+### Attachment cap
+
+Some apps operate under the assumption that no more than 4 attachments can be returned or uploaded. Pleroma however does not enforce any limits on attachment count neither when returning the status object nor when posting.
+
+### Limitations
+
+Pleroma does not process remote images and therefore cannot include fields such as `meta` and `blurhash`. It does not support focal points or aspect ratios. The frontend is expected to handle it.
+
 ## Accounts
 
 The `id` parameter can also be the `nickname` of the user. This only works in these endpoints, not the deeper nested ones for following etc.
@@ -61,6 +65,7 @@ Has these additional fields under the `pleroma` object:
 - `deactivated`: boolean, true when the user is deactivated
 - `allow_following_move`: boolean, true when the user allows automatically follow moved following accounts
 - `unread_conversation_count`: The count of unread conversations. Only returned to the account owner.
+- `unread_notifications_count`: The count of unread notifications. Only returned to the account owner.
 
 ### Source
 
@@ -215,6 +220,13 @@ Has theses additional parameters (which are the same as in Pleroma-API):
 - `avatar_upload_limit`: The same for avatars
 - `background_upload_limit`: The same for backgrounds
 - `banner_upload_limit`: The same for banners
+- `background_image`: A background image that frontends can use
 - `pleroma.metadata.features`: A list of supported features
 - `pleroma.metadata.federation`: The federation restrictions of this instance
 - `vapid_public_key`: The public key needed for push messages
+
+## Markers
+
+Has these additional fields under the `pleroma` object:
+
+- `unread_count`: contains number unread notifications
diff --git a/docs/API/pleroma_api.md b/docs/API/pleroma_api.md
index b927be026..70d4755b7 100644
--- a/docs/API/pleroma_api.md
+++ b/docs/API/pleroma_api.md
@@ -70,7 +70,49 @@ Request parameters can be passed via [query strings](https://en.wikipedia.org/wi
 * Response: JSON. Returns `{"status": "success"}` if the account was successfully disabled, `{"error": "[error message]"}` otherwise
 * Example response: `{"error": "Invalid password."}`
 
-## `/api/pleroma/admin/`…
+## `/api/pleroma/accounts/mfa`
+#### Gets current MFA settings
+* method: `GET`
+* Authentication: required
+* OAuth scope: `read:security`
+* Response: JSON. Returns `{"enabled": "false", "totp": false }`
+
+## `/api/pleroma/accounts/mfa/setup/totp`
+#### Pre-setup the MFA/TOTP method
+* method: `GET`
+* Authentication: required
+* OAuth scope: `write:security`
+* Response: JSON. Returns `{"key": [secret_key], "provisioning_uri": "[qr code uri]"  }` when successful, otherwise returns HTTP 422 `{"error": "error_msg"}`
+
+## `/api/pleroma/accounts/mfa/confirm/totp`
+#### Confirms & enables MFA/TOTP support for user account.
+* method: `POST`
+* Authentication: required
+* OAuth scope: `write:security`
+* Params:
+    * `password`: user's password
+    * `code`: token from TOTP App
+* Response: JSON. Returns `{}` if the enable was successful, HTTP 422 `{"error": "[error message]"}` otherwise
+
+
+## `/api/pleroma/accounts/mfa/totp`
+####  Disables MFA/TOTP method for user account.
+* method: `DELETE`
+* Authentication: required
+* OAuth scope: `write:security`
+* Params:
+    * `password`: user's password
+* Response: JSON. Returns `{}` if the disable was successful, HTTP 422 `{"error": "[error message]"}` otherwise
+* Example response: `{"error": "Invalid password."}`
+
+## `/api/pleroma/accounts/mfa/backup_codes`
+####  Generstes backup codes MFA for user account.
+* method: `GET`
+* Authentication: required
+* OAuth scope: `write:security`
+* Response: JSON. Returns `{"codes": codes}`when successful, otherwise HTTP 422 `{"error": "[error message]"}`
+
+## `/api/pleroma/admin/`
 See [Admin-API](admin_api.md)
 
 ## `/api/v1/pleroma/notifications/read`
@@ -223,7 +265,7 @@ See [Admin-API](admin_api.md)
 * Method `PUT`
 * Authentication: required
 * Params:
-    * `image`: Multipart image
+    * `file`: Multipart image
 * Response: JSON. Returns a mastodon media attachment entity
   when successful, otherwise returns HTTP 415 `{"error": "error_msg"}`
 * Example response:
@@ -316,7 +358,7 @@ The status posting endpoint takes an additional parameter, `in_reply_to_conversa
     * `recipients`: A list of ids of users that should receive posts to this conversation. This will replace the current list of recipients, so submit the full list. The owner of owner of the conversation will always be part of the set of recipients, though.
 * Response: JSON, statuses (200 - healthy, 503 unhealthy)
 
-## `GET /api/v1/pleroma/conversations/read`
+## `POST /api/v1/pleroma/conversations/read`
 ### Marks all user's conversations as read.
 * Method `POST`
 * Authentication: required
@@ -384,7 +426,7 @@ The status posting endpoint takes an additional parameter, `in_reply_to_conversa
 * Authentication: required
 * Params:
   * `file`: file needs to be uploaded with the multipart request or link to remote file.
-  * `shortcode`: (*optional*) shortcode for new emoji, must be uniq for all emoji. If not sended, shortcode will be taken from original filename.
+  * `shortcode`: (*optional*) shortcode for new emoji, must be unique for all emoji. If not sended, shortcode will be taken from original filename.
   * `filename`: (*optional*) new emoji file name. If not specified will be taken from original filename.
 * Response: JSON, list of files for updated pack (hashmap -> shortcode => filename) with status 200, either error status with error message.
 
@@ -494,7 +536,7 @@ Emoji reactions work a lot like favourites do. They make it possible to react to
 ```
 
 ## `GET /api/v1/pleroma/statuses/:id/reactions/:emoji`
-### Get an object of emoji to account mappings with accounts that reacted to the post for a specific emoji`
+### Get an object of emoji to account mappings with accounts that reacted to the post for a specific emoji
 * Method: `GET`
 * Authentication: optional
 * Params: None
diff --git a/docs/administration/CLI_tasks/database.md b/docs/administration/CLI_tasks/database.md
index ff400c8ed..647f6f274 100644
--- a/docs/administration/CLI_tasks/database.md
+++ b/docs/administration/CLI_tasks/database.md
@@ -69,3 +69,32 @@ mix pleroma.database update_users_following_followers_counts
 ```sh tab="From Source"
 mix pleroma.database fix_likes_collections
 ```
+
+## Vacuum the database
+
+### Analyze
+
+Running an `analyze` vacuum job can improve performance by updating statistics used by the query planner. **It is safe to cancel this.**
+
+```sh tab="OTP"
+./bin/pleroma_ctl database vacuum analyze
+```
+
+```sh tab="From Source"
+mix pleroma.database vacuum analyze
+```
+
+### Full
+
+Running a `full` vacuum job rebuilds your entire database by reading all of the data and rewriting it into smaller
+and more compact files with an optimized layout. This process will take a long time and use additional disk space as
+it builds the files side-by-side the existing database files. It can make your database faster and use less disk space,
+but should only be run if necessary. **It is safe to cancel this.**
+
+```sh tab="OTP"
+./bin/pleroma_ctl database vacuum full
+```
+
+```sh tab="From Source"
+mix pleroma.database vacuum full
+```
\ No newline at end of file
diff --git a/docs/administration/CLI_tasks/user.md b/docs/administration/CLI_tasks/user.md
index f535dad82..afeb8d52f 100644
--- a/docs/administration/CLI_tasks/user.md
+++ b/docs/administration/CLI_tasks/user.md
@@ -95,33 +95,33 @@ mix pleroma.user sign_out <nickname>
 ```
 
 
-## Deactivate or activate a user 
+## Deactivate or activate a user
 ```sh tab="OTP"
- ./bin/pleroma_ctl user toggle_activated <nickname> 
+ ./bin/pleroma_ctl user toggle_activated <nickname>
 ```
 
 ```sh tab="From Source"
-mix pleroma.user toggle_activated <nickname> 
+mix pleroma.user toggle_activated <nickname>
 ```
 
 
-## Unsubscribe local users from a user and deactivate the user
+## Deactivate a user and unsubscribes local users from the user
 ```sh tab="OTP"
- ./bin/pleroma_ctl user unsubscribe NICKNAME
+ ./bin/pleroma_ctl user deactivate NICKNAME
 ```
 
 ```sh tab="From Source"
-mix pleroma.user unsubscribe NICKNAME
+mix pleroma.user deactivate NICKNAME
 ```
 
 
-## Unsubscribe local users from an instance and deactivate all accounts on it
+## Deactivate all accounts from an instance and unsubscribe local users on it
 ```sh tab="OTP"
- ./bin/pleroma_ctl user unsubscribe_all_from_instance <instance>
+ ./bin/pleroma_ctl user deactivate_all_from_instance <instance>
 ```
 
 ```sh tab="From Source"
-mix pleroma.user unsubscribe_all_from_instance <instance>
+mix pleroma.user deactivate_all_from_instance <instance>
 ```
 
 
@@ -177,4 +177,3 @@ mix pleroma.user untag <nickname> <tags>
 ```sh tab="From Source"
 mix pleroma.user toggle_confirmed <nickname>
 ```
-
diff --git a/docs/configuration/cheatsheet.md b/docs/configuration/cheatsheet.md
index 8b8988fa7..456762151 100644
--- a/docs/configuration/cheatsheet.md
+++ b/docs/configuration/cheatsheet.md
@@ -8,6 +8,10 @@ For from source installations Pleroma configuration works by first importing the
 
 To add configuration to your config file, you can copy it from the base config. The latest version of it can be viewed [here](https://git.pleroma.social/pleroma/pleroma/blob/develop/config/config.exs). You can also use this file if you don't know how an option is supposed to be formatted.
 
+## :chat
+
+* `enabled` - Enables the backend chat. Defaults to `true`.
+
 ## :instance
 * `name`: The instance’s name.
 * `email`: Email used to reach an Administrator/Moderator of the instance.
@@ -146,6 +150,11 @@ config :pleroma, :mrf_user_allowlist,
   * `:strip_followers` removes followers from the ActivityPub recipient list, ensuring they won't be delivered to home timelines
   * `:reject` rejects the message entirely
 
+#### mrf_steal_emoji
+* `hosts`: List of hosts to steal emojis from
+* `rejected_shortcodes`: Regex-list of shortcodes to reject
+* `size_limit`: File size limit (in bytes), checked before an emoji is saved to the disk
+
 #### :mrf_activity_expiration
 
 * `days`: Default global expiration time for all local Create activities (in days)
@@ -250,6 +259,40 @@ This section describe PWA manifest instance-specific values. Currently this opti
 * `base_url`: The base URL to access a user-uploaded file. Useful when you want to proxy the media files via another host/CDN fronts.
 * `proxy_opts`: All options defined in `Pleroma.ReverseProxy` documentation, defaults to `[max_body_length: (25*1_048_576)]`.
 * `whitelist`: List of domains to bypass the mediaproxy
+* `invalidation`: options for remove media from cache after delete object:
+    * `enabled`: Enables purge cache
+    * `provider`: Which one of  the [purge cache strategy](#purge-cache-strategy) to use.
+
+### Purge cache strategy
+
+#### Pleroma.Web.MediaProxy.Invalidation.Script
+
+This strategy allow perform external bash script to purge cache.
+Urls of attachments pass to script as arguments.
+
+* `script_path`: path to external script.
+
+Example:
+```elixir
+config :pleroma, Pleroma.Web.MediaProxy.Invalidation.Script,
+  script_path: "./installation/nginx-cache-purge.example"
+```
+
+#### Pleroma.Web.MediaProxy.Invalidation.Http
+
+This strategy allow perform custom http request to purge cache.
+
+* `method`: http method. default is `purge`
+* `headers`: http headers. default is empty
+* `options`: request options. default is empty
+
+Example:
+```elixir
+config :pleroma, Pleroma.Web.MediaProxy.Invalidation.Http,
+  method: :purge,
+  headers: [],
+  options: []
+```
 
 ## Link previews
 
@@ -460,6 +503,7 @@ the source code is here: https://github.com/koto-bank/kocaptcha. The default end
 * `base_url`: The base URL to access a user-uploaded file. Useful when you want to proxy the media files via another host.
 * `proxy_remote`: If you're using a remote uploader, Pleroma will proxy media requests instead of redirecting to it.
 * `proxy_opts`: Proxy options, see `Pleroma.ReverseProxy` documentation.
+* `filename_display_max_length`: Set max length of a filename to display. 0 = no limit. Default: 30.
 
 !!! warning
     `strip_exif` has been replaced by `Pleroma.Upload.Filter.Mogrify`.
@@ -620,24 +664,6 @@ config :pleroma, :workers,
 * `enabled: false` corresponds to `config :pleroma, :workers, retries: [federator_outgoing: 1]`
 * deprecated options: `max_jobs`, `initial_timeout`
 
-### Pleroma.Scheduler
-
-Configuration for [Quantum](https://github.com/quantum-elixir/quantum-core) jobs scheduler.
-
-See [Quantum readme](https://github.com/quantum-elixir/quantum-core#usage) for the list of supported options.
-
-Example:
-
-```elixir
-config :pleroma, Pleroma.Scheduler,
-  global: true,
-  overlap: true,
-  timezone: :utc,
-  jobs: [{"0 */6 * * * *", {Pleroma.Web.Websub, :refresh_subscriptions, []}}]
-```
-
-The above example defines a single job which invokes `Pleroma.Web.Websub.refresh_subscriptions()` every 6 hours ("0 */6 * * * *", [crontab format](https://en.wikipedia.org/wiki/Cron)).
-
 ## :web_push_encryption, :vapid_details
 
 Web Push Notifications configuration. You can use the mix task `mix web_push.gen.keypair` to generate it.
@@ -908,12 +934,33 @@ config :auto_linker,
 
 * `runtime_dir`: A path to custom Elixir modules (such as MRF policies).
 
-
 ## :configurable_from_database
 
 Boolean, enables/disables in-database configuration. Read [Transfering the config to/from the database](../administration/CLI_tasks/config.md) for more information.
 
+## :database_config_whitelist
 
+List of valid configuration sections which are allowed to be configured from the
+database. Settings stored in the database before the whitelist is configured are
+still applied, so it is suggested to only use the whitelist on instances that
+have not migrated the config to the database.
+
+Example:
+```elixir
+config :pleroma, :database_config_whitelist, [
+  {:pleroma, :instance},
+  {:pleroma, Pleroma.Web.Metadata},
+  {:auto_linker}
+]
+```
+
+### Multi-factor authentication -  :two_factor_authentication
+* `totp` - a list containing TOTP configuration
+  - `digits` - Determines the length of a one-time pass-code in characters. Defaults to 6 characters.
+  - `period` - a period for which the TOTP code will be valid in seconds. Defaults to 30 seconds.
+* `backup_codes` - a list containing backup codes configuration
+  - `number` - number of backup codes to generate.
+  - `length` - backup code length. Defaults to 16 characters.
 
 ## Restrict entities access for unauthenticated users
 
@@ -929,4 +976,9 @@ Restrict access for unauthenticated users to timelines (public and federate), us
   * `remote`
 * `activities` - statuses
   * `local`
-  * `remote`
\ No newline at end of file
+  * `remote`
+
+
+## Pleroma.Web.ApiSpec.CastAndValidate
+
+* `:strict` a boolean, enables strict input validation (useful in development, not recommended in production). Defaults to `false`.
diff --git a/docs/configuration/postgresql.md b/docs/configuration/postgresql.md
new file mode 100644
index 000000000..6983fb459
--- /dev/null
+++ b/docs/configuration/postgresql.md
@@ -0,0 +1,31 @@
+# Optimizing your PostgreSQL performance
+
+Pleroma performance depends to a large extent on good database performance. The default PostgreSQL settings are mostly fine, but often you can get better performance by changing a few settings.
+
+You can use [PGTune](https://pgtune.leopard.in.ua) to get recommendations for your setup. If you do, set the "Number of Connections" field to 20, as Pleroma will only use 10 concurrent connections anyway. If you don't, it will give you advice that might even hurt your performance.
+
+We also recommend not using the "Network Storage" option.
+
+## Example configurations
+
+Here are some configuration suggestions for PostgreSQL 10+.
+
+### 1GB RAM, 1 CPU
+```
+shared_buffers = 256MB
+effective_cache_size = 768MB
+maintenance_work_mem = 64MB
+work_mem = 13107kB
+```
+
+### 2GB RAM, 2 CPU
+```
+shared_buffers = 512MB
+effective_cache_size = 1536MB
+maintenance_work_mem = 128MB
+work_mem = 26214kB
+max_worker_processes = 2
+max_parallel_workers_per_gather = 1
+max_parallel_workers = 2
+```
+
diff --git a/docs/configuration/storing_remote_media.md b/docs/configuration/storing_remote_media.md
new file mode 100644
index 000000000..7e91fe7d9
--- /dev/null
+++ b/docs/configuration/storing_remote_media.md
@@ -0,0 +1,38 @@
+# Storing Remote Media
+
+Pleroma does not store remote/federated media by default. The best way to achieve this is to change Nginx to keep its reverse proxy cache
+for a year and to activate the `MediaProxyWarmingPolicy` MRF policy in Pleroma which will automatically fetch all media through the proxy
+as soon as the post is received by your instance.
+
+## Nginx
+
+```
+    proxy_cache_path /long/term/storage/path/pleroma-media-cache levels=1:2
+        keys_zone=pleroma_media_cache:10m inactive=1y use_temp_path=off;
+
+    location ~ ^/(media|proxy) {
+        proxy_cache        pleroma_media_cache;
+        slice              1m;
+        proxy_cache_key    $host$uri$is_args$args$slice_range;
+        proxy_set_header   Range $slice_range;
+        proxy_http_version 1.1;
+        proxy_cache_valid  206 301 302 304 1h;
+        proxy_cache_valid  200 1y;
+        proxy_cache_use_stale error timeout invalid_header updating;
+        proxy_ignore_client_abort on;
+        proxy_buffering    on;
+        chunked_transfer_encoding on;
+        proxy_ignore_headers Cache-Control Expires;
+        proxy_hide_header  Cache-Control Expires;
+        proxy_pass         http://127.0.0.1:4000;
+    }
+```
+
+## Pleroma
+
+Add to your `prod.secret.exs`:
+
+```
+config :pleroma, :instance,
+  rewrite_policy: [Pleroma.Web.ActivityPub.MRF.MediaProxyWarmingPolicy]
+```
diff --git a/docs/installation/debian_based_en.md b/docs/installation/debian_based_en.md
index 62d8733f7..2c20d521a 100644
--- a/docs/installation/debian_based_en.md
+++ b/docs/installation/debian_based_en.md
@@ -38,8 +38,8 @@ sudo apt install git build-essential postgresql postgresql-contrib
 * Download and add the Erlang repository:
 
 ```shell
-wget -P /tmp/ https://packages.erlang-solutions.com/erlang-solutions_1.0_all.deb
-sudo dpkg -i /tmp/erlang-solutions_1.0_all.deb
+wget -P /tmp/ https://packages.erlang-solutions.com/erlang-solutions_2.0_all.deb
+sudo dpkg -i /tmp/erlang-solutions_2.0_all.deb
 ```
 
 * Install Elixir and Erlang:
diff --git a/docs/installation/debian_based_jp.md b/docs/installation/debian_based_jp.md
index a3c4621d8..1e5a9be91 100644
--- a/docs/installation/debian_based_jp.md
+++ b/docs/installation/debian_based_jp.md
@@ -40,8 +40,8 @@ sudo apt install git build-essential postgresql postgresql-contrib
 
 * Erlangのリポジトリをダウンロードおよびインストールします。
 ```
-wget -P /tmp/ https://packages.erlang-solutions.com/erlang-solutions_1.0_all.deb
-sudo dpkg -i /tmp/erlang-solutions_1.0_all.deb
+wget -P /tmp/ https://packages.erlang-solutions.com/erlang-solutions_2.0_all.deb
+sudo dpkg -i /tmp/erlang-solutions_2.0_all.deb
 ```
 
 * ElixirとErlangをインストールします、
diff --git a/docs/installation/otp_en.md b/docs/installation/otp_en.md
index fb99af699..86135cd20 100644
--- a/docs/installation/otp_en.md
+++ b/docs/installation/otp_en.md
@@ -63,7 +63,7 @@ apt install postgresql-11-rum
 ```
 
 #### (Optional) Performance configuration
-For optimal performance, you may use [PGTune](https://pgtune.leopard.in.ua), don't forget to restart postgresql after editing the configuration
+It is encouraged to check [Optimizing your PostgreSQL performance](../configuration/postgresql.md) document, for tips on PostgreSQL tuning.
 
 ```sh tab="Alpine"
 rc-service postgresql restart