Merge remote-tracking branch 'remotes/origin/develop' into 2168-media-preview-proxy

# Conflicts:
#	config/config.exs
#	lib/pleroma/web/media_proxy/media_proxy.ex
#	lib/pleroma/web/media_proxy/media_proxy_controller.ex

4 files changed, 454 insertions, 39 deletions

diff --git a/docs/API/admin_api.md b/docs/API/admin_api.md
index c455047cc..baf895d90 100644
--- a/docs/API/admin_api.md
+++ b/docs/API/admin_api.md
@@ -488,30 +488,52 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
 
 ### Change the user's email, password, display and settings-related fields
 
-- Params:
-  - `email`
-  - `password`
-  - `name`
-  - `bio`
-  - `avatar`
-  - `locked`
-  - `no_rich_text`
-  - `default_scope`
-  - `banner`
-  - `hide_follows`
-  - `hide_followers`
-  - `hide_followers_count`
-  - `hide_follows_count`
-  - `hide_favorites`
-  - `allow_following_move`
-  - `background`
-  - `show_role`
-  - `skip_thread_containment`
-  - `fields`
-  - `discoverable`
-  - `actor_type`
-
-- Response: none (code `200`)
+* Params:
+  * `email`
+  * `password`
+  * `name`
+  * `bio`
+  * `avatar`
+  * `locked`
+  * `no_rich_text`
+  * `default_scope`
+  * `banner`
+  * `hide_follows`
+  * `hide_followers`
+  * `hide_followers_count`
+  * `hide_follows_count`
+  * `hide_favorites`
+  * `allow_following_move`
+  * `background`
+  * `show_role`
+  * `skip_thread_containment`
+  * `fields`
+  * `discoverable`
+  * `actor_type`
+
+* Responses:
+
+Status: 200
+
+```json
+{"status": "success"}
+```
+
+Status: 400
+
+```json
+{"errors":
+  {"actor_type": "is invalid"},
+  {"email": "has invalid format"},
+  ...
+ }
+```
+
+Status: 404
+
+```json
+{"error": "Not found"}
+```
 
 ## `GET /api/pleroma/admin/reports`
 
@@ -531,7 +553,7 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
 
 ```json
 {
-  "totalReports" : 1,
+  "total" : 1,
   "reports": [
     {
       "account": {
@@ -752,7 +774,7 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
     - 400 Bad Request `"Invalid parameters"` when `status` is missing
   - On success: `204`, empty response
 
-## `POST /api/pleroma/admin/reports/:report_id/notes/:id`
+## `DELETE /api/pleroma/admin/reports/:report_id/notes/:id`
 
 ### Delete report note
 
@@ -1096,6 +1118,10 @@ Loads json generated from `config/descriptions.exs`.
 
 ### Stats
 
+- Query Params:
+  - *optional* `instance`: **string** instance hostname (without protocol) to get stats for
+- Example: `https://mypleroma.org/api/pleroma/admin/stats?instance=lain.com`
+
 - Response:
 
 ```json
@@ -1208,4 +1234,66 @@ Loads json generated from `config/descriptions.exs`.
 - Response:
   - On success: `204`, empty response
   - On failure:
-    - 400 Bad Request `"Invalid parameters"` when `status` is missing
\ No newline at end of file
+    - 400 Bad Request `"Invalid parameters"` when `status` is missing
+
+## `GET /api/pleroma/admin/media_proxy_caches`
+
+### Get a list of all banned MediaProxy URLs in Cachex
+
+- Authentication: required
+- Params:
+- *optional* `page`: **integer** page number
+- *optional* `page_size`: **integer** number of log entries per page (default is `50`)
+
+- Response:
+
+``` json
+{
+  "urls": [
+    "http://example.com/media/a688346.jpg",
+    "http://example.com/media/fb1f4d.jpg"
+  ]
+}
+
+```
+
+## `POST /api/pleroma/admin/media_proxy_caches/delete`
+
+### Remove a banned MediaProxy URL from Cachex
+
+- Authentication: required
+- Params:
+  - `urls` (array)
+
+- Response:
+
+``` json
+{
+  "urls": [
+    "http://example.com/media/a688346.jpg",
+    "http://example.com/media/fb1f4d.jpg"
+  ]
+}
+
+```
+
+## `POST /api/pleroma/admin/media_proxy_caches/purge`
+
+### Purge a MediaProxy URL
+
+- Authentication: required
+- Params:
+  - `urls` (array)
+  - `ban` (boolean)
+
+- Response:
+
+``` json
+{
+  "urls": [
+    "http://example.com/media/a688346.jpg",
+    "http://example.com/media/fb1f4d.jpg"
+  ]
+}
+
+```
diff --git a/docs/API/chats.md b/docs/API/chats.md
new file mode 100644
index 000000000..aa6119670
--- /dev/null
+++ b/docs/API/chats.md
@@ -0,0 +1,248 @@
+# Chats
+
+Chats are a way to represent an IM-style conversation between two actors. They are not the same as direct messages and they are not `Status`es, even though they have a lot in common.
+
+## Why Chats?
+
+There are no 'visibility levels' in ActivityPub, their definition is purely a Mastodon convention. Direct Messaging between users on the fediverse has mostly been modeled by using ActivityPub addressing following Mastodon conventions on normal `Note` objects. In this case, a 'direct message' would be a message that has no followers addressed and also does not address the special public actor, but just the recipients in the `to` field. It would still be a `Note` and is presented with other `Note`s as a `Status` in the API.
+
+This is an awkward setup for a few reasons:
+
+- As DMs generally still follow the usual `Status` conventions, it is easy to accidentally pull somebody into a DM thread by mentioning them. (e.g. "I hate @badguy so much")
+- It is possible to go from a publicly addressed `Status` to a DM reply, back to public, then to a 'followers only' reply, and so on. This can be become very confusing, as it is unclear which user can see which part of the conversation.
+- The standard `Status` format of implicit addressing also leads to rather ugly results if you try to display the messages as a chat, because all the recipients are always mentioned by name in the message.
+- As direct messages are posted with the same api call (and usually same frontend component) as public messages, accidentally making a public message private or vice versa can happen easily. Client bugs can also lead to this, accidentally making private messages public.
+
+As a measure to improve this situation, the `Conversation` concept and related Pleroma extensions were introduced. While it made it possible to work around a few of the issues, many of the problems remained and it didn't see much adoption because it was too complicated to use correctly. 
+
+## Chats explained
+For this reasons, Chats are a new and different entity, both in the API as well as in ActivityPub. A quick overview:
+
+- Chats are meant to represent an instant message conversation between two actors. For now these are only 1-on-1 conversations, but the other actor can be a group in the future.
+- Chat messages have the ActivityPub type `ChatMessage`. They are not `Note`s. Servers that don't understand them will just drop them.
+- The only addressing allowed in `ChatMessage`s is one single ActivityPub actor in the `to` field.
+- There's always only one Chat between two actors. If you start chatting with someone and later start a 'new' Chat, the old Chat will be continued.
+- `ChatMessage`s are posted with a different api, making it very hard to accidentally send a message to the wrong person.
+- `ChatMessage`s don't show up in the existing timelines.
+- Chats can never go from private to public. They are always private between the two actors.
+
+## Caveats
+
+- Chats are NOT E2E encrypted (yet). Security is still the same as email.
+
+## API
+
+In general, the way to send a `ChatMessage` is to first create a `Chat`, then post a message to that `Chat`. `Group`s will later be supported by making them a sub-type of `Account`.
+
+This is the overview of using the API. The API is also documented via OpenAPI, so you can view it and play with it by pointing SwaggerUI or a similar OpenAPI tool to `https://yourinstance.tld/api/openapi`.
+
+### Creating or getting a chat.
+
+To create or get an existing Chat for a certain recipient (identified by Account ID)
+you can call:
+
+`POST /api/v1/pleroma/chats/by-account-id/:account_id`
+
+The account id is the normal FlakeId of the user
+```
+POST /api/v1/pleroma/chats/by-account-id/someflakeid
+```
+
+If you already have the id of a chat, you can also use
+
+```
+GET /api/v1/pleroma/chats/:id
+```
+
+There will only ever be ONE Chat for you and a given recipient, so this call
+will return the same Chat if you already have one with that user.
+
+Returned data:
+
+```json
+{
+  "account": {
+    "id": "someflakeid",
+    "username": "somenick",
+    ...
+  },
+  "id" : "1",
+  "unread" : 2,
+  "last_message" : {...}, // The last message in that chat
+  "updated_at": "2020-04-21T15:11:46.000Z"
+}
+```
+
+### Marking a chat as read
+
+To mark a number of messages in a chat up to a certain message as read, you can use
+
+`POST /api/v1/pleroma/chats/:id/read`
+
+
+Parameters:
+- last_read_id: Given this id, all chat messages until this one will be marked as read. Required.
+
+
+Returned data:
+
+```json
+{
+  "account": {
+    "id": "someflakeid",
+    "username": "somenick",
+    ...
+  },
+  "id" : "1",
+  "unread" : 0,
+  "updated_at": "2020-04-21T15:11:46.000Z"
+}
+```
+
+### Marking a single chat message as read
+
+To set the `unread` property of a message to `false`
+
+`POST /api/v1/pleroma/chats/:id/messages/:message_id/read`
+
+Returned data:
+
+The modified chat message
+
+### Getting a list of Chats
+
+`GET /api/v1/pleroma/chats`
+
+This will return a list of chats that you have been involved in, sorted by their
+last update (so new chats will be at the top).
+
+Returned data:
+
+```json
+[
+   {
+      "account": {
+        "id": "someflakeid",
+        "username": "somenick",
+        ...
+      },
+      "id" : "1",
+      "unread" : 2,
+      "last_message" : {...}, // The last message in that chat
+      "updated_at": "2020-04-21T15:11:46.000Z"
+   }
+]
+```
+
+The recipient of messages that are sent to this chat is given by their AP ID.
+No pagination is implemented for now.
+
+### Getting the messages for a Chat
+
+For a given Chat id, you can get the associated messages with
+
+`GET /api/v1/pleroma/chats/:id/messages`
+
+This will return all messages, sorted by most recent to least recent. The usual
+pagination options are implemented.
+
+Returned data:
+
+```json
+[
+  {
+    "account_id": "someflakeid",
+    "chat_id": "1",
+    "content": "Check this out :firefox:",
+    "created_at": "2020-04-21T15:11:46.000Z",
+    "emojis": [
+      {
+        "shortcode": "firefox",
+        "static_url": "https://dontbulling.me/emoji/Firefox.gif",
+        "url": "https://dontbulling.me/emoji/Firefox.gif",
+        "visible_in_picker": false
+      }
+    ],
+    "id": "13",
+    "unread": true
+  },
+  {
+    "account_id": "someflakeid",
+    "chat_id": "1",
+    "content": "Whats' up?",
+    "created_at": "2020-04-21T15:06:45.000Z",
+    "emojis": [],
+    "id": "12",
+    "unread": false
+  }
+]
+```
+
+### Posting a chat message
+
+Posting a chat message for given Chat id works like this:
+
+`POST /api/v1/pleroma/chats/:id/messages`
+
+Parameters:
+- content: The text content of the message. Optional if media is attached.
+- media_id: The id of an upload that will be attached to the message.
+
+Currently, no formatting beyond basic escaping and emoji is implemented.
+
+Returned data:
+
+```json
+{
+  "account_id": "someflakeid",
+  "chat_id": "1",
+  "content": "Check this out :firefox:",
+  "created_at": "2020-04-21T15:11:46.000Z",
+  "emojis": [
+    {
+      "shortcode": "firefox",
+      "static_url": "https://dontbulling.me/emoji/Firefox.gif",
+      "url": "https://dontbulling.me/emoji/Firefox.gif",
+      "visible_in_picker": false
+    }
+  ],
+  "id": "13",
+  "unread": false
+}
+```
+
+### Deleting a chat message
+
+Deleting a chat message for given Chat id works like this:
+
+`DELETE /api/v1/pleroma/chats/:chat_id/messages/:message_id`
+
+Returned data is the deleted message.
+
+### Notifications
+
+There's a new `pleroma:chat_mention` notification, which has this form. It is not given out in the notifications endpoint by default, you need to explicitly request it with `include_types[]=pleroma:chat_mention`:
+
+```json
+{
+  "id": "someid",
+  "type": "pleroma:chat_mention",
+  "account": { ... } // User account of the sender,
+  "chat_message": {
+    "chat_id": "1",
+    "id": "10",
+    "content": "Hello",
+    "account_id": "someflakeid",
+    "unread": false
+  },
+  "created_at": "somedate"
+}
+```
+
+### Streaming
+
+There is an additional `user:pleroma_chat` stream. Incoming chat messages will make the current chat be sent to this `user` stream. The `event` of an incoming chat message is `pleroma:chat_update`. The payload is the updated chat with the incoming chat message in the `last_message` field.
+
+### Web Push
+
+If you want to receive push messages for this type, you'll need to add the `pleroma:chat_mention` type to your alerts in the push subscription.
diff --git a/docs/API/differences_in_mastoapi_responses.md b/docs/API/differences_in_mastoapi_responses.md
index e65fd5da4..13920e5f9 100644
--- a/docs/API/differences_in_mastoapi_responses.md
+++ b/docs/API/differences_in_mastoapi_responses.md
@@ -6,10 +6,6 @@ A Pleroma instance can be identified by "<Mastodon version> (compatible; Pleroma
 
 Pleroma uses 128-bit ids as opposed to Mastodon's 64 bits. However just like Mastodon's ids they are lexically sortable strings
 
-## Attachment cap
-
-Some apps operate under the assumption that no more than 4 attachments can be returned or uploaded. Pleroma however does not enforce any limits on attachment count neither when returning the status object nor when posting.
-
 ## Timelines
 
 Adding the parameter `with_muted=true` to the timeline queries will also return activities by muted (not by blocked!) users.
@@ -31,13 +27,22 @@ Has these additional fields under the `pleroma` object:
 - `expires_at`: a datetime (iso8601) that states when the post will expire (be deleted automatically), or empty if the post won't expire
 - `thread_muted`: true if the thread the post belongs to is muted
 - `emoji_reactions`: A list with emoji / reaction maps. The format is `{name: "☕", count: 1, me: true}`. Contains no information about the reacting users, for that use the `/statuses/:id/reactions` endpoint.
+- `parent_visible`: If the parent of this post is visible to the user or not.
 
-## Attachments
+## Media Attachments
 
 Has these additional fields under the `pleroma` object:
 
 - `mime_type`: mime type of the attachment.
 
+### Attachment cap
+
+Some apps operate under the assumption that no more than 4 attachments can be returned or uploaded. Pleroma however does not enforce any limits on attachment count neither when returning the status object nor when posting.
+
+### Limitations
+
+Pleroma does not process remote images and therefore cannot include fields such as `meta` and `blurhash`. It does not support focal points or aspect ratios. The frontend is expected to handle it.
+
 ## Accounts
 
 The `id` parameter can also be the `nickname` of the user. This only works in these endpoints, not the deeper nested ones for following etc.
@@ -47,11 +52,14 @@ The `id` parameter can also be the `nickname` of the user. This only works in th
 
 Has these additional fields under the `pleroma` object:
 
+- `ap_id`: nullable URL string, ActivityPub id of the user
+- `background_image`: nullable URL string, background image of the user
 - `tags`: Lists an array of tags for the user
-- `relationship{}`: Includes fields as documented for Mastodon API https://docs.joinmastodon.org/entities/relationship/
+- `relationship` (object): Includes fields as documented for Mastodon API https://docs.joinmastodon.org/entities/relationship/
 - `is_moderator`: boolean, nullable,  true if user is a moderator
 - `is_admin`: boolean, nullable, true if user is an admin
 - `confirmation_pending`: boolean, true if a new user account is waiting on email confirmation to be activated
+- `hide_favorites`: boolean, true when the user has hiding favorites enabled
 - `hide_followers`: boolean, true when the user has follower hiding enabled
 - `hide_follows`: boolean, true when the user has follow hiding enabled
 - `hide_followers_count`: boolean, true when the user has follower stat hiding enabled
@@ -62,6 +70,7 @@ Has these additional fields under the `pleroma` object:
 - `allow_following_move`: boolean, true when the user allows automatically follow moved following accounts
 - `unread_conversation_count`: The count of unread conversations. Only returned to the account owner.
 - `unread_notifications_count`: The count of unread notifications. Only returned to the account owner.
+- `notification_settings`: object, can be absent. See `/api/pleroma/notification_settings` for the parameters/keys returned.
 
 ### Source
 
@@ -226,3 +235,47 @@ Has theses additional parameters (which are the same as in Pleroma-API):
 Has these additional fields under the `pleroma` object:
 
 - `unread_count`: contains number unread notifications
+
+## Streaming
+
+There is an additional `user:pleroma_chat` stream. Incoming chat messages will make the current chat be sent to this `user` stream. The `event` of an incoming chat message is `pleroma:chat_update`. The payload is the updated chat with the incoming chat message in the `last_message` field.
+
+## Not implemented
+
+Pleroma is generally compatible with the Mastodon 2.7.2 API, but some newer features and non-essential features are omitted. These features usually return an HTTP 200 status code, but with an empty response. While they may be added in the future, they are considered low priority.
+
+### Suggestions
+
+*Added in Mastodon 2.4.3*
+
+- `GET /api/v1/suggestions`: Returns an empty array, `[]`
+
+### Trends
+
+*Added in Mastodon 3.0.0*
+
+- `GET /api/v1/trends`: Returns an empty array, `[]`
+
+### Identity proofs
+
+*Added in Mastodon 2.8.0*
+
+- `GET /api/v1/identity_proofs`: Returns an empty array, `[]`
+
+### Endorsements
+
+*Added in Mastodon 2.5.0*
+
+- `GET /api/v1/endorsements`: Returns an empty array, `[]`
+
+### Profile directory
+
+*Added in Mastodon 3.0.0*
+
+- `GET /api/v1/directory`: Returns HTTP 404
+
+### Featured tags
+
+*Added in Mastodon 3.0.0*
+
+- `GET /api/v1/featured_tags`: Returns HTTP 404
diff --git a/docs/API/pleroma_api.md b/docs/API/pleroma_api.md
index a25456b7d..b7eee5192 100644
--- a/docs/API/pleroma_api.md
+++ b/docs/API/pleroma_api.md
@@ -358,7 +358,7 @@ The status posting endpoint takes an additional parameter, `in_reply_to_conversa
     * `recipients`: A list of ids of users that should receive posts to this conversation. This will replace the current list of recipients, so submit the full list. The owner of owner of the conversation will always be part of the set of recipients, though.
 * Response: JSON, statuses (200 - healthy, 503 unhealthy)
 
-## `GET /api/v1/pleroma/conversations/read`
+## `POST /api/v1/pleroma/conversations/read`
 ### Marks all user's conversations as read.
 * Method `POST`
 * Authentication: required
@@ -450,18 +450,44 @@ The status posting endpoint takes an additional parameter, `in_reply_to_conversa
 * Response: JSON, list with updated files for updated pack (hashmap -> shortcode => filename) with status 200, either error status with error message.
 
 ## `GET /api/pleroma/emoji/packs`
+
 ### Lists local custom emoji packs
+
 * Method `GET`
 * Authentication: not required
-* Params: None
-* Response: JSON, "ok" and 200 status and the JSON hashmap of pack name to pack contents
+* Params:
+  * `page`: page number for packs (default 1)
+  * `page_size`: page size for packs (default 50)
+* Response: `packs` key with JSON hashmap of pack name to pack contents and `count` key for count of packs.
+
+```json
+{
+  "packs": {
+    "pack_name": {...}, // pack contents
+    ...
+  },
+  "count": 0 // packs count
+}
+```
 
 ## `GET /api/pleroma/emoji/packs/:name`
+
 ### Get pack.json for the pack
+
 * Method `GET`
 * Authentication: not required
-* Params: None
-* Response: JSON, pack json with `files` and `pack` keys with 200 status or 404 if the pack does not exist
+* Params:
+  * `page`: page number for files (default 1)
+  * `page_size`: page size for files (default 30)
+* Response: JSON, pack json with `files`, `files_count` and `pack` keys with 200 status or 404 if the pack does not exist.
+
+```json
+{
+  "files": {...},
+  "files_count": 0, // emoji count in pack
+  "pack": {...}
+}
+```
 
 ## `GET /api/pleroma/emoji/packs/:name/archive`
 ### Requests a local pack archive from the instance
@@ -536,7 +562,7 @@ Emoji reactions work a lot like favourites do. They make it possible to react to
 ```
 
 ## `GET /api/v1/pleroma/statuses/:id/reactions/:emoji`
-### Get an object of emoji to account mappings with accounts that reacted to the post for a specific emoji`
+### Get an object of emoji to account mappings with accounts that reacted to the post for a specific emoji
 * Method: `GET`
 * Authentication: optional
 * Params: None