Commit graph

76 commits

Author SHA1 Message Date
floatingghost
59bfdf2ca4 Merge pull request 'Add limit CLI flags to prune jobs' (#655) from Oneric/akkoma:prune-batch into develop
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/655
2024-06-17 20:47:53 +00:00
Floatingghost
16bed0562d Fix tests 2024-06-09 18:28:00 +01:00
Oneric
bed7ff8e89 mix: consistently use shell_info and shell_error
Logger output being visible depends on user configuration, but most of
the prints in mix tasks should always be shown. When running inside a
mix shell, it’s probably preferable to send output directly to it rather
than using raw IO.puts and we already have shell_* functions for this,
let’s use them everywhere.
2024-05-31 17:17:42 +02:00
Oneric
1d4c212441 dbprune: shortcut array activity search
This brought down query costs from 7,953,740.90 to 47,600.97
2024-05-31 17:16:40 +02:00
Oneric
6e7cbf1885 Test both standalone and flag mode for pruning orphaned activities 2024-05-31 17:16:40 +02:00
Floatingghost
66b3248dd3 mix tests probably shouldn't be async 2024-05-27 04:03:13 +01:00
Floatingghost
f15eded3e1 Add extra test case for nonsense field, increase timeouts 2024-05-27 02:09:48 +01:00
timorl
3f54945033
Fix the one test that wasn't just being flaky 2024-04-21 19:43:26 +02:00
timorl
cd7af81896
Rename StripLocation to StripMetadata for temporal-proofing reasons 2024-04-16 20:37:00 +02:00
timorl
59d32c10d9
Formatting 2024-04-16 08:02:13 +02:00
timorl
b144218dce
Merge branch 'develop' into elseinspe 2024-04-14 20:31:33 +02:00
Oneric
0ec62acb9d Always insert Dedupe upload filter
This actually was already intended before to eradict all future
path-traversal-style exploits and to fix issues with some
characters like akkoma#610 in 0b2ec0ccee. However, Dedupe and
AnonymizeFilename got mixed up. The latter only anonymises the name
in Content-Disposition headers GET parameters (with link_name),
_not_ the upload path.

Even without Dedupe, the upload path is prefixed by an UUID,
so it _should_ already be hard to guess for attackers. But now
we actually can be sure no path shenanigangs occur, uploads
reliably work and save some disk space.

While this makes the final path predictable, this prediction is
not exploitable. Insertion of a back-reference to the upload
itself requires pulling off a successfull preimage attack against
SHA-256, which is deemed infeasible for the foreseeable futures.

Dedupe was already included in the default list in config.exs
since 28cfb2c37a, but this will get overridde by whatever the
config generated by the "pleroma.instance gen" task chose.

Upload+delete tests running in parallel using Dedupe might be flaky, but
this was already true before and needs its own commit to fix eventually.
2024-03-18 22:33:10 -01:00
Oneric
fef773ca35 Drop media base_url default and recommend different domain
Same-domain setups enabled now at least two exploits,
so they ought to be discouraged and definitely not be the default.
2024-03-18 22:33:10 -01:00
Oneric
8cf183cb42 Drop Chat tables
Chats were removed in 0f132b802d
2024-02-11 05:15:08 +01:00
FloatingGhost
c7aeeec232 fix yet another keyword equality check 2023-08-07 17:00:16 +01:00
FloatingGhost
c193b4d507 Remove frankly awful config file test 2023-08-06 16:20:46 +01:00
FloatingGhost
87cc5a2110 Fix uploads test being reliant on being run later 2023-08-04 12:59:33 +01:00
FloatingGhost
64e233ca20 Tag Mock-tests as "mocked" and run them seperately 2023-08-04 12:50:50 +01:00
FloatingGhost
fe8c166b8f Remove IO.inspects 2023-08-04 12:01:52 +01:00
FloatingGhost
d5de05bbe4 scream 2023-08-03 18:10:27 +01:00
FloatingGhost
98cb255d12 Support elixir1.15
OTP builds to 1.15

Changelog entry

Ensure policies are fully loaded

Fix :warn

use main branch for linkify

Fix warn in tests

Migrations for phoenix 1.17

Revert "Migrations for phoenix 1.17"

This reverts commit 6a3b2f15b7.

Oban upgrade

Add default empty whitelist

mix format

limit test to amd64

OTP 26 tests for 1.15

use OTP_VERSION tag

baka

just 1.15

Massive deps update

Update locale, deps

Mix format

shell????

multiline???

?

max cases 1

use assert_recieve

don't put_env in async tests

don't async conn/fs tests

mix format

FIx some uploader issues

Fix tests
2023-08-03 17:44:09 +01:00
ilja
f49e9e6d4c Clean up bookmarks after prune_objects
When doing prune_objects, it's possible that bookmarked objects are deleted.
This gave problems when fetching the bookmark TL.
Here we clean up the bookmarks during pruning in the case were it's possible that bookmarked objects are deleted.
2023-05-21 13:02:28 +02:00
ilja
c7fb78cc32 Move deadline and old_insert_date to setup
Several tests for prune_objetcs need a date older than the deadline for pruning, so I moved that to the setup
2023-05-21 12:01:54 +02:00
ilja
328b4d93b7 Changelog + remove some unneeded comments from the tests 2023-02-26 14:43:19 +01:00
ilja
57eef6d764 prune_objects can prune orphaned activities who reference an array of objects
E.g. Flag activities have an array of objects

We prune the activity when NONE of the objects can be found

Note that the cost of finding and deleting these is ~4x higher than finding and deleting the non-array ones

Only string:
Delete on activities  (cost=506573.48..506580.38 rows=0 width=0)

Only Array:
Delete on activities  (cost=3570359.68..4276365.34 rows=0 width=0)

(They are still executed separately, so the total cost is the sum of the two)
2023-02-26 14:41:50 +01:00
ilja
a7ec6e039c prune_objects can prune orphaned activities
We add an option to also prune remote activities who don't have existing objects any more they reference.
Rn, we only check for activities who only reference one object, not an array or embeded object.
2023-02-26 14:41:50 +01:00
ilja
7695010268 Prune Objects --keep-threads option (#350)
This adds an option to the prune_objects mix task.
The original way deleted all non-local public posts older than a certain time frame.
Here we add a different query which you can call using the option --keep-threads.

We query from the activities table all context id's where
    1. the newest activity with this context is still old
    2. none of the activities with this context is is local
    3. none of the activities with this context is bookmarked
and delete all objects with these contexts.

The idea is that posts with local activities (posts, replies, likes, repeats...) may be interesting to keep.
Besides that, a post lives in a certain context (the thread), so we keep the whole thread as well.

Caveats:
* ~~Quotes have a different context. Therefore, when someone quotes a post, it's possible the quoted post will still be deleted.~~ fixed in https://akkoma.dev/AkkomaGang/akkoma/pulls/379
* Although undocumented (in docs/docs/administration/CLI_tasks/database.md/#prune-old-remote-posts-from-the-database), the 'normal' delete action still kept old remote non-public posts. I added an option to keep this behaviour, but this also means that you now have to explicitly provide that option. **This could be considered a breaking change!**
* ~~Note that this removes from the objects table, but not from the activities.~~ See https://akkoma.dev/AkkomaGang/akkoma/pulls/427 for that.

Some statistics from explain analyse:
(cost=1402845.92..1933782.00 rows=3810907 width=62) (actual time=2562455.486..2562455.495 rows=0 loops=1)
 Planning Time: 505.327 ms
 Trigger for constraint chat_message_references_object_id_fkey: time=651939.797 calls=921740
 Trigger for constraint deliveries_object_id_fkey: time=52036.009 calls=921740
 Trigger for constraint hashtags_objects_object_id_fkey: time=20665.778 calls=921740
 Execution Time: 3287933.902 ms

***
**TODO**
1. [x] **Question:** Is it OK to keep it like this in regard to quote posts? If not (ie post quoted by local users should also be kept), should we give quotes the same context as the post they are quoting? (If we don't want to give them the same context, I'll have to see how/if I can do it without being too costly)
    * See https://akkoma.dev/AkkomaGang/akkoma/pulls/379
2. [x] **Question:** the "original" query only deletes public posts (this is undocumented, but you can check the code). This new one doesn't care for scope. From the docs I get that the idea is that posts can be refetched when needed. But I have from a trusted source that Pleroma can't refetch non-public posts. I assume that's the reason why they are kept here. I see different options to deal with this
    1. ~~We keep it as currently implemented and just don't care about scope with this option~~
    2. ~~We add logic to not delete non-public posts either (I'll have to see how costly that becomes)~~
    3. We add an extra --keep-non-public parameter. This is technically speaking breakage (you didn't have to provide a param before for this, now you do), but I'm inclined to not care much because it wasn't documented nor tested in the first place.
3. [x] See if we can do the query using Elixir
4. [x] Test on a bigger DB to see that we don't run into a timeout
5. [x] Add docs

Co-authored-by: ilja <git@ilja.space>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/350
Co-authored-by: ilja <akkoma.dev@ilja.space>
Co-committed-by: ilja <akkoma.dev@ilja.space>
2023-01-09 22:15:41 +00:00
FloatingGhost
f752126427 Remove quack, ensure adapter is finch 2022-12-11 23:22:35 +00:00
Ilja
338612d72b Use EXIF data of image to prefill image description
During attachment upload Pleroma returns a "description" field.

* This MR allows Pleroma to read the EXIF data during upload and return the description to the FE using this field.
    * If a description is already present (e.g. because a previous module added it), it will use that
    * Otherwise it will read from the EXIF data. First it will check -ImageDescription, if that's empty, it will check -iptc:Caption-Abstract
    * If no description is found, it will simply return nil, which is the default value
* When people set up a new instance, they will be asked if they want to read metadata and this module will be activated if so

There was an Exiftool module, which has now been renamed to Exiftool.StripLocation
2022-10-23 14:46:16 +02:00
floatingghost
8e4de118c1 Don't persist local undone follow (#194)
same deal but backwards this time

Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/194
2022-08-31 18:00:36 +00:00
FloatingGhost
0f132b802d purge chat and shout endpoints 2022-07-21 11:29:28 +01:00
Tusooa Zhu
4ed0a780b7 Fix digest test 2022-06-29 20:45:05 +01:00
Alex Gleason
54af527759
Upgrade Ecto to v3.6.2, remove deprecated ecto_explain 2021-06-23 13:22:32 -05:00
Mark Felder
fe40f6f291 Switch from the deprecated "use Mix.config" to "import Config" 2021-05-20 14:02:58 -05:00
Alexander Strizhakov
03843a5386
migrating config to tmp folder 2021-03-23 14:29:02 +03:00
feld
2217b35bd4 Merge branch 'ecto-rollback-in-test-env' into 'develop'
Possibility to run rollback in test env

See merge request pleroma/pleroma!3212
2021-02-26 16:47:53 +00:00
Mark Felder
e854c35e65 Convert tests to all use clear_config instead of Pleroma.Config.put 2021-01-26 11:58:43 -06:00
Mark Felder
f868dcf3ac Ensure consistent behavior between pleroma.user activate/deactivate mix tasks 2021-01-25 10:48:28 -06:00
Mark Felder
5e8da27e14 Provide pleroma.user mix task for both activate and deactivate 2021-01-25 10:36:51 -06:00
Mark Felder
28581e03ad Merge branch 'develop' into refactor/deactivated_user_field 2021-01-18 14:58:21 -06:00
Mark Felder
2c0fe2ea9e Remove toggle_confirmation; require explicit state change
Also cosmetic changes to make the code clearer
2021-01-15 13:11:51 -06:00
Mark Felder
d36182c088 Change user.confirmation_pending field to user.is_confirmed 2021-01-15 12:44:41 -06:00
Mark Felder
7516660753 Remove toggle_activation 2021-01-15 11:25:32 -06:00
Mark Felder
860b5c7804 Change user.deactivated field to user.is_active 2021-01-15 11:24:46 -06:00
Haelwenn (lanodan) Monnier
c4439c630f
Bump Copyright to 2021
grep -rl '# Copyright © .* Pleroma' * | xargs sed -i 's;Copyright © .* Pleroma .*;Copyright © 2017-2021 Pleroma Authors <https://pleroma.social/>;'
2021-01-13 07:49:50 +01:00
lain
e1e7e4d379 Object: Rework how Object.normalize works
Now it defaults to not fetching, and the option is named.
2021-01-04 13:38:31 +01:00
lain
e4f1d8f48c Merge branch 'cachex-test' into 'develop'
Test framework overhaul (speed, reliability)

See merge request pleroma/pleroma!3209
2020-12-26 10:26:35 +00:00
Alexander Strizhakov
538af14d52
possibility to run rollback in test env 2020-12-22 17:55:30 +03:00
lain
b17c36c45a Tests: More test fixes. 2020-12-21 12:59:11 +01:00
lain
9ba60f70d2 Tests: Make as many tests as possible async.
In general, tests that match these criteria can be made async:

- Doesn't use real Cachex.
- Doesn't write to the Config / Application Environment.
- Uses Mock. Using Mox is fine.
- Uses the streamer.
2020-12-21 12:21:40 +01:00