because of #606, we now have a relatively decent way of representing
how post/comment state is in the database. we'll use this in our code
which is used to determine what is shown in the UI
The notifications schema is already set up for reddit-style
chronological notifications. We simply have to remove the logic that
builds the context and patch up a few places in the frontend that
were expecting full reply trees.
`Comment.header_msg` previously expected the user's own comment to
be top-level in comment replies. Logic is revised to expect the actual
reply.
`files.routes.front.notifications_main` now has reduced query volume
because we aren't expiring the session when marking notifications read.
This also allows us to remove the unused `comms` variable, which is a
makeshift "pattern" of storing duplicate database replies across a
commit so the templates don't requery (thus losing data attached to
the runtime object, like `c.unread` / `c.notif_utc`).
We move the `is_notification_page` flag to the route callers rather
than templates checking `request.path`.
Minor UI style: "Clear all notifications" -> "Mark All Read", since
this was a persistent point of user confusion upstream, with people
expecting their inbox to empty out. Also less margin between notifs
to be consistent with tighter comment display elsewhere and removed
need to separate groups of comments vs single comments.
* invisibleify completely removed trees only (fixes#431)
* fix visibility state for shadowbanned users.
this also ends up moving some of the complexity out of the templates.
* comments: remove unused variable
* moderation state machine
* no seriously this really should check for v not being None
* fix shadowban state
* fix visibility state
* update stateful counters
* don't use bespoke function for show_descendants
* properly mock ModerationState for cron submissions
* fix approval discrepency
* remove treenukes for removed comments
* show shadowbans as removed
* remove holes
* don't import class that doesn't exist
* remove subs in template code
* remove subs_created
* fix sticky bug
* uh... i think this will fix it????
* fix stupid nearly ungrokkable nesting error
* ...
* ....
* ....
* *sigh*
* .............................i think i got it.
* Revert ".............................i think i got it."
This reverts commit 419a545875.
* Revert "*sigh*"
This reverts commit fcfc9d7995.
* Revert "...."
This reverts commit e200c8f6f0.
* Revert "...."
This reverts commit 681db8cb02.
* Revert "..."
This reverts commit c54372b9ff.
* Revert "fix stupid nearly ungrokkable nesting error"
This reverts commit e202fd774c.
* Revert "uh... i think this will fix it????"
This reverts commit e4d9366093.
* readd missing endif
* fix email templates.
* fix frontpage listing
* some minor fixes wrt saving
* fix some listing errors
* Remove more references to holes system
A couple of these came from the merge of #554. A few others were just
left in the templates and presumably hard to catch on the first
removal pass.
* remove unnecessary lazy
* Add migration
Tested. Very sensitive to order. I wound up borrowing the statement
order we used for deleting sub rows on upstream (manually, of course).
---------
Co-authored-by: TLSM <duolsm@outlook.com>
* prepare codebase to create scheduled tasks
there is some prep work involved with this. the scheduler would be happy
if this work was done. simply, we extract out the `created_utc`
interface from *everything* that uses it such that we don't have to
repeat ourselves a bunch. all fun stuff.
next commit is the meat of it.
* cron: basic backend work for scheduler
* avoid ipmort loop
* attempt 2 at fixing import loops
* parathensize because operator precedence
* delete file that came back for some reason.
* does NOPing the oauth apps work?
* import late and undo clients.py change
* stringify column names.
* reorder imports.
* remove task reference
* fix missing mapper object
* make coupled to repeatabletask i guess
* sanitize: fix sanitize imports
* import shadowing crap
* re-shadow shadowed variable
* fix regexes
* use the correct not operator
* readd missing commit
* scheduler: SQLA only allows concrete relations
* implement submission scheduler
* fix import loop with db_session
* get rid of import loop in submission.py and comment.py
* remove import loops by deferring import until function clal
* i give up.
* awful.
* ...
* fix another app import loop
* fix missing import in route handler
* fix import error in wrappers.py
* fix wrapper error
* call update wrapper in the admin_level_required case
* :marseyshrug:
* fix issue with wrapper
* some cleanup and some fixes
* some more cleanup
let's avoid polluting scopes where we can.
* ...
* add SCHEDULED_POSTS permission.
* move const.py into config like the other files.
* style fixes.
* lock table for concurrency improvements
* don't attempt to commit on errors
* Refactor code, create `TaskRunContext`, create python callable task type.
* use import contextlib
* testing stuff i guess.
* handle repeatable tasks properly.
* Attempt another fix at fighting the mapper
* do it right ig
* SQLA1.4 doesn't support nested polymorphism ig
* fix errenous class import
* fix mapper errors
* import app in wrappers.py
* fix import failures and stuff like that.
* embed and import fixes
* minor formatting changes.
* Add running state enum and don't attempt to check for currently running tasks.
* isort
* documentation, style, and commit after each task.
* Add completion time and more docs, rename, etc
* document `CRON_SLEEP_SECONDS` better.
* add note about making LiteralString
* filter out tasks that have been run in the future
* reference RepeatableTask's `__tablename__` directly
* use a master/slave configuration for tasks
the master periodically checks to see if the slave is alive, healthy,
and not taking too many resources, and if applicable kills its
child and restarts it.
only one relation is supported at the moment.
* don't duplicate process unnecessarily
* note impl detail, add comments
* fix imports.
* getting imports to stop being stupid.
* environment notes.
* syntax derp
* *sigh*
* stupid environment stuff
* add UI for submitting a scheduled post
* stupid things i need to fix the user class
* ...
* fix template
* add formkey
* pass v
* add hour and minute field
* bleh
* remove concrete
* the sqlalchemy docs are wrong
* fix me being dumb and not understanding error messages
* missing author attribute for display
* author_name property
* it's a property
* with_polymorphic i think fixes this
* dsfavgnhmjk
* *sigh*
* okay try this again
* try getting rid of the comment section
* include -> extends
* put the div outside of the thing.
* fix user page listings :/
* mhm
* i hate this why isn't this working
* this should fix it
* Fix posts being set as disabled by default
* form UI imrpovements
* label
* <textarea>s should have their closing tag
* UI fixes.
* and fix errenous spinner thing.
* don't abort(415) when browsers send 0 length files for some reason
* UI improvements
* line break.
* CSS :S
* better explainer
* don't show moderation buttons for scheduled posts
* ...
* meh
* add edit form
* include forms on default page.
* fix hour minute selectino.
* improve ui i guess and add api
* Show previous postings on scheduled task page
* create task id
* sqla
* posts -> submissions
* fix OTM relationship
* edit URL
* use common formkey control
* Idk why this isn't working
* Revert "Idk why this isn't working"
This reverts commit 3b93f741df.
* does removing viewonly fix it?
* don't import routes on db migrations
* apparently this has to be a string
* UI improvements redux
* margins and stuff
* add cron to supervisord
* remove stupid duplication
* typo fix
* postgres syntax error
* better lock and error handling
* add relationship between task and runs
* fix some ui stuff
* fix incorrect timestamp comparison
* ...
* Fix logic errors blocking scheduled posts
Two bugs here:
- RepeatableTask.run_time_last <= now: run_time_last is NULL by
default. NULL is not greater than, less than, or equal to any
value. We use NULL to signify a never-run task; check for that
condition when building the task list.
- `6 <= weekday <= 0`: there is no integer that is both gte 6 and
lte 0. This was always false.
* pasthrough worker process STDOUT and STDERR
* Add scheduler to admin panel
* scheduler
* fix listing and admin home
* date formatting ixes
* fix ages
* task user interface
* fix some more import crap i have to deal with
* fix typing
* avoid import loop
* UI fixes
* fix incorrect type
* task type
* Scheduled task UI improvements (add runs and stuff)
* make the width a lil bit smaller
* task runs.
* fix submit page
* add alembic migration
* log on startup
* Fix showing edit button
* Fix logic for `can_edit` (accidentally did `author_id` instead of `id`)
* Broad review pass
Review:
- Call `invalidate_cache` with `is_html=` explicitly for clarity,
rather than a bare boolean in the call args.
- Remove `marseys_const*` and associated stateful const system:
the implementation was good if we needed them, but TheMotte
doesn't use emoji, and a greenfield emoji system would likely
not keep those darned lists floating in thread-local scope.
Also they were only needed for goldens and random emoji, which
are fairly non-central features.
- Get `os.environ` fully out of the templates by using the new
constants we already have in files.helpers.config.environment.
- Given files.routes.posts cleanup,get rid of shop discount dict.
It's already a mapping of badge IDs to discounts for badges that
likely won't continue to exist (if they even do at present).
- RepeatableTaskRun.exception: use `@property.setter` instead of
overriding `__setattr__`.
Fix:
- Welcome message literal contained an indented Markdown code block.
- Condition to show "View source" button changed to show source to
logged out. This may well be a desirable change, but it's not
clearly intended here.
* Fix couple of routing issues
* fix 400 with post body editing
* Add error handler for HTTP 415
* fix router giving wrong arg name to handler
* Use supervisord to monitor memory rather than DIY
Also means we're using pip for getting supervisord now, so we don't rely
on the Debian image base for any packages.
* fix task run elapsed time display
* formatting and removing redundant code
* Fix missing ModAction import
* dates and times fixes
* Having to modify imports here anyway, might as
well change it.
* correct documentation.
* don't use urlunparse
* validators: import sanitize instead of from syntax
* cron: prevent races on task running
RepeatableTask.run_state_enum acts as the mutex on repeatable tasks.
Previously, the list of tasks to run was acquired before individually
locking each task. However, there was a period where the table is both
unlocked and the tasks are in state WAITING between those points.
This could potentially have led to two 'cron' processes each running the
same task simultaneously. Instead, we check for runnability both when
building the preliminary list and when mutexing the task via run state
in the database.
Also:
- g.db and the cron db object are both instances of `Session`, not
`scoped_session` because they are obtained from
`scoped_session.__call__`, which acts as a `Session` factory.
Propagate this to the type hints.
- Sort order of task run submissions so /tasks/scheduled_posts/<id>
"Previous Task Runs" listings are useful.
* Notify followers on post publication
This was old behavior lost in the refactoring of the submit endpoint.
Also fix an AttributeError in `Follow.__repr__` which carried over
from all the repr copypasta.
* Fix image attachment
Any check for `file.content_length` relies on browsers sending
Content-Length headers with the request. It seems that few actually do.
The pre-refactor approach was to check for truthiness, which excludes
both None and the strange empty strings that we seem to get in absence
of a file upload. We return to doing so.
---------
Co-authored-by: TLSM <duolsm@outlook.com>
PR #547 introduced a UI bug where the comment score tooltip would
show the wrong values (and in fact, the displayed score was also
wrong, though this was not originally noticed).
Specifically, a comment with e.g. +4 | -1 would display in the tooltip
as being +4 | -4 and display a score of 4. The desired behavior would
be +4 | -1 and score of 3. Precisely, the upvote value was displayed
for each of upvotes, downvotes, and net score.
Root cause was the `@lazy` decorator at:
`files.classes.comment._score_context_str(⋅)`
`@lazy` is very dumb. I don't entirely know why we don't just use
`functools.cache`, but we use `@lazy` everywhere. It is entirely
ignorant of the parameters to a function--not a substitute for
memoization.
comments.html contains the following snippet:
{%- set ups = c.upvotes_str(render_ctx) -%}
{%- set score = c.score_str(render_ctx) -%}
{%- set downs = c.downvotes_str(render_ctx) -%}
Each of those three functions internally calls to `_score_context_str`
but with different arguments. The first call to upvotes gets cached by
`@lazy` and the two subsequent calls get the upvotes string, rather than
what they wanted. It's a cheap enough operation that it's not really
worth memoizing, so we just remove the decorator.
changed in it
the comments.html template (along with submission.html) has numerous
undesirable properties which i will describe now. unless you are very
familiar with the codebase, it can be extremely difficult to grok.
this is pretty insane as there is nothing fundamentally complex about
the goal of comments.html: return a component that shows a username
and info, reports if any, comment content, and actions a user can
take.
this behemeoth was initially 886 lines in the old version of this
codebase, and this is with awards and a lot of other cruft removed.
anyway, the maintainability of this file is about on par with some
legacy application that keels over and dies if you sneeze vaguely
in its direction.
the nicest thing i can say about it is that it isn't currently
crashing.
anyway some of the problems include:
* large, splittable components, are not split into separate files.
this makes it incredibly difficult to find or make changes across
the template and makes it nearly impossible to find or change a
specific thing.
this is most easily exemplified in the modals, which should by all
accounts be separate templates, just inlined into comments.html.
* the nesting is oftentimes incorrect.
inexplicably, probably out of laziness from when the code was first
written, things will end up fully left aligned, while multiple layers
deep into a nesting context.
if an if statement or an endif is changed, it is *incredibly*
difficult to figure out where the error was. you can't trust the
nesting.
* multiple repeated checks for things that are always true.
this is probably a symptom of the above two problems but it's very
noticeable once you fix the nesting. for example there is a block
near the very top of the actions bar which checks for
parent_submission which effectively checks "is this in a post" (this
commit won't complain about parent_submission checks but i do have
opinions on those).
all of the action buttons further down the chain also check for
parent_submission, or even check inconsistently (by using if c.post)
within this context this is a completely unnecessary check in this
context.
while it is potentially useful (and in fact because #251 requires we
dismantle the assumption a little bit) to have these checks now, the
fact that they were initially added shows that when the code was all
initial written, there was little care into thinking about comment
state.
* mobile actions are duplicated and duplicated inline.
i actually do find it probably pretty hard to support this normally
given the codebase's DOM so whatever, duplicate the things, *but* if
we're going to do that, inlining it into the middle of an incredibly
long template is really difficult to comprehend as a design decision.
...anyway yeah this PR intends to fix these problems and enable work
to be done on #251. this is a "perfect is the enemy of good" commit.
it doesn't change much fundamental and is not intended to erase the
sins of the original file, but at least make it maintainable.
this also fixes a minor bug with #473 where the GIF modal was left
in by accident.
these functions i don't believe exist upstream and i guess were to
diagnose an issue with author_ids not being saved properly. anyway not
really useful now as if an author_id doesn't exist, some major db
corruption has prolly already happened.
Bug accidentally introduced with 70c8a942b6, when a removed clause
in Comment.collapse_for_user was treated as always True, rather than
as always False--which would've removed this entire line, as this
commit does now.
Original intent of the logic was for auto-collapsing game comments
like blackjack and slots that necessarily entailed somewhat spammy
commenting behavior.
Due to use of Submission.{choices, options, bet_options} in realbody,
generating submission_listings resulted in extremely high volume of
SELECT queries.
In local testing with 6 posts, one of which had a poll with 2 options,
the removal of these calls reduced quantity of queries on the homepage
from 84 to 22.
Given that it was previously decided to remove the polls feature after
a regression while adding comment filtering, the remaining dead code
paths for polls were also removed.
Given that coins are not visible in many contexts, the conspicuous
appearance of treasure chests (random coin rewards on 1% of comments)
seems out of place. This removes the logic which rewards treasure,
the visible display of treasure, and drops the column containing
treasure information which has already been awarded to at least one
comment on prod.
In four contexts, Comment.replies(.) was not updated to reflect the
interface changes with comment filtering. This directly caused #170
and #172 (which was a stack trace from the former).
- Updating notifications for DMs (routes/users.py L690)
- Updating notifications for modmail (routes/users.py L729)
- morecomments for logged out users (routes/posts.py L421)
- JSON for API access (classes/comment.py L347)
All four contexts seem to behave correctly after the change. However,
strictly speaking the JSON generation will not include a user's own
filtered or removed comments, though this is hard to remedy without
passing the user object `v` to json_core. Propagating that through the
codebase seems a worse option than leaving it as is.
* #39 Add Flask-Migrate dep
* #39 Make it such that flask db init can run
https://github.com/miguelgrinberg/Flask-Migrate/issues/196#issuecomment-381343393
* Run flask db init, update migrations.env, commit artifacts
* Set up a script such that you can `docker-compose exec files bash -c 'cd /service; FLASK_APP="files/cli:app" flask '` and have it do whatever flask thing you want
* Fix circular dependency
* import * is evil
* Initial alembic migration, has issues with constraints and nullable columns
* Bring alts table up to date with alembic autogenerate
* Rerun flask db revision autogenerate
* Bring award_relationships table up to date with alembic autogenerate
* [#39/alembic] files/classes/__init__.py is evil but is at least explicitly evil now
* #39 fix model in files/classes/badges.py
* #39 fix model in files/classes/domains.py and files/classes/clients.py
* #39 fix models: comment saves, comment flags
* #39 fix models: comments
* Few more imports
* #39 columns that are not nullable should be flagged as not nullable
* #39 Add missing indexes to model defs
* [#39] add missing unique constraints to model defs
* [#39] Temporarily undo any model changes which cause the sqlalchemy model to be out of sync with the actual dump
* #39 Deforeignkeyify the correct column to make alembic happy
* #39 flask db revision --autogenerate now creates an empty migration
* #39 Migration format such that files are listed in creation order
* #39 Better first revision
* #39 Revert the model changes that were required to get to zero differences between db revision --autogenerate and the existing schema
* #39 The first real migration
* #39 Ensure that foreign key constraints are named in migration
* #39 Alembic migrations for FK constraints, column defs
* [#39] Run DB migrations before starting tests
* [#39] New test to ensure migrations are up to date
* [#39] More descriptive test failure message
* Add -T flag to docker-compose exec
* [#39] Run alembic migrations when starting the container
